{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":688818431,"defaultBranch":"main","name":"mteb-long-documents","ownerLogin":"jina-ai","currentUserCanPush":false,"isFork":true,"isEmpty":false,"createdAt":"2023-09-08T07:07:14.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/60539444?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726753938.0","currentOid":""},"activityList":{"items":[{"before":"0b6b00eca96b5d9f62a80eb35d7176a99e91306e","after":null,"ref":"refs/heads/feat-code-search-net","pushedAt":"2024-09-19T13:52:18.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"}},{"before":"ae67a56ee49a216f3e3cf3b20a06578eb25d3176","after":"5ad97a9058a1d0c7c80474e9349568a7bb8813b8","ref":"refs/heads/main","pushedAt":"2024-09-19T13:52:14.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"feat: add code search net retrieval mrr (#9)\n\n* feat: add code search net retrieval mrr\r\n\r\n* feat: add code search net\r\n\r\n* feat: add code search net\r\n\r\n* fix: corpus sohuld be dict\r\n\r\n* feat: use func code tokens as query\r\n\r\n* feat: use new cleaned dataset\r\n\r\n* feat: rename params\r\n\r\n* fix: use doc tokens as query\r\n\r\n* fix: add MTEB_SINGLE_GPU env variable\r\n\r\n* feat: Added WebQuery test set as MTEB task\r\n\r\n* fix: use CoSQA\r\n\r\n* fix: fixed relevant docs for CoSQA retrieval\r\n\r\n* only add docstring to queries if it also has matching code\r\n\r\n* feat: add Adv task\r\n\r\n* fix: removed unnecessary imports\r\n\r\n* feat: add information about language to query\r\n\r\n* fix: add Adv task to import in __init__\r\n\r\n* fix: fixed filename of validation dataset\r\n\r\n* fix: added assertions\r\n\r\n* fix: use subprocess.rub\r\n\r\n* fix: use subprocess.rub\r\n\r\n* fix: check output of subprocess\r\n\r\n* fix: capture output\r\n\r\n* fix: try to fix location where we unzip\r\n\r\n* fix: try to fix location where we unzip\r\n\r\n* fix: don't append to dictionary\r\n\r\n* feat: improved quality of comment removal function that I copied from StackOverflow\r\n\r\n* feat: added query version of CodeSearchNet benchmark\r\n\r\n* fix: fixed corpus of CodeSearchNetQueryRetrieval\r\n\r\n* fix: fixed class name of CodeSearchNetQueryRetrieval\r\n\r\n* fix: fixed name of CodeSearchNetQueryRetrieval\r\n\r\n* feat: use simpler filtering of docs\r\n\r\n* feat: fixed issue where we access _queries instead of queries\r\n\r\n* fix: user .lower() on programming language labels\r\n\r\n* fix: parse relevance judgement to int\r\n\r\n* fix: removed debugging print statement\r\n\r\n* fix: removed debugging print statement\r\n\r\n* Refactored CodeSearchNetAdvRetrieval, now able to pass eval_splits to load_data\r\n\r\n* Refactored CosqaRetrieval, now able to pass eval_splits\r\n\r\n* feat: replaced cosqa data url with commit url\r\n\r\n* feat: refactored CodeSearchNetAdvRetrieval, use commit url\r\n\r\n* feat: made CodeSearchNetRetrieval multilingual\r\n\r\n* fix: use lang instead of language in CodeSearchNetRetrieval\r\n\r\n* fix: use rstrip and lstrip in CodeSearchNetQueryRetrieval and don't prepend language in CodeSearchNetAdvRetrieval\r\n\r\n* fix: updated languages of CodeSearchNetRetrieval and skip languages that are unknown\r\n\r\n* fix: use dev set of cosqa\r\n\r\n* fix: fixed cosqa pair classification\r\n\r\n* fix: use doc instead of query in CodeSearchNetRetrieval\r\n\r\n* fix: use mrr_at_10\r\n\r\n* fix: return scores from AbsTaaskRetrieval.evaluate\r\n\r\n* feat: added option to pass dictionary for eval_splits\r\n\r\n* fix: fixed formatting, removed useless branch\r\n\r\n* debugging print statement\r\n\r\n* removed debugging print statement\r\n\r\n* feat: use default eval_splits, fix category of CoSQAPc\r\n\r\n* fix: fixed url for CodeSearchNetAdvRetrieval\r\n\r\n* feat: added assertion that queries in CoSQA are unique\r\n\r\n* fix: change language of CoSQA and CSNAdv to python\r\n\r\n* Add assertion that code and doc are strings\r\n\r\n* feat: make CodeSearchNetQueryRetrieval multilingual\r\n\r\n* fix: formatting\r\n\r\n* fix: import MultilingualTask in CodeSearchNetQueryRetrieval\r\n\r\n* fix: define queries, corpus, relevant_docs in CodeSearchNetQueryRetrieval\r\n\r\n* fix: updated the filtering method in CodeSearchNetQueryRetrieval\r\n\r\n* fix: compute the NDCG as in CodeSearchNet\r\n\r\n* feat: make loading faster by using smaller dataset\r\n\r\n---------\r\n\r\nCo-authored-by: Markus Krimmel <markus.krimmel@jina.ai>","shortMessageHtmlLink":"feat: add code search net retrieval mrr (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"1994810627\" data-permission-text=\"Title is private\" data-url=\"https://github.com/jina-ai/mteb-long-documents/issues/9\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/jina-ai/mteb-long-documents/pull/9/hovercard\" href=\"https://github.com/jina-ai/mteb-long-documents/pull/9\">#9</a>)"}},{"before":"b31ecf4e28e3e8cbf591ea9ec67868b189f7582d","after":"e06860cd0f18978a21d00e4d294279a26de1c6a4","ref":"refs/heads/clinical","pushedAt":"2024-03-04T10:21:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"use paraphrase","shortMessageHtmlLink":"use paraphrase"}},{"before":"fa9c86b08c844de856d28bcde10af58c87169b8a","after":"b31ecf4e28e3e8cbf591ea9ec67868b189f7582d","ref":"refs/heads/clinical","pushedAt":"2024-03-04T09:20:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"9f60375c67dafb2e6dde6c3ecd1dd245534258eb","after":"fa9c86b08c844de856d28bcde10af58c87169b8a","ref":"refs/heads/clinical","pushedAt":"2024-03-04T08:34:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"fix","shortMessageHtmlLink":"fix"}},{"before":"d497ca5f440ce46ff90cfa78ca3d76ad4d4de083","after":"9f60375c67dafb2e6dde6c3ecd1dd245534258eb","ref":"refs/heads/clinical","pushedAt":"2024-03-02T16:30:45.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"fix: only use 30% data","shortMessageHtmlLink":"fix: only use 30% data"}},{"before":"a70eaa6bef9ed9ab0a3ebc86cfd651c3957e10ba","after":"d497ca5f440ce46ff90cfa78ca3d76ad4d4de083","ref":"refs/heads/clinical","pushedAt":"2024-03-02T15:45:44.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"fix: use test as eval split","shortMessageHtmlLink":"fix: use test as eval split"}},{"before":"86afc9554d36f4dcf61d91299e23ca33c2169e37","after":"a70eaa6bef9ed9ab0a3ebc86cfd651c3957e10ba","ref":"refs/heads/clinical","pushedAt":"2024-03-02T15:42:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"feat: fix style","shortMessageHtmlLink":"feat: fix style"}},{"before":"32e5aed921ff928e923445b089ff5a7d94ecff6d","after":"86afc9554d36f4dcf61d91299e23ca33c2169e37","ref":"refs/heads/clinical","pushedAt":"2024-03-02T15:38:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"feat: add clinical retrieval","shortMessageHtmlLink":"feat: add clinical retrieval"}},{"before":null,"after":"32e5aed921ff928e923445b089ff5a7d94ecff6d","ref":"refs/heads/clinical","pushedAt":"2024-03-02T15:32:38.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"bwanglzu","name":"Wang Bo","path":"/bwanglzu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/9794489?s=80&v=4"},"commit":{"message":"feat: add clinical qa","shortMessageHtmlLink":"feat: add clinical qa"}},{"before":"fef8b332bb26213a4b74c6b7556cf432eae21637","after":"0b6b00eca96b5d9f62a80eb35d7176a99e91306e","ref":"refs/heads/feat-code-search-net","pushedAt":"2024-01-05T15:02:01.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"feat: make loading faster by using smaller dataset","shortMessageHtmlLink":"feat: make loading faster by using smaller dataset"}},{"before":"b750460129c0b0c7b604b782775b9a3d8cee3c17","after":"fef8b332bb26213a4b74c6b7556cf432eae21637","ref":"refs/heads/feat-code-search-net","pushedAt":"2024-01-05T14:21:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: compute the NDCG as in CodeSearchNet","shortMessageHtmlLink":"fix: compute the NDCG as in CodeSearchNet"}},{"before":"1ed17d65b4b9a1c12bcbb0535b9d4cf70a471d53","after":"b750460129c0b0c7b604b782775b9a3d8cee3c17","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-15T12:40:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: updated the filtering method in CodeSearchNetQueryRetrieval","shortMessageHtmlLink":"fix: updated the filtering method in CodeSearchNetQueryRetrieval"}},{"before":"34e0a3ad7c51df95564ba99ae177a4cf80ba2b96","after":"1ed17d65b4b9a1c12bcbb0535b9d4cf70a471d53","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-14T14:47:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: define queries, corpus, relevant_docs in CodeSearchNetQueryRetrieval","shortMessageHtmlLink":"fix: define queries, corpus, relevant_docs in CodeSearchNetQueryRetri…"}},{"before":"0d72a8b57a6964e898a3fc20c4884dc1fa3f2264","after":"34e0a3ad7c51df95564ba99ae177a4cf80ba2b96","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-14T14:44:47.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: import MultilingualTask in CodeSearchNetQueryRetrieval","shortMessageHtmlLink":"fix: import MultilingualTask in CodeSearchNetQueryRetrieval"}},{"before":"f09ed038089fa6b385fdf3591574e92521e644c7","after":"0d72a8b57a6964e898a3fc20c4884dc1fa3f2264","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-13T13:08:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: formatting","shortMessageHtmlLink":"fix: formatting"}},{"before":"578d7b0d60f4faac6f794b8a6d3d6b1712c952a3","after":"f09ed038089fa6b385fdf3591574e92521e644c7","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-13T13:08:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"feat: make CodeSearchNetQueryRetrieval multilingual","shortMessageHtmlLink":"feat: make CodeSearchNetQueryRetrieval multilingual"}},{"before":"564a49231d80f413d10c4adba7f547b240007c78","after":"578d7b0d60f4faac6f794b8a6d3d6b1712c952a3","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-11T10:28:06.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"Add assertion that code and doc are strings","shortMessageHtmlLink":"Add assertion that code and doc are strings"}},{"before":"a3d9f21b049adeb56c3c11dab24c3a779bd8c201","after":"564a49231d80f413d10c4adba7f547b240007c78","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T16:03:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: change language of CoSQA and CSNAdv to python","shortMessageHtmlLink":"fix: change language of CoSQA and CSNAdv to python"}},{"before":"79af5528e6695d1ef003d45b5eb2a32c02b32be2","after":"a3d9f21b049adeb56c3c11dab24c3a779bd8c201","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T15:54:29.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"feat: added assertion that queries in CoSQA are unique","shortMessageHtmlLink":"feat: added assertion that queries in CoSQA are unique"}},{"before":"9467d01a93927247a383dcbd4d78659277dde9e3","after":"79af5528e6695d1ef003d45b5eb2a32c02b32be2","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T15:00:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: fixed url for CodeSearchNetAdvRetrieval","shortMessageHtmlLink":"fix: fixed url for CodeSearchNetAdvRetrieval"}},{"before":"52bbc8203315ad73a73a45d6d61d919040470219","after":"9467d01a93927247a383dcbd4d78659277dde9e3","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T14:59:16.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"feat: use default eval_splits, fix category of CoSQAPc","shortMessageHtmlLink":"feat: use default eval_splits, fix category of CoSQAPc"}},{"before":"a00fc7b434c26cc1b948da36f6ce9143ca157c7d","after":"52bbc8203315ad73a73a45d6d61d919040470219","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T14:53:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"removed debugging print statement","shortMessageHtmlLink":"removed debugging print statement"}},{"before":"bddf9e4cd67e800ef477b344445e843a52028a9d","after":"a00fc7b434c26cc1b948da36f6ce9143ca157c7d","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T14:50:18.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"debugging print statement","shortMessageHtmlLink":"debugging print statement"}},{"before":"b0f41db20d30f3677eb55074b74bf3f30523bc7b","after":"bddf9e4cd67e800ef477b344445e843a52028a9d","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T14:49:17.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: fixed formatting, removed useless branch","shortMessageHtmlLink":"fix: fixed formatting, removed useless branch"}},{"before":"5070dfd3aed14e702b48622c87972a206f580cda","after":"b0f41db20d30f3677eb55074b74bf3f30523bc7b","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T14:32:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"feat: added option to pass dictionary for eval_splits","shortMessageHtmlLink":"feat: added option to pass dictionary for eval_splits"}},{"before":"81d2199736a2a9c6267b9d91514ef02e675b403d","after":"5070dfd3aed14e702b48622c87972a206f580cda","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T13:53:00.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: return scores from AbsTaaskRetrieval.evaluate","shortMessageHtmlLink":"fix: return scores from AbsTaaskRetrieval.evaluate"}},{"before":"367761eea654d5185ca1b0446cba2efc822755fb","after":"81d2199736a2a9c6267b9d91514ef02e675b403d","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T13:51:07.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: use mrr_at_10","shortMessageHtmlLink":"fix: use mrr_at_10"}},{"before":"9739ae192283ae33f005e534896af297b1f15470","after":"367761eea654d5185ca1b0446cba2efc822755fb","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T13:33:47.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: use doc instead of query in CodeSearchNetRetrieval","shortMessageHtmlLink":"fix: use doc instead of query in CodeSearchNetRetrieval"}},{"before":"2c3631959b6d33806df52cbb56cdc0471b609425","after":"9739ae192283ae33f005e534896af297b1f15470","ref":"refs/heads/feat-code-search-net","pushedAt":"2023-12-08T13:30:03.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Markus28","name":"Markus Krimmel","path":"/Markus28","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/15806078?s=80&v=4"},"commit":{"message":"fix: fixed cosqa pair classification","shortMessageHtmlLink":"fix: fixed cosqa pair classification"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEuu0MHgA","startCursor":null,"endCursor":null}},"title":"Activity · jina-ai/mteb-long-documents"}