{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":478037973,"defaultBranch":"main","name":"mteb","ownerLogin":"embeddings-benchmark","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2022-04-05T08:25:47.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/103029531?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726814725.0","currentOid":""},"activityList":{"items":[{"before":"2aa93be190d9822f3dca03700d80162b3307fe8a","after":"15a511baf8fdadeb1e449fa5fdb452c58c52946f","ref":"refs/heads/mieb","pushedAt":"2024-09-20T13:19:59.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"gowitheflow-1998","name":"chenghao xiao","path":"/gowitheflow-1998","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/85804993?s=80&v=4"},"commit":{"message":"[mieb] Add blip and blip2 models, and ImageNetDog15Clustering task (#1226)\n\n* wip: start adding BLIP models\r\n\r\n* add other blip variants\r\n\r\n* wip: add blip2_models.py\r\n\r\n* make lint\r\n\r\n* wip: implement blip2 wrapper\r\n\r\n* feat: add blip2 models, still mismatched names\r\n\r\n* fix: remove projections from image and text embeddings\r\n\r\n* make lint\r\n\r\n* wip: add coco BLIP2\r\n\r\n* fix: BLIP2 better zero-shot classification without text_proj and vision_proj\r\n\r\n* tidy blip2\r\n\r\n* add imagenet-dog-15 dataset\r\n\r\n* tidy and lint\r\n\r\n* remove unused import\r\n\r\n* add cluster_accuracy, ari and nmi to Image.ClusteringEvaluator\r\n\r\n* add imagenet-10 clustering task\r\n\r\n* add results forclip on ImageNet10Clustering and ImageNetDog15Clustering","shortMessageHtmlLink":"[mieb] Add blip and blip2 models, and ImageNetDog15Clustering task (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2536705222\" data-permission-text=\"Title is private\" data-url=\"https://github.com/embeddings-benchmark/mteb/issues/1226\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/embeddings-benchmark/mteb/pull/1226/hovercard\" href=\"https://github.com/embeddings-benchmark/mteb/pull/1226\">#…</a>"}},{"before":"694e5639c0622af3bf90de5855c1af1ab01d86df","after":"5943979a6fee3c72cbafeec8afd3c5318a7cc04c","ref":"refs/heads/main","pushedAt":"2024-09-20T07:03:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"1.14.21\n\nAutomatically generated by python-semantic-release","shortMessageHtmlLink":"1.14.21"}},{"before":"6509a110d545498611b6c8bfc1f7715c903920d8","after":"92394ab5f3c360d6908369c65b9720a1b28ead83","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:56:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Add contributors","shortMessageHtmlLink":"Add contributors"}},{"before":"ad8fc63525bd03c1e0ce1fdc6f406f3c72eb1881","after":"6509a110d545498611b6c8bfc1f7715c903920d8","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:52:56.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Update contrib","shortMessageHtmlLink":"Update contrib"}},{"before":"bedcfb3e3991c0573aebb05add24c4f627e14f92","after":"694e5639c0622af3bf90de5855c1af1ab01d86df","ref":"refs/heads/main","pushedAt":"2024-09-20T06:48:15.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"Update points table","shortMessageHtmlLink":"Update points table"}},{"before":"20387b508bb609d4e1463b8faa93a9ab47720a01","after":"ad8fc63525bd03c1e0ce1fdc6f406f3c72eb1881","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:47:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Create 971.jsonl","shortMessageHtmlLink":"Create 971.jsonl"}},{"before":"d60933090ec4035b2cfc327804c803fd14467ca7","after":null,"ref":"refs/heads/add_repllama","pushedAt":"2024-09-20T06:45:25.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"}},{"before":"25b7a2fd1c6c1c24a48bf3c2c4d8c00dfa2820b9","after":"bedcfb3e3991c0573aebb05add24c4f627e14f92","ref":"refs/heads/main","pushedAt":"2024-09-20T06:45:24.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"fix: Add RepLLaMA style models (#1223)\n\n* init commit\r\n\r\n* working and reproducing\r\n\r\n* lint\r\n\r\n* update hashes\r\n\r\n* warning\r\n\r\n* add pyproject","shortMessageHtmlLink":"fix: Add RepLLaMA style models (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2529539281\" data-permission-text=\"Title is private\" data-url=\"https://github.com/embeddings-benchmark/mteb/issues/1223\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/embeddings-benchmark/mteb/pull/1223/hovercard\" href=\"https://github.com/embeddings-benchmark/mteb/pull/1223\">#1223</a>)"}},{"before":"2e0012273a4623d0df9b34a9eda9b8cb029f5815","after":"20387b508bb609d4e1463b8faa93a9ab47720a01","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:36:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Fix case","shortMessageHtmlLink":"Fix case"}},{"before":"12e9638f6d6bf8ea5d8416d63491e286c12190d5","after":"2e0012273a4623d0df9b34a9eda9b8cb029f5815","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:36:20.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Fix case","shortMessageHtmlLink":"Fix case"}},{"before":"e6dbc1ca783623d62389ee292839f784695f97fb","after":"12e9638f6d6bf8ea5d8416d63491e286c12190d5","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:35:39.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Fix casing","shortMessageHtmlLink":"Fix casing"}},{"before":null,"after":"e6dbc1ca783623d62389ee292839f784695f97fb","ref":"refs/heads/updatepoints","pushedAt":"2024-09-20T06:34:55.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Fix case","shortMessageHtmlLink":"Fix case"}},{"before":"8f621fad2911bdab6a9706213d19f42b169748b6","after":"d60933090ec4035b2cfc327804c803fd14467ca7","ref":"refs/heads/add_repllama","pushedAt":"2024-09-19T11:01:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"orionw","name":"Orion Weller","path":"/orionw","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/31665361?s=80&v=4"},"commit":{"message":"add pyproject","shortMessageHtmlLink":"add pyproject"}},{"before":"eee5470edd7e4b9da6015a625e53de757d62a22a","after":null,"ref":"refs/heads/Muennighoff-patch-1","pushedAt":"2024-09-18T17:32:53.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"}},{"before":"4de39ee917565c8b5da0a101c4769e5b025745e2","after":"25b7a2fd1c6c1c24a48bf3c2c4d8c00dfa2820b9","ref":"refs/heads/main","pushedAt":"2024-09-18T17:32:52.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"docs: clarify adding a model (#1222)","shortMessageHtmlLink":"docs: clarify adding a model (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2529417186\" data-permission-text=\"Title is private\" data-url=\"https://github.com/embeddings-benchmark/mteb/issues/1222\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/embeddings-benchmark/mteb/pull/1222/hovercard\" href=\"https://github.com/embeddings-benchmark/mteb/pull/1222\">#1222</a>)"}},{"before":"94ea632eb30956fbb41469a147192a362af6b6d5","after":"8f621fad2911bdab6a9706213d19f42b169748b6","ref":"refs/heads/add_repllama","pushedAt":"2024-09-17T15:10:51.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"orionw","name":"Orion Weller","path":"/orionw","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/31665361?s=80&v=4"},"commit":{"message":"warning","shortMessageHtmlLink":"warning"}},{"before":"adab31ad3abf9705cd1558985d3dee1c2a240168","after":"94ea632eb30956fbb41469a147192a362af6b6d5","ref":"refs/heads/add_repllama","pushedAt":"2024-09-17T15:07:48.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"orionw","name":"Orion Weller","path":"/orionw","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/31665361?s=80&v=4"},"commit":{"message":"update hashes","shortMessageHtmlLink":"update hashes"}},{"before":"b66aaeb0919707530a41ad983c61413bd6933a53","after":"4de39ee917565c8b5da0a101c4769e5b025745e2","ref":"refs/heads/main","pushedAt":"2024-09-17T07:21:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"Update points table","shortMessageHtmlLink":"Update points table"}},{"before":"ca21636f8234edba8ea42cdb1b92fea637b765b1","after":"b66aaeb0919707530a41ad983c61413bd6933a53","ref":"refs/heads/main","pushedAt":"2024-09-17T07:21:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"Update points table","shortMessageHtmlLink":"Update points table"}},{"before":"00260b5497b4c82583be6383d8b22a3fceb64b54","after":"ca21636f8234edba8ea42cdb1b92fea637b765b1","ref":"refs/heads/main","pushedAt":"2024-09-17T07:18:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"1.14.20\n\nAutomatically generated by python-semantic-release","shortMessageHtmlLink":"1.14.20"}},{"before":"c8e8c8552b3748147af2bd3e64ba2a2f8e66eb85","after":null,"ref":"refs/heads/add-splits-to-bench","pushedAt":"2024-09-17T07:12:00.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"}},{"before":"7fa38ceedcfb5b123417a0c0504115765485a0b1","after":"00260b5497b4c82583be6383d8b22a3fceb64b54","ref":"refs/heads/main","pushedAt":"2024-09-17T07:11:59.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"fix: Allow benchmark to specify eval_splits (#1217)\n\n* fix: Allow benchmark to specify eval_splits\r\n\r\nThis PR allow for benchmarks to specify specific eval. splits. This allow us to fully specify a benchmark within the benchmark object.\r\n\r\nTo do this it add the following:\r\n- added eval_splits to the Abstask object, which default to metadata.eval_splits\r\n- use the task.eval_splits unless overwritten in mteb.MTEB.run\r\n- added eval_splits arg to mteb.get_tasks, which filter the tasks based on splits\r\n- updated documentation\r\n  - renamed the \"Advanced Usage\" to \"Usage Documentation\" to make it more accicible\r\n- added tests where relevant\r\n\r\n* Added correction based on feedback","shortMessageHtmlLink":"fix: Allow benchmark to specify eval_splits (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2522477646\" data-permission-text=\"Title is private\" data-url=\"https://github.com/embeddings-benchmark/mteb/issues/1217\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/embeddings-benchmark/mteb/pull/1217/hovercard\" href=\"https://github.com/embeddings-benchmark/mteb/pull/1217\">#1217</a>)"}},{"before":"ba89812f76742daaa43d5a2a44411a7bd2ff3d28","after":"c8e8c8552b3748147af2bd3e64ba2a2f8e66eb85","ref":"refs/heads/add-splits-to-bench","pushedAt":"2024-09-17T07:02:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"fix error","shortMessageHtmlLink":"fix error"}},{"before":"ce9bbcdd4f09ef4ce6214604c886c92c79844d6a","after":"adab31ad3abf9705cd1558985d3dee1c2a240168","ref":"refs/heads/add_repllama","pushedAt":"2024-09-16T21:35:11.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"orionw","name":"Orion Weller","path":"/orionw","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/31665361?s=80&v=4"},"commit":{"message":"lint","shortMessageHtmlLink":"lint"}},{"before":"10a0a7757163abf60a6fd8003195f975adb6684f","after":"ce9bbcdd4f09ef4ce6214604c886c92c79844d6a","ref":"refs/heads/add_repllama","pushedAt":"2024-09-16T21:22:31.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"orionw","name":"Orion Weller","path":"/orionw","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/31665361?s=80&v=4"},"commit":{"message":"working and reproducing","shortMessageHtmlLink":"working and reproducing"}},{"before":null,"after":"eee5470edd7e4b9da6015a625e53de757d62a22a","ref":"refs/heads/Muennighoff-patch-1","pushedAt":"2024-09-16T20:26:07.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"Muennighoff","name":"Niklas Muennighoff","path":"/Muennighoff","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/62820084?s=80&v=4"},"commit":{"message":"Clarify","shortMessageHtmlLink":"Clarify"}},{"before":"4efb4109ea097a97f2863a20a1b08f88bb4fccef","after":"2aa93be190d9822f3dca03700d80162b3307fe8a","ref":"refs/heads/mieb","pushedAt":"2024-09-15T23:28:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"gowitheflow-1998","name":"chenghao xiao","path":"/gowitheflow-1998","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/85804993?s=80&v=4"},"commit":{"message":"add visual STS 12-16","shortMessageHtmlLink":"add visual STS 12-16"}},{"before":"aeeb5f34a22ef4dcbe2e35eebb4522ae2c7b2ab4","after":"ba89812f76742daaa43d5a2a44411a7bd2ff3d28","ref":"refs/heads/add-splits-to-bench","pushedAt":"2024-09-14T10:26:46.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"docs: fix incorrect listing","shortMessageHtmlLink":"docs: fix incorrect listing"}},{"before":"f82ce3699c588296c3250910548f26a2a8756e3a","after":"aeeb5f34a22ef4dcbe2e35eebb4522ae2c7b2ab4","ref":"refs/heads/add-splits-to-bench","pushedAt":"2024-09-14T10:26:02.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"Added correction based on feedback","shortMessageHtmlLink":"Added correction based on feedback"}},{"before":null,"after":"a363594dce2c07404e20d2fe086268ef7c6f5edb","ref":"refs/heads/leaderboard_dev","pushedAt":"2024-09-14T10:18:18.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"KennethEnevoldsen","name":"Kenneth Enevoldsen","path":"/KennethEnevoldsen","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/23721977?s=80&v=4"},"commit":{"message":"Add leaderboard dev","shortMessageHtmlLink":"Add leaderboard dev"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEu_06KAA","startCursor":null,"endCursor":null}},"title":"Activity · embeddings-benchmark/mteb"}