Skip to content

Issues: embeddings-benchmark/mteb

MMTEB Overview
#360 by KennethEnevoldsen was closed Sep 9, 2024
Closed 5
Finalizing MMTEB
#784 opened May 22, 2024 by KennethEnevoldsen
Open 6
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

61 Open 332 Closed
61 Open 332 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

PreTrainedTokenizerFast._batch_encode_plus() got an unexpected keyword argument 'prompt_name'
#1224 opened Sep 18, 2024 by violenil
Cross-Encoder Support for Reranking Abstract Task
#1214 opened Sep 11, 2024 by orionw
Figure out an approach for adding experiments to the leaderboard
#1211 opened Sep 9, 2024 by KennethEnevoldsen
4
Add benchmark overview table documentation Improvements or additions to documentation good first issue Good for newcomers
#1209 opened Sep 9, 2024 by KennethEnevoldsen
2
Evaluating only on English question Further information is requested
#1205 opened Sep 8, 2024 by stephantul
2
CrossEncoder object has no attribute model_card_data
#1192 opened Sep 3, 2024 by isaac-chung
Re-add CQADupstack merge script / find other way to make merging easier
#1171 opened Aug 20, 2024 by Muennighoff
Add updated Touche-2020 enhancement New feature or request good first issue Good for newcomers
#1170 opened Aug 20, 2024 by orionw
Add CRAG good first issue Good for newcomers new-dataset
#1151 opened Aug 11, 2024 by Muennighoff
[New dataset request] Please add MKQA good first issue Good for newcomers new-dataset
#1149 opened Aug 11, 2024 by PrithivirajDamodaran
8
Evaluators ambiguity: class/module
#1124 opened Jul 30, 2024 by r0mer0m v2.0.0
4
Solution for transforming retrieval datasets into parquet
#1090 opened Jul 15, 2024 by gowitheflow-1998
1
Distinguishing instructions for RAR-b
#1066 opened Jul 10, 2024 by Muennighoff
9
Implements check on existing and new datasets enhancement New feature or request
#1049 opened Jul 5, 2024 by KennethEnevoldsen
1
Implements a custom for documents to make data format consistent enhancement New feature or request
#1047 opened Jul 5, 2024 by KennethEnevoldsen
Update prints to use a consistent format: good first issue Good for newcomers help wanted Extra attention is needed
#1046 opened Jul 5, 2024 by KennethEnevoldsen
2
Leaks and duplications in the MTEB leaderboard bug Something isn't working
#1036 opened Jul 3, 2024 by lbourdois
10
Avoid try except when raise_errors = True
#1035 opened Jul 3, 2024 by KennethEnevoldsen
2
Confusion re: Retrieval w/Instructions leaderboard issues related to the leaderboard
#1013 opened Jun 30, 2024 by Muennighoff
6
Paper Writing: Speeding up the benchmark [waiting for retrieval]
#1007 opened Jun 28, 2024 by KennethEnevoldsen
@KennethEnevoldsen @orionw @isaac-chung
7
extracting embedding from other than CLS token during eval mteb
#999 opened Jun 28, 2024 by riyajatar37003
2
Option to remove cached dataset files on large runs enhancement New feature or request
#984 opened Jun 25, 2024 by isaac-chung
2
Be able to cache embeddings and load them enhancement New feature or request good first issue Good for newcomers
#946 opened Jun 17, 2024 by orionw
12
Allow for model specific similarity in Bitext Mining and Retrieval enhancement New feature or request
#943 opened Jun 17, 2024 by KennethEnevoldsen
Avoid using global seeds enhancement New feature or request
#942 opened Jun 17, 2024 by KennethEnevoldsen
1
ProTip! Exclude everything labeled bug with -label:bug.