-
Notifications
You must be signed in to change notification settings - Fork 240
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Be able to cache embeddings and load them #946
Comments
Hey @orionw , I would like to try this if possible! |
Awesome @tenzu15! It would be great to be able to pass the two flags in the This would need to be changed in the RetrievalEvaluator class for now. If it's useful for other tasks, we can implement it there also. Also cc'ing @KennethEnevoldsen who may have opinions on where this should be added/what the names should be. But feel free to start @tenzu15. If you have any questions feel free to make a draft PR and cc me! |
@orionw, wouldn't it be better to implement a more general model wrapping for this so that it works for all tasks?
|
There's some background discussion related to the topic from #354 (comment) as well. |
+1 @KennethEnevoldsen, I think a wrapper is a great idea and even simpler to implement. |
It sounds like we settled on a wrapper here. In which case I don't think it is something that should be within |
Personally, I think it’d be nice to have it be full functionality in MTEB so you can cache things. Maybe it’s just my research but not having to recompute the embeddings would save a lot of time and I frequently store them with Pyserini instead. If this was in MTEB it would also allow us to put the indexes on HF so people can just grab and use it. If no one else finds it useful we can leave it unimplemented but I personally would find it very useful. |
Will leave this open then. Def. think public caches are important - The Scandinavian embedding benchmark implements it for results. Is there a reason why such an approach would not work here? debugging, error analysis I presume? |
Thanks! I wasn't aware of the Scandinavian embedding benchmark cache - do you mind linking? |
cache for results is here: https://github.com/KennethEnevoldsen/scandinavian-embedding-benchmark/tree/main/src/seb/cache It is simply implemented as a part of the package. If you try to rerun an already run model it will simply use the cache. |
These are the cached results right? I don't see any embeddings but maybe missed them |
Yeah only results so no embeddings |
For most users, being able to cache their embedded docs and/or provide a cached embedding file is probably overkill.
However, there are many situations where it would be helpful to have an option to cache them. For example, experiments where you alter the query/document set to for speedups (as I'm doing now) or if you're testing the effect of different prefixes/instructions over the same dataset.
I typically use
pyserini
for caching the index so that we can quickly search over it later, but that doesn't integrate nicely withmteb
. I think it would fairly straightforward to implement this: (1) we need to take in a flag of whether to cache the embeddings, cache them to a file that corresponds to the dataset and model name and (2) provide an option to read in a cached embedding file.I don't have bandwidth for this right now, but if anyone does it would be an excellent addition.
The text was updated successfully, but these errors were encountered: