Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add PLAID indexing #72

Open
NohTow opened this issue Nov 27, 2024 · 0 comments
Open

Add PLAID indexing #72

NohTow opened this issue Nov 27, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@NohTow
Copy link
Collaborator

NohTow commented Nov 27, 2024

The current Voyager HNSW index works great but does not scale, which limits the usage and the tests on small databases.
To allow evaluation on all the BEIR datasets and other larger DB, we need to implement PLAID.

I started building it on this branch.
Right now, it'll mostly be a wrapper of the stanford-nlp implementation to work with input embeddings instead of computing them in the code, enabling it to be model agnostic and so to work with PyLate (but also ColPali for that matter).

First step was to make it work with all the stanford-nlp embedded, which is now done.
Now I have to make sure every parameter are correctly plugged and clean up the plugging in general.
After that, we'll clean up unused code from stanford-nlp and it should be good to go.

The whole processing will be somewhat very black-boxy compared to the rest of the codebase, but recoding PLAID from the ground up would require way more time (and will actually be easier with this base), so I think it is a good target for a v1 allowing people to use PLAID.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant