Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chunkize index[query] for large queries #117

Open
piskvorky opened this issue Jul 14, 2013 · 0 comments
Open

chunkize index[query] for large queries #117

piskvorky opened this issue Jul 14, 2013 · 0 comments
Labels
difficulty medium Medium issue: required good gensim understanding & python skills feature Issue described a new feature wishlist Feature request

Comments

@piskvorky
Copy link
Owner

... the same way chunking of the index itself is done, in for doc in index: aka SimilarityABC.__iter__.

In other words, don't construct the #query_docs x #index_docs result matrix at once, but chunkize query_docs and yield the result rows iteratively, in SimilarityABC.__getitem__.

Test thoroughly, because some code may depend on __getitem__ returning a matrix, not a generator.

@tmylk tmylk added the difficulty medium Medium issue: required good gensim understanding & python skills label Jan 23, 2016
@menshikh-iv menshikh-iv added the feature Issue described a new feature label Oct 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
difficulty medium Medium issue: required good gensim understanding & python skills feature Issue described a new feature wishlist Feature request
Projects
None yet
Development

No branches or pull requests

3 participants