Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error occurred in hard negative examples mining #1339

Open
LLLLucensus opened this issue Jan 17, 2025 · 1 comment
Open

Error occurred in hard negative examples mining #1339

LLLLucensus opened this issue Jan 17, 2025 · 1 comment

Comments

@LLLLucensus
Copy link

LLLLucensus commented Jan 17, 2025

已安装faiss-gpu-cu11,遇到以下报错何解?感谢~~
机器:4*v100 16G
运行命令:
python hn_mine.py
--input_file toy_finetune_data.jsonl
--output_file toy_finetune_data_minedHN.jsonl
--range_for_sampling 5-8
--negative_number 2
--use_gpu_for_searching
--embedder_name_or_path ../../BAAI/bge-m3

inferencing embedding for corpus (number=80)--------------
initial target device: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
Chunks: 0%| | 0/4 [00:00<?, ?it/s]
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
inferencing embedding for queries (number=10)--------------
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 67.19it/s]
create index and search------------------
Traceback (most recent call last):
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 233, in
main(data_args, model_args)
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 214, in main
find_knn_neg(
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 164, in find_knn_neg
index = create_index(p_vecs, use_gpu=use_gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 96, in create_index
index = faiss.IndexFlatIP(len(embeddings[0]))
~~~~~~~~~~^^^
KeyError: 0

@LLLLucensus LLLLucensus changed the title bge-multilingual-gemma2微调OOM Error occurred in hard negative examples mining Jan 17, 2025
@hanhainebula
Copy link
Collaborator

你好,@LLLLucensus。感谢指出这个 bug,现已修复:#1340

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants