You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
inferencing embedding for corpus (number=80)--------------
initial target device: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
Chunks: 0%| | 0/4 [00:00<?, ?it/s]
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
inferencing embedding for queries (number=10)--------------
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 67.19it/s]
create index and search------------------
Traceback (most recent call last):
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 233, in
main(data_args, model_args)
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 214, in main
find_knn_neg(
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 164, in find_knn_neg
index = create_index(p_vecs, use_gpu=use_gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 96, in create_index
index = faiss.IndexFlatIP(len(embeddings[0]))
~~~~~~~~~~^^^
KeyError: 0
The text was updated successfully, but these errors were encountered:
LLLLucensus
changed the title
bge-multilingual-gemma2微调OOM
Error occurred in hard negative examples mining
Jan 17, 2025
已安装faiss-gpu-cu11,遇到以下报错何解?感谢~~
机器:4*v100 16G
运行命令:
python hn_mine.py
--input_file toy_finetune_data.jsonl
--output_file toy_finetune_data_minedHN.jsonl
--range_for_sampling 5-8
--negative_number 2
--use_gpu_for_searching
--embedder_name_or_path ../../BAAI/bge-m3
inferencing embedding for corpus (number=80)--------------
initial target device: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
Chunks: 0%| | 0/4 [00:00<?, ?it/s]
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:13<00:00, 3.31s/it]
inferencing embedding for queries (number=10)--------------
Chunks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 67.19it/s]
create index and search------------------
Traceback (most recent call last):
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 233, in
main(data_args, model_args)
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 214, in main
find_knn_neg(
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 164, in find_knn_neg
index = create_index(p_vecs, use_gpu=use_gpu)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/myspace/FlagEmbedding/scripts/hn_mine.py", line 96, in create_index
index = faiss.IndexFlatIP(len(embeddings[0]))
~~~~~~~~~~^^^
KeyError: 0
The text was updated successfully, but these errors were encountered: