Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add unittest for ray text dedup #540

Merged
merged 21 commits into from
Jan 23, 2025
Merged

Add unittest for ray text dedup #540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
eadf93b
add actors for ray dedup
chenyushuo Jan 2, 2025
051493b
code format
chenyushuo Jan 3, 2025
4fc12a6
doc format
chenyushuo Jan 3, 2025
a6fcd6e
fix in ray dedup
chenyushuo Jan 7, 2025
891219f
Merge branch 'main' of github.com:modelscope/data-juicer into ray_ded…
chenyushuo Jan 9, 2025
f475338
Merge branch 'main' of github.com:modelscope/data-juicer into ray_ded…
chenyushuo Jan 10, 2025
06b1730
add unittest for ray text dedup
chenyushuo Jan 10, 2025
35cf0e7
add TEST_TAG ray and operator level test
chenyushuo Jan 13, 2025
a0f72fb
add `test_ray_minhash_dedup`, `test_ray_compute_stats_single_filter` …
chenyushuo Jan 13, 2025
218742c
add TEST_TAG for RayBTSDedup
chenyushuo Jan 17, 2025
9e94c19
Merge branch 'main' of github.com:modelscope/data-juicer into ray_ded…
chenyushuo Jan 17, 2025
ecc4c0a
doc fix
chenyushuo Jan 17, 2025
24a8640
fix in bts test
chenyushuo Jan 20, 2025
faec9f2
Merge branch 'ray_dedup_with_actor' of github.com:chenyushuo/data-jui…
chenyushuo Jan 20, 2025
46ba4dd
merge
chenyushuo Jan 20, 2025
8cac8b7
unittest test
chenyushuo Jan 21, 2025
b4dedbc
unittest test
chenyushuo Jan 21, 2025
79ce0ab
unittest test
chenyushuo Jan 21, 2025
85fe608
unittest test
chenyushuo Jan 21, 2025
b283767
unittest test
chenyushuo Jan 22, 2025
0c6787e
unittest test
chenyushuo Jan 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs/Operators.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,10 +78,10 @@ All the specific operators are listed below, each featured with several capabili
| document_simhash_deduplicator | 🔤Text 💻CPU 🟢Stable | Deduplicator to deduplicate samples at document-level using SimHash. Deduplicator使用SimHash在文档级别对样本进行重复数据删除。 | [code](../data_juicer/ops/deduplicator/document_simhash_deduplicator.py) | [tests](../tests/ops/deduplicator/test_document_simhash_deduplicator.py) |
| image_deduplicator | 🏞Image 💻CPU 🟢Stable | Deduplicator to deduplicate samples at document-level using exact matching of images between documents. Deduplicator使用文档之间的图像精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/image_deduplicator.py) | [tests](../tests/ops/deduplicator/test_image_deduplicator.py) |
| ray_basic_deduplicator | 💻CPU 🔴Alpha | Backend for deduplicator. deduplicator的后端。 | [code](../data_juicer/ops/deduplicator/ray_basic_deduplicator.py) | - |
| ray_bts_minhash_deduplicator | 🔤Text 💻CPU 🔴Alpha | A distributed implementation of Union-Find with load balancing. 具有负载平衡的Union-Find的分布式实现。 | [code](../data_juicer/ops/deduplicator/ray_bts_minhash_deduplicator.py) | - |
| ray_document_deduplicator | 🔤Text 💻CPU 🔴Alpha | Deduplicator to deduplicate samples at document-level using exact matching. Deduplicator使用精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_document_deduplicator.py) | - |
| ray_image_deduplicator | 🏞Image 💻CPU 🔴Alpha | Deduplicator to deduplicate samples at document-level using exact matching of images between documents. Deduplicator使用文档之间的图像精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_image_deduplicator.py) | - |
| ray_video_deduplicator | 🎬Video 💻CPU 🔴Alpha | Deduplicator to deduplicate samples at document-level using exact matching of videos between documents. Deduplicator使用文档之间的视频精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_video_deduplicator.py) | - |
| ray_bts_minhash_deduplicator | 🔤Text 💻CPU 🟡Beta | A distributed implementation of Union-Find with load balancing. 具有负载平衡的Union-Find的分布式实现。 | [code](../data_juicer/ops/deduplicator/ray_bts_minhash_deduplicator.py) | [tests](../tests/ops/deduplicator/test_ray_bts_minhash_deduplicator.py) |
| ray_document_deduplicator | 🔤Text 💻CPU 🟡Beta | Deduplicator to deduplicate samples at document-level using exact matching. Deduplicator使用精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_document_deduplicator.py) | [tests](../tests/ops/deduplicator/test_ray_document_deduplicator.py) |
| ray_image_deduplicator | 🏞Image 💻CPU 🟡Beta | Deduplicator to deduplicate samples at document-level using exact matching of images between documents. Deduplicator使用文档之间的图像精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_image_deduplicator.py) | [tests](../tests/ops/deduplicator/test_ray_image_deduplicator.py) |
| ray_video_deduplicator | 🎬Video 💻CPU 🟡Beta | Deduplicator to deduplicate samples at document-level using exact matching of videos between documents. Deduplicator使用文档之间的视频精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/ray_video_deduplicator.py) | [tests](../tests/ops/deduplicator/test_ray_video_deduplicator.py) |
| video_deduplicator | 🎬Video 💻CPU 🟢Stable | Deduplicator to deduplicate samples at document-level using exact matching of videos between documents. Deduplicator使用文档之间的视频精确匹配在文档级别删除重复的样本。 | [code](../data_juicer/ops/deduplicator/video_deduplicator.py) | [tests](../tests/ops/deduplicator/test_video_deduplicator.py) |

## filter <a name="filter"/>
Expand Down
Loading
Loading