Dedup methods

[ch-shin]

Hi team!

It seems Exact Dedup, MinHash, Suffix Array, and Bloom Filter are studied in the dedup ablation experiments. I have a couple of questions related to these:

• Where can I find the code for the Suffix Array?
• Are there any examples of how to use MinHash and Bloom Filter? I see that their code is uploaded 2 weeks ago, but I'm unsure how to integrate them into the process code.

Thank you!

[GeorgiosSmyrnis]

Hi @ch-shin ,

Apologies for the late response!

• Our suffix array deduplication is based on the implementation from Lee et al., you can find the original implementation here: https://github.com/google-research/deduplicate-text-datasets
• Currently we recommend running deduplication as a separate stage from the process code, by following the respective instructions e.g. for BFF: https://github.com/mlfoundations/dclm/tree/main/dedup/bff

Let us know if the above are helpful!