New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

如何把1T的数据处理成pretrain需要的格式 #56

Open

Kathrine94 opened this issue Jul 7, 2023 · 1 comment

Kathrine94 commented Jul 7, 2023

我已经拿到了1T的训练数据，如何把这个数据处理成预训练模型时所需要的输入的形式？

Contributor

zhaoxudong01-ieisystem commented Jul 10, 2023

我已经拿到了1T的训练数据，如何把这个数据处理成预训练模型时所需要的输入的形式？

python tools/preprocess_data_cn.py --input txt_data_path --output_path output_path --vocab_path ./vocab.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment