Trouble with Arabic in F5-TTS and Sample Outputs

[P-uck]

Checks

• This template is only for question, not feature requests or bug reports.
• I have thoroughly reviewed the project documentation and read the related paper(s).
• I have searched for existing issues, including closed ones, no similar questions.
• I confirm that I am using English to submit this report in order to facilitate communication.

Question details

Hi everyone,

I’ve been working with F5-TTS for the past week to fine-tune it on Arabic lang. My dataset is around 5 hours of very clean, diverse, single-speaker. my GPU is 4090 RTX and I used around 1000 epoches for this,

The problem is that during training, after about 50% of the epochs pass, the sample outputs in ckpts/Arabic_new/samples/ start sounding good—almost identical to the base data. But when the training is done, and I try generating audio using the final checkpoints, the output becomes pretty much unintelligible. The tone and style are fine, but you can’t understand the words at all. I’ve tried both the Gradio interface and CLI tools with the same checkpoints, but no luck—they both give me the same garbled output.

I check the code, specially how it generates these sample outputs during training. from what I see it uses intermediate checkpoints to generate mel-spectrograms for the given gen_text and then passes those through the vocoder.
So, if the samples during training are generated like this, why can’t I recreate the same quality when I use the final checkpoints? Am I missing something obvious here?

Also, can I learn F5-TTS for Arabic with its complex phonemes and vocabulary using only 5 hours of dataset? Has anyone tried something similar and had success? Maybe there are some specific settings I should tweak?

Thanks

[SWivid]

Hi @P-uck

Try turn off use_ema if using an early-stage finetuned checkpoint (which goes just few updates).

uses intermediate checkpoints to generate mel-spectrograms for the given gen_text

the sample saved with training is not using ema

you could turn off when inference (or in infer_gradio/infer_cli pass in use_ema=False):

F5-TTS/src/f5_tts/infer/utils_infer.py

Lines 223 to 232 in f992c4e

	def load_model(
	model_cls,
	model_cfg,
	ckpt_path,
	mel_spec_type=mel_spec_type,
	vocab_file="",
	ode_method=ode_method,
	use_ema=True,
	device=device,
	):

[ftk789]

Would you be kind in sharing the FineTuned model of the Arabic language?

[P-uck]

Hi @SWivid,

Thank you for your response.

Now I got a clean dataset with 15 hours of data, but I’ve noticed my 4090 GPU is running quite slow. I’m wondering how I can set up multiple GPUs on an online platform like Vast ai. I tried using the Accelerator, but it didn’t work because all the GPUs are on a single workstation.

@ftk789, I absolutely will but, I need to make some progress first because there are currently many issues with handling Arabic diacritics (Tashkil).

[ftk789]

@ftk789, I absolutely will but, I need to make some progress first because there are currently many issues with handling Arabic diacritics (Tashkil).

That's so kind of you @P-uck . Thank you in advance. It'd be nice to see what kind of dataset are you training on and how good will it sound. but thank you in advance!

[P-uck]

@SWivid
update:
I tired to run on 2* 4090 GPUs, but it kind of freez:
i tired also conf.yaml for hydra but no luck

`run command :
accelerate launch --mixed_precision=fp16 /workspace/F5-TTS/src/f5_tts/train/finetune_cli.py --exp_name F5TTS_Base --learning_rate 7.5e-05 --batch_size_per_gpu 6000 --batch_size_type frame --max_samples 64 --grad_accumulation_steps 1 --max_grad_norm 1 --epochs 1742 --num_warmup_updates 188 --save_per_updates 376 --keep_last_n_checkpoints -1 --last_per_updates 94 --dataset_name ahmad4090 --tokenizer char --log_samples --logger wandb --bnb_optimizer

The following values were not passed to accelerate launch and had defaults used instead:
--num_processes was set to a value of 2
More than one GPU was found, enabling multi-GPU training.
If this was unintended please pass in --num_processes=1.
--num_machines was set to a value of 1
--dynamo_backend was set to a value of 'no'
To avoid this warning pass in values for each of the problematic parameters or run accelerate config.
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Building prefix dict from the default dictionary ...
Loading model from cache /tmp/jieba.cache
Word segmentation module jieba initialized.

Loading model cost 0.535 seconds.
Prefix dict has been built successfully.
Word segmentation module jieba initialized.

Loading model cost 0.535 seconds.
Prefix dict has been built successfully.

vocab : 2587

vocoder : vocos

vocab : 2587

vocoder : vocos
Using logger: None
Loading dataset ...
Loading dataset ...
/opt/conda/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/opt/conda/lib/python3.10/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpeg or libpng installed before building torchvision from source?
warn(
/opt/conda/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: '/opt/conda/lib/python3.10/site-packages/torchvision/image.so: undefined symbol: _ZN3c1017RegisterOperatorsD1Ev'If you don't plan on using image functionality from torchvision.io, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have libjpeg or libpng installed before building torchvision from source?
warn(
Download Vocos from huggingface charactr/vocos-mel-24khz
Download Vocos from huggingface charactr/vocos-mel-24khz

Sorting with sampler... if slow, check whether dataset is provided with duration: 0%| | 0/3764 [00:00<?, ?it/s]
Sorting with sampler... if slow, check whether dataset is provided with duration: 100%|█████████████████████████████████████| 3764/3764 [00:00<00:00, 1664806.52it/s]

Creating dynamic batches with 6000 audio frames per gpu: 0%| | 0/3764 [00:00<?, ?it/s]
Creating dynamic batches with 6000 audio frames per gpu: 100%|██████████████████████████████████████████████████████████████| 3764/3764 [00:00<00:00, 2439709.51it/s]

Sorting with sampler... if slow, check whether dataset is provided with duration: 0%| | 0/3764 [00:00<?, ?it/s]
Sorting with sampler... if slow, check whether dataset is provided with duration: 100%|█████████████████████████████████████| 3764/3764 [00:00<00:00, 1648121.96it/s]

Creating dynamic batches with 6000 audio frames per gpu: 0%| | 0/3764 [00:00<?, ?it/s]
Creating dynamic batches with 6000 audio frames per gpu: 100%|██████████████████████████████████████████████████████████████| 3764/3764 [00:00<00:00, 2840475.04it/s]

Epoch 1/1742: 0%| | 0/162 [00:00<?, ?update/s][rank1]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank0]:[W reducer.cpp:1389] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

Epoch 1/1742: 1%|▋ | 1/162 [00:01<04:50, 1.80s/update]
`