Univnet Gradient Error

[francois-vz]

Hi there! First, thanks for creating such an awesome all-in-one TTS toolkit. This is my first TTS project and I have been using Coqui's TTS for approximately three weeks. I am trying to fine-tune a Tacotron2 model to 4 hours of semi-clean Afrikaans data, as well as train a Univnet from scratch on the same data. I am experiencing some issues with the vocoder training and I would just like to further discuss that with somebody more experienced :) . I use the following command when training the unvinet model (python3.8 in coqui conda environment created from the git repo [origin/dev]):

$ cd ~/work/coqui/TTS/TTS/bin
$ python3 train_vocoder.py --config_path "./config/univnet.json" 

where univnet.json is given by

{
    "output_path": "/home/francois/work/coqui/runs/univnet",
    "logger_uri": null,
    "run_name": "univnet-from-scratch",
    "project_name": null,
    "run_description": "",
    "print_step": 25,
    "plot_step": 100,
    "model_param_stats": false,
    "wandb_entity": null,
    "dashboard_logger": "tensorboard",
    "log_model_step": null,
    "save_step": 10000,
    "save_n_checkpoints": 15,
    "save_checkpoints": true,
    "save_all_best": false,
    "save_best_after": 10000,
    "target_loss": "loss_0",
    "print_eval": false,
    "test_delay_epochs": -1,
    "run_eval": false,
    "run_eval_steps": null,
    "distributed_backend": "nccl",
    "distributed_url": "tcp://localhost:54321",
    "mixed_precision": false,
    "epochs": 5000,
    "batch_size": 64,
    "eval_batch_size": 16,
    "grad_clip": null,
    "scheduler_after_epoch": false,
    "lr": 0.001,
    "optimizer": "AdamW",
    "optimizer_params": {
        "betas": [
            0.5,
            0.9
        ],
        "weight_decay": 0.0
    },
    "lr_scheduler": null,
    "lr_scheduler_params": {},
    "use_grad_scaler": false,
    "cudnn_enable": true,
    "cudnn_deterministic": false,
    "cudnn_benchmark": false,
    "training_seed": 54321,
    "model": "univnet",
    "num_loader_workers": 4,
    "num_eval_loader_workers": 4,
    "use_noise_augment": true,
    "audio": {
        "fft_size": 1024,
        "win_length": 1024,
        "hop_length": 256,
        "frame_shift_ms": null,
        "frame_length_ms": null,
        "stft_pad_mode": "reflect",
        "sample_rate": 22050,
        "resample": false,
        "preemphasis": 0.0,
        "ref_level_db": 20,
        "do_sound_norm": false,
        "log_func": "np.log10",
        "do_trim_silence": true,
        "trim_db": 60,
        "do_rms_norm": false,
        "db_level": null,
        "power": 1.5,
        "griffin_lim_iters": 60,
        "num_mels": 80,
        "mel_fmin": 50.0,
        "mel_fmax": 7600.0,
        "spec_gain": 1,
        "do_amp_to_db_linear": true,
        "do_amp_to_db_mel": true,
        "pitch_fmax": 640.0,
        "pitch_fmin": 0,
        "signal_norm": true,
        "min_level_db": -100,
        "symmetric_norm": true,
        "max_norm": 4.0,
        "clip_norm": true,
        "stats_path": "/home/francois/.local/share/tts/vocoder_models--en--ljspeech--univnet/scale_stats.npy"
    },
    "eval_split_size": 10,
    "data_path": "/home/francois/work/coqui/data/afr_final_corpus/wavs",
    "feature_path": null,
    "seq_len": 8192,
    "pad_short": 2000,
    "conv_pad": 0,
    "use_cache": false,
    "wd": 0.0,
    "use_stft_loss": true,
    "use_subband_stft_loss": false,
    "use_mse_gan_loss": true,
    "use_hinge_gan_loss": false,
    "use_feat_match_loss": false,
    "use_l1_spec_loss": false,
    "stft_loss_weight": 2.5,
    "subband_stft_loss_weight": 0.0,
    "mse_G_loss_weight": 1.0,
    "hinge_G_loss_weight": 0.0,
    "feat_match_loss_weight": 0.0,
    "l1_spec_loss_weight": 0.0,
    "stft_loss_params": {
        "n_ffts": [
            1024,
            2048,
            512
        ],
        "hop_lengths": [
            120,
            240,
            50
        ],
        "win_lengths": [
            600,
            1200,
            240
        ]
    },
    "l1_spec_loss_params": {
        "use_mel": true,
        "sample_rate": 22050,
        "n_fft": 1024,
        "hop_length": 256,
        "win_length": 1024,
        "n_mels": 80,
        "mel_fmin": 0.0,
        "mel_fmax": null
    },
    "lr_gen": 0.0001,
    "lr_disc": 0.0001,
    "lr_scheduler_gen": null,
    "lr_scheduler_gen_params": {
        "gamma": 0.999,
        "last_epoch": -1
    },
    "lr_scheduler_disc": null,
    "lr_scheduler_disc_params": {
        "gamma": 0.999,
        "last_epoch": -1
    },
    "use_pqmf": false,
    "diff_samples_for_G_and_D": false,
    "discriminator_model": "univnet_discriminator",
    "generator_model": "univnet_generator",
    "generator_model_params": {
        "in_channels": 64,
        "out_channels": 1,
        "hidden_channels": 32,
        "cond_channels": 80,
        "upsample_factors": [
            8,
            8,
            4
        ],
        "lvc_layers_each_block": 4,
        "lvc_kernel_size": 3,
        "kpnet_hidden_channels": 64,
        "kpnet_conv_size": 3,
        "dropout": 0.0
    },
    "steps_to_start_discriminator": 100000
}

I based my config file on the default LJSpeech Univnet config file and just added some extra epochs and pointed to my own data [ Please ignore my poor checkpointing practice :) ]. I performed a few checks and I think my data is healthy with no empty or corrupted audio files and weird sample rates. When executing the script, it fails at the first epoch of training with the following error

RuntimeError: element 0 of variables does not require grad and does not have a grad_fn

I read up a bit about this error, and I found the following link. According to the discussion, the error is due to tensor variables created with requires_grad=False. I modified the script at

/home/francois/miniconda3/envs/coqui/lib/python3.8/site-packages/trainer/trainer.py

and changed the line

param.requires_grad = False

to

param.requires_grad = True

in the following function in the trainer.py script at lines 1127-1154

def toggle_optimizer(self, optimizer: torch.optim.Optimizer) -> None:
        """Makes sure only the gradients of the current optimizer's parameters are calculated in the training step
        to prevent dangling gradients in multiple-optimizer setup.
        It is especially useful for GAN training where the discriminator and generator can leak gradients with multiple
        optimizers.
        Args:
            optimizer: The optimizer to toggle.
        """
        # Iterate over all optimizer parameters to preserve their requires_grad information
        # in case these are pre-defined during c
        param_requires_grad_state = {}
        for opt in self.optimizer:
            for group in opt.param_groups:
                for param in group["params"]:
                    # If a param already appear in param_requires_grad_state, continue
                    if param in param_requires_grad_state:
                        continue
                    param_requires_grad_state[param] = param.requires_grad
                    param.requires_grad = True # Note: The default value is False

        # Then iterate over the current optimizer's parameters and set its requires_grad
        # properties accordingly
        for group in optimizer.param_groups:  # type: ignore[union-attr]
            for param in group["params"]:
                param.requires_grad = param_requires_grad_state[param]
        self._param_requires_grad_state = param_requires_grad_state  # pylint: disable=attribute-defined-outside-init

It appears that the model is training correctly and the eval loss is decreasing as expected. Can somebody please explain to me what the purpose of param.requires_grad = False is. The function docstring is not as clear of an explanation as I would like. Also, can I expect this change to deteriorate the performance of the vocoder?

[erogol]

If you reinstall TTS or update the trainer, it should be fixed

[francois-vz]

Okay, thanks :)

[francois-vz]

Heads up to anybody else who gets this problem, I used the following to install coqui

$ conda create -n coqui python=3.8
$ cd TTS 
$ git status # On branch dev
$ make system-deps
$ make install