Training collapses after 72k iterations

[Lauenburg]

I am currently trying to test the model on the maps data set (satellite and card images from google maps). The training went well until the 72k iteration. Afterward, the training collapses or rather no result is returned:

Images for A2B until 72000 iterations:

Images for A2B after 72000 iterations:

Images for B2A until 72000 iterations:

Images for B2A until 72000 iterations:

Any idea what could have happened here?

[Q-Zhang98]

I am currently trying to test the model on the maps data set (satellite and card images from google maps). The training went well until the 72k iteration. Afterward, the training collapses or rather no result is returned:

Images for A2B until 72000 iterations:

Images for A2B after 72000 iterations:

Images for B2A until 72000 iterations:

Images for B2A until 72000 iterations:

Any idea what could have happened here?

Hello, I have the same problem. How did you solve it?

[alpc91]

It may be that there is a phenomenon of model collapse. The training schedule can be adjusted according to the dataset, and the checkpoint can be saved for manual inspection or FID for model selection.