fix: updates sorting logic to correctly compare numbers

[RobotSail]

This PR fixes a bug currently present in the checkpoint selection logic,
where the checkpoint names are sorted by string, e.g.:

lst = ["samples_1", "samples_10","samples_19", "samples_2", "samples_9999"]
print(sorted(lst))
> ['samples_1', 'samples_10', 'samples_19', 'samples_2', 'samples_9999']

This PR fixes this issue by specifying the key parameter such that it now extracts the number at the end and uses it for comparison:

sorted(lst, key=lambda x: int(x.split("_")[-1]), reverse=True)
> ['samples_9999', 'samples_19', 'samples_10', 'samples_2', 'samples_1']

Fixes #236

Signed-off-by: Oleg S [email protected]

[Maxusmusti]

LGTM, tested save and loading to make sure expected ckpt was picked

[JamesKunstle]

LGTM- in the future we could stat the folders and take the youngest but idk if that's really valuable.