Experiments on Mania's dataset

[dtch1997]

Summary of changes:

• Make URDF configurable, add Mania's URDF
• Add discriminator temperature config option
• Change sigma to be learnable by default (I may end up reverting this)
• Increase default kP, kD to match simulation

[dtch1997]

According to VIPER, simply training a likelihood-based model as opposed to an adversarial model may improve performance. https://arxiv.org/abs/2305.14343
What that would look like for our repo: Instead of predicting binary logit for real / fake, predict a Gaussian distribution and maximize likelihood on dataset examples.

[dtch1997]

The results of experiment are actually somewhat confusing. Refer to this run: the demo logit is high (about +5) while the agent logit is low (-15); we would expect that agent receives essentially no reward from this, and yet the average reward is quite good.

Edit: I think I understand why now, the rewards/frame is actually calculating the task reward not the AMP reward. So it doesn't sync.

In this light, the reason why policy isn't learning, may be that the reward provided by discriminator is 'too sparse'. So we should actually decrease the temperature