Fault experience

[NDR008]

I just understood the meaning of this:

          # If failure happens for too many steps, the episode terminates
          if self.step_counter > self.min_nb_steps_before_failure:
              self.failure_counter += 1
              if self.failure_counter > self.nb_zero_rew_before_failure:
                  terminated = True

Question: Would bad experience still not be useful experience?
Is there a good reason to terminate early?
If the reward were 0 (or as I did in my case - negative) would the NN still not be able to learn something useful from it?

[yannbouteiller]

It is really just a hack to avoid the situation where the car starts driving in the wrong direction in TrackMania.

The problem with this would be that, since we don't provide a signal telling the car that it is going in the wrong direction, the model wouldn't understand why its reward is suddenly bad, especially in the "lidar" environment.

[NDR008]

Why not let the agent see the Delta checkpoints instead then?
That is almost like a speed that can go negative.

[yannbouteiller]

Because this is not something available in the general real-world scenario (for which we decided to use no "GPS-like" observation as we focus on vision), so we don't include that in the observation space.

The tmrl default environment is mostly concerned with vision-based autonomous control in the real-world robot context, not with hacking into video-games internals. For TrackMania, this means playing in real-time by looking at the screen.