Add QDT

[takuyamagata]

Hi Takuma-san,

I have implemented QDT. As QDT is not a model but data augmentation (a data-centric approach), it is currently implemented within reproductions/offline/qdt.py.

I am not 100% convinced that it is the right place. Please let me know if you have a better idea of where it should go or how it should be structured. I am happy to change it.

Also, I appreciate any suggestions to improve the code quality!

Thanks,
Taku

[takuyamagata]

Hi Takuma-san,

Sorry, I fixed some bugs in qdt.py.

regards,
Taku

[takuseno]

@takuyamagata Thank you for making this contribution! Please let me take a look in this weekend.

[takuseno]

@takuyamagata Hi, Yamagata-san. The code looks good to me! Before merging your PR, can you fix the format error in CI? You should be able to fix this as follows:

pip install -r dev.requirements.txt
./scripts/lint

It seems that there is something wrong in the unit test. But, I can follow up on this in a separate thread. Once the format check is passed, let me merge this PR. Thanks!

[takuyamagata]

@takuseno Thank you, Takuma-san. I have run the lint script. I hope it fixed the format error.
Thanks,
Taku

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 91.74%. Comparing base (0e8097b) to head (e8f7e61).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #442   +/-   ##
=======================================
  Coverage   91.74%   91.74%           
=======================================
  Files         122      122           
  Lines        8719     8719           
=======================================
  Hits         7999     7999           
  Misses        720      720           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[takuseno]

@takuyamagata Hi Yamagata-san. It seems that there are still some format issues. Can you check this error message and resolve it?
https://github.com/takuseno/d3rlpy/actions/runs/13216850197/job/36909944022?pr=442

[takuyamagata]

@takuseno Hi Takuma-san, I have locally fixed most of the format errors, but one of the fixes causes an error in my environment.

I need the following one line to make MuJoCo environments recognised in the script. I also manually add this to other scripts in reproductions/ to make it work.
import d4rl.gym_mujoco

Could I include the line in qdt.py despite it causing a lint error? (Or is it just me have this issue??)

[takuseno]

Thank you for your work! d4rl is internally imported inside get_d4rl function:

d3rlpy/d3rlpy/datasets.py

Line 418 in aa5ede9

import d4rl # noqa

Thus, you don't have to import it by yourself in the script.

[takuseno]

@takuyamagata There are mypy errors as follows:

reproductions/offline/qdt.py:44: error: Argument 1 to "relabel_dataset_rtg" has incompatible type "BufferProtocol"; expected "InfiniteBuffer"  [arg-type]
reproductions/offline/qdt.py:44: error: Argument 2 to "relabel_dataset_rtg" has incompatible type "LearnableBase[ImplBase, LearnableConfig]"; expected "Union[CQL, IQL]"  [arg-type]
reproductions/offline/qdt.py:58: error: Function is missing a return type annotation  [no-untyped-def]

Can you make sure that scripts/lint is passed without errors in your local environment?

[takuyamagata]

@takuseno I have fixed all format issues (including type check issues). Hope it is all good now.

Regarding d4rl.gym_mujoco, I see why I get the error. I haven't set up other MuJoCo-based environments correctly, so d4rl.init.py fails to call 'import d4rl.gym_mujoco'. So it is my problem. :(

[takuseno]

Thank you for the changes! LGTM.

[takuseno]

In a follow-up commit, the script has been a little refactored:
https://github.com/takuseno/d3rlpy/blob/master/reproductions/offline/qdt.py