pre-commit and infinite length horizons documentation (HumanCompatibl…

…eAI#603) * add pre-commit-config.yaml and document nonsupport for infinite horizons * add circleci config check to precommit * Reorganize config file and add missing flake8 dependencies * Add notebooks check * Add pre-commit to test requirements * Auto-fix trailing whitespace or end of files * Fix flake8 errors * Fix typo in exclude types * Make pytype and mypy run locally with dependencies; add docs build * Speed up doc build in precommit by skipping tests and building notebooks * Merge darglint into flake8 (gives us parallelism and better handling of noqa) * Remove code checks script * Try pinning pyglet * Move check-ast first * Fix import os bug in doctest Co-authored-by: Adam Gleave <[email protected]>
maxijsseldijk · Nov 7, 2022 · f64f5f8 · f64f5f8
1 parent e143489
commit f64f5f8
Show file tree

Hide file tree

Showing 26 changed files with 156 additions and 103 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -37,7 +37,7 @@ executors:
       # If you change these, also change ci/code_checks.sh
       SRC_FILES: src/ tests/ experiments/ examples/ docs/conf.py setup.py ci/
       NUM_CPUS: 2
-      EXCLUDE_MYPY: | 
+      EXCLUDE_MYPY: |
         (?x)(
           src/imitation/algorithms/preference_comparisons.py$
           | src/imitation/rewards/serialize.py$

diff --git a/.coveragerc b/.coveragerc
@@ -3,4 +3,4 @@ exclude_lines =
     pragma: no cover
     @overload
     @typing.overload
-    raise NotImplementedError
+    raise NotImplementedError
diff --git a/.gitignore b/.gitignore
@@ -159,4 +159,4 @@ dmypy.json
 # Produced by examples/quickstart.sh
 /quickstart/
 
-.vscode/
+.vscode/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -0,0 +1,87 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+# Linting
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v4.3.0
+  hooks:
+  - id: check-ast
+  - id: trailing-whitespace
+  - id: end-of-file-fixer
+    exclude_types: [jupyter]
+  - id: check-toml
+  - id: check-added-large-files
+- repo: https://github.com/psf/black
+  rev: 22.6.0
+  hooks:
+  - id: black
+  - id: black-jupyter
+- repo: https://github.com/PyCQA/isort
+  rev: 5.10.1
+  hooks:
+  - id: isort
+# Python static analysis
+- repo: https://github.com/pycqa/flake8
+  rev: '5.0.4'
+  hooks:
+  - id: flake8
+    additional_dependencies:
+      - darglint~=1.8.1
+      - flake8-blind-except==0.2.1
+      - flake8-builtins~=1.5.3
+      - flake8-commas~=2.1.0
+      - flake8-debugger~=4.1.2
+      - flake8-docstrings~=1.6.0
+# Shell static analysis
+- repo: https://github.com/koalaman/shellcheck-precommit
+  rev: v0.8.0
+  hooks:
+  - id: shellcheck
+# Misc
+- repo: https://github.com/codespell-project/codespell
+  rev: v2.2.2
+  hooks:
+  - id: codespell
+    args: ["--skip=*.pyc,tests/testdata/*,*.ipynb,*.csv"]
+- repo: https://github.com/syntaqx/git-hooks
+  rev: v0.0.17
+  hooks:
+  - id: circleci-config-validate
+# Hooks that run in local environment (not isolated venv) as they need
+# same dependencies as our package.
+- repo: local
+  hooks:
+  - id: check-notebooks
+    name: check-notebooks
+    entry: ./ci/clean_notebooks.py --check ./docs/tutorials/
+    language: script
+    types: [jupyter]
+    pass_filenames: false
+  # Run mypy directly from local repo rather than using mirror-mypy
+  # so that it uses installed dependencies. Adapted from:
+  # https://jaredkhan.com/blog/mypy-pre-commit
+  - id: mypy
+    name: mypy
+    language: system
+    types: [python]
+    entry: mypy --follow-imports=silent --show-error-codes
+    # use require_serial so that script
+    # is only called once per commit
+    require_serial: true
+    # Print the number of files as a sanity-check
+    verbose: true
+  - id: pytype
+    name: pytype
+    language: system
+    types: [python]
+    entry: pytype -j auto
+    require_serial: true
+    verbose: true
+  - id: docs
+    name: docs
+    language: system
+    types_or: [python, rst]
+    entry: bash -c "cd docs/ && make clean && SKIP_DOCTEST=True NB_EXECUTION_MODE=off make html"
+    require_serial: true
+    verbose: true
+    pass_filenames: false
diff --git a/ci/code_checks.sh b/ci/code_checks.sh
diff --git a/docs/.gitignore b/docs/.gitignore
@@ -1 +1 @@
-_api
+_api
diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css
@@ -1,4 +1,4 @@
 .sidebar-container, .toc-drawer{
     box-sizing:border-box;
     width:20em;
-}
+}
diff --git a/docs/_templates/autosummary/module.rst b/docs/_templates/autosummary/module.rst
@@ -17,12 +17,13 @@
    {% if functions %}
    .. rubric:: {{ _('Functions') }}
    .. testsetup::
+      :skipif: skip_doctests
 
       # import all functions from module since examples don't import them
       from {{ fullname }} import *
 
    .. doctest::
-      
+
       # empty test needed in case the module has no example usage.
       # otherwise, testsetup throws an error
       pass

diff --git a/docs/algorithms/airl.rst b/docs/algorithms/airl.rst
@@ -20,6 +20,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/4_train_airl`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import numpy as np
     import gym

diff --git a/docs/algorithms/bc.rst b/docs/algorithms/bc.rst
@@ -18,6 +18,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/1_train_bc`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import numpy as np
     import gym

diff --git a/docs/algorithms/dagger.rst b/docs/algorithms/dagger.rst
@@ -22,6 +22,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/2_train_dagger`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import tempfile
     import numpy as np

diff --git a/docs/algorithms/density.rst b/docs/algorithms/density.rst
@@ -8,6 +8,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/7_train_density`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import pprint
     import numpy as np

diff --git a/docs/algorithms/gail.rst b/docs/algorithms/gail.rst
@@ -18,6 +18,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/3_train_gail`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import numpy as np
     import gym

diff --git a/docs/algorithms/mce_irl.rst b/docs/algorithms/mce_irl.rst
@@ -10,6 +10,7 @@ Example
 Detailed example notebook: :doc:`../tutorials/6_train_mce`
 
 .. testcode::
+    :skipif: skip_doctests
 
     from functools import partial
 
@@ -71,7 +72,7 @@ Detailed example notebook: :doc:`../tutorials/6_train_mce`
 
 .. testoutput::
     :hide:
-    
+
     ...
 
 API

diff --git a/docs/algorithms/preference_comparisons.rst b/docs/algorithms/preference_comparisons.rst
@@ -6,21 +6,22 @@ The preference comparison algorithm learns a reward function from preferences be
 The comparisons are modeled as being generated from a Bradley-Terry (or Boltzmann rational) model,
 where the probability of preferring trajectory A over B is proportional to the exponential of the
 difference between the return of trajectory A minus B. In other words, the difference in returns forms a logit
-for a binary classification problem, and accordingly the reward function is trained using a cross-entropy loss 
+for a binary classification problem, and accordingly the reward function is trained using a cross-entropy loss
 to predict the preference comparison.
 
 Notes
 -----
 - Our implementation is based on the  `Deep Reinforcement Learning from Human Preferences <https://arxiv.org/pdf/1706.03741.pdf>`_ algorithm.
 
 - An ensemble of reward networks can also be trained instead of a single network. The uncertainty in the preference between the member networks can be used to actively select preference queries.
-    
+
 Example
 =======
 
 Detailed example notebook: :doc:`../tutorials/5_train_preference_comparisons`
 
 .. testcode::
+    :skipif: skip_doctests
 
     import numpy as np
 
@@ -87,7 +88,7 @@ Detailed example notebook: :doc:`../tutorials/5_train_preference_comparisons`
 
 .. testoutput::
     :hide:
-    
+
     ...
 
 API

diff --git a/docs/conf.py b/docs/conf.py
@@ -56,6 +56,13 @@
 nb_merge_streams = True
 nb_output_stderr = "remove"
 
+# skip_doctests is checked in our :skipif: directives in doctest examples
+doctest_global_setup = """
+import os
+
+skip_doctests = os.getenv("SKIP_DOCTEST")
+"""
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 

diff --git a/docs/development/contributing/ways-to-contribute.rst b/docs/development/contributing/ways-to-contribute.rst
@@ -91,13 +91,13 @@ Once you're ready to start working on your pull request, please make sure to fol
 * Your PR includes unit tests for any new features.
 * Your PR includes type annotations, except when it would make the code significantly more complex.
 * You have run the unit tests and there are no errors. We use ``pytest`` for unit testing: run ``pytest tests/`` to run the test suite.
-* You should run ``ci/code_checks.sh`` to run linting and static type checks. We use ``pytype`` for static type analysis.
+* You should run ``pre-commit run`` to run linting and static type checks. We use ``pytype`` for static type analysis.
 
 You may wish to configure this as a Git commit hook:
 
 .. code-block:: bash
 
-   ln -s ../../ci/code_checks.sh .git/hooks/pre-commit
+   pre-commit install
 
 These checks are run on CircleCI and are required to pass before merging.
 Additionally, we track test coverage by CodeCov and require that code coverage