Merge pull request #26 from ONSdigital/develop

First release

.github/pull_request_template.md

@@ -1,34 +1,80 @@
-# Summary
+## Pull Request submission
 
-Add your summary here - keep it brief, to the point, and in plain English. [For further
-information about pull requests, check out the GDS
-Way](https://gds-way.cloudapps.digital/standards/pull-requests.html).
+*Insert detailed bullet points about your changes here!*
 
-# Checklists
+*Insert any instructions to help the reviewer, e.g. "install new requirements from `environment.yml`"*
 
-<!--
-These are do-confirm checklists; it confirms that you have DOne each item.
+*Let the reviewer know what data files are needed (and if applicable, where they are to be found)
 
-Outstanding actions should be completed before reviewers are assigned; if actions are
-irrelevant, please try and add a comment stating why.
+#### Closes or fixes
 
-Incomplete pull/merge requests may be blocked until actions are resolved, or closed at
-the reviewers' discretion.
--->
+* Detail the ticket(s) you are closing with this PR
+Closes #
 
-This pull/merge request meets the following requirements:
 
-- [ ] code runs
-- [ ] [developments are ethical][data-ethics-framework] and secure
-- [ ] you have made proportionate checks that the code works correctly
-- [ ] test suite passes
-- [ ] developments adhere to AQA plan (see `docs/aqa/aqa_plan.md`)
-- [ ] data log updated (see `docs/aqa/data_log.md`), if necessary
-- [ ] assumptions, and caveats log updated (see `docs/aqa/assumptions_caveats.md`), if
-  necessary
-- [ ] [minimum usable documentation][agilemodeling] written in the `docs` folder
+#### Code
 
-Comments have been added below around the incomplete checks.
+- [ ] **Code runs** The code runs on my machine and/or CDSW
+- [ ] **Conflicts resolved** There are no conflicts (I have performed a rebase if necessary)
+- [ ] **Requirements** My/our code functions according to the requirements of the ticket
+- [ ] **Dependencies** I have updated the environment yaml so it includes any new libraries I have used
+- [ ] **Configuration file updated** any high level parameters that the user may interact with have been put into the config file (and imported to the script)
+- [ ] **Clean Code**
+    - [ ] Code is as [PEP 8]([url](https://peps.python.org/pep-0008/)) compliant as I can humanly make it
+    - [ ] Code passess flake8 linting check
+    - [ ] Code adheres to [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself)
+- [ ] **Type hints** All new functions have [type hints ](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html)
 
-[agilemodeling]: http://agilemodeling.com/essays/documentLate.htm
-[data-ethics-framework]: https://www.gov.uk/government/publications/data-ethics-framework
+
+#### Documentation
+
+Any new code includes all the following forms of documentation:
+
+- [ ] **Function Documentation** Docstrings within the function(s')/methods have been created
+    - [ ] Includes `Args` and `returns` for all major functions
+    - [ ] The docstring details data types
+- [ ] **Updated Documentation**: User and/or developer working doc has been updated
+
+#### Data
+- [ ] All data needed to run this script is available in Dev/Test
+- [ ] All data is excluded from this pull request
+- [ ] Secrets checker pre-commit passes
+
+#### Testing
+- [ ] **Unit tests** Unit tests have been created and are passing _or a new ticket to create tests has been created_
+
+---
+
+# Peer Review Section
+
+- [ ] All requirements install from (updated) `environment.yaml`
+- [ ] Documentation has been created and is clear - **check the working document**
+- [ ] Doctrings ([Google format](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html)) have been created and accurately describe the function's functionality
+- [ ] Unit tests pass, or if not present _a new ticket to create tests has been created_
+- [ ] **Code runs** The code runs on reviewer's machine and/or CDSW
+
+#### Final approval (post-review)
+
+The author has responded to my review and made changes to my satisfaction.
+- [ ] **I recommend merging this request.**
+
+---
+
+### Review comments
+
+*Insert detailed comments here!*
+
+These might include, but not exclusively:
+
+- bugs that need fixing (does it work as expected? and does it work with other code
+  that it is likely to interact with?)
+- alternative methods (could it be written more efficiently or with more clarity?)
+- documentation improvements (does the documentation reflect how the code actually works?)
+- additional tests that should be implemented (do the tests effectively assure that it
+  works correctly?)
+- code style improvements (could the code be written more clearly?)
+- Do the changes represent a change in functionality so the version number should increase? Start a discussion if so.
+- As a review you can generates the same outputs from running the code
+
+Your suggestions should be tailored to the code that you are reviewing.
+Be critical and clear, but not mean. Ask questions and set actions.

.github/workflows/pytest-action.yaml

@@ -0,0 +1,42 @@
+name: Run pytest on pull request to develop
+
+# runs on every pull request to develop
+on:
+  pull_request:
+    branches:
+      - develop
+# runs on version 20.04 of ubuntu
+jobs:
+  pytest-coverage-comment:
+    runs-on: ubuntu-20.04
+    steps:
+
+# 1) Checkout the code
+      - uses: actions/checkout@v3
+# 2) Removing PyDoop from the environment yaml
+      - name: Remove pydoop dependency
+        shell: bash -l {0}
+        run: |
+          awk '!/pydoop.*/' environment.yml > temp && mv temp environment.yml
+# 3) Use Setup Miniconda github action to setup environment
+      - uses: conda-incubator/setup-miniconda@v2
+        with:
+          python-version: 3.6
+          environment-file: environment.yml
+          activate-environment: resdev36
+# 3) Run pytest to run all tests in the tests folder
+      - name: Use coverage to run pytest
+      # Specify shell to run the command in
+        shell: bash -l {0}
+        run: |
+          coverage run --branch --source=./src -m pytest -ra ./tests --junitxml=junit_result.xml && coverage xml -o python_coverage.xml && coverage report -m --fail-under=10
+# 4) Get the coverage report in to the pull request comments
+      - name: Pytest coverage comment
+        uses: MishaKav/pytest-coverage-comment@main
+        with:
+          title: Detailed Coverage Report
+          badge-title: Percentage Coverage for this PR
+          pytest-xml-coverage-path: ./python_coverage.xml
+          coverage-path-prefix: src/
+          junitxml-title: Summary of tests
+          junitxml-path: ./junit_result.xml

.gitignore

@@ -900,3 +900,7 @@ docs/reference/api/*
 # Ignore Sphinx documentation link checking folder
 docs/_linkcheck/
 
+# logs files
+logs/*
+!logs/.gitkeep
+!logs/logs.md

.pre-commit-config.yaml

@@ -60,3 +60,10 @@ repos:
         name: detect-secrets - Detect secrets in staged code
         args: [ "--baseline", ".secrets.baseline" ]
         exclude: .*/tests/.*|^\.cruft\.json$
+  - repo: local
+    hooks:
+    - id: restricted-filenames
+      name: Check commits for restricted file extensions
+      entry: These file extensions are restricted. Data should be removed from the commit
+      language: fail
+      files: .*\.(csv|feather|xlsx|zip|hdf5|h5|txt|json|xml|hd|parquet)

config/userconfig.toml

@@ -0,0 +1,26 @@
+# This is the TOML config file for users to set
+# the parameters before running the pipeline.
+
+title = "TOML Example config"
+
+# Period to be considered (must be in YYYY-MM-DD format)
+[period]
+start_period = 1990-10-10
+end_period = 2000-10-05
+
+# Source file set up
+[source_file]
+location = "D:/Data"
+fileName = "file.txt"
+
+# Output location including Hive Database and Table Name
+[output_location]
+hive_db = "hive.db"
+tableName = "name.table"
+
+# Location, file name and true/false flag for an outlier
+# correction file if needed
+[outlier_correction]
+location = "D:/"
+filename = "outliers.txt"
+bool = true

environment.yml

@@ -1,69 +1,21 @@
-name: resdev310
+name: resdev36
 dependencies:
-  - arrow=1.2.2=pyhd3eb1b0_0
-  - binaryornot=0.4.4=pyhd3eb1b0_1
-  - brotlipy=0.7.0=py310h2bbff1b_1002
-  - bzip2=1.0.8=he774522_0
-  - ca-certificates=2022.10.11=haa95532_0
-  - certifi=2022.9.24=py310haa95532_0
-  - cffi=1.15.1=py310h2bbff1b_0
-  - cfgv=3.3.1=py310haa95532_0
-  - chardet=4.0.0=py310haa95532_1003
-  - charset-normalizer=2.0.4=pyhd3eb1b0_0
-  - click=8.0.4=py310haa95532_0
-  - colorama=0.4.5=py310haa95532_0
-  - cookiecutter=1.7.3=pyhd3eb1b0_0
-  - cryptography=37.0.1=py310h21b164f_0
-  - distlib=0.3.2=pyhd3eb1b0_0
-  - filelock=3.6.0=pyhd3eb1b0_0
-  - identify=2.5.5=py310haa95532_0
-  - idna=3.4=py310haa95532_0
-  - jinja2=3.0.3=pyhd3eb1b0_0
-  - jinja2-time=0.2.0=pyhd3eb1b0_3
-  - libffi=3.4.2=hd77b12b_4
-  - markupsafe=2.1.1=py310h2bbff1b_0
-  - nodeenv=1.7.0=py310haa95532_0
-  - openssl=1.1.1q=h2bbff1b_0
-  - pip=22.2.2=py310haa95532_0
-  - platformdirs=2.5.2=py310haa95532_0
-  - poyo=0.5.0=pyhd3eb1b0_0
-  - pre-commit=2.20.0=py310haa95532_0
-  - pre_commit=2.20.0=hd3eb1b0_0
-  - pycparser=2.21=pyhd3eb1b0_0
-  - pyopenssl=22.0.0=pyhd3eb1b0_0
-  - pysocks=1.7.1=py310haa95532_0
-  - python=3.10.6=hbb2ffb3_0
-  - python-dateutil=2.8.2=pyhd3eb1b0_0
-  - python-slugify=5.0.2=pyhd3eb1b0_0
-  - pyyaml=6.0=py310h2bbff1b_0
-  - requests=2.28.1=py310haa95532_0
-  - six=1.16.0=pyhd3eb1b0_1
-  - sqlite=3.39.3=h2bbff1b_0
-  - text-unidecode=1.3=pyhd3eb1b0_0
-  - tk=8.6.12=h2bbff1b_0
-  - toml=0.10.2=pyhd3eb1b0_0
-  - typing_extensions=4.3.0=py310haa95532_0
-  - tzdata=2022e=h04d1e81_0
-  - ukkonen=1.0.1=py310h59b6b97_0
-  - unidecode=1.2.0=pyhd3eb1b0_0
-  - urllib3=1.26.11=py310haa95532_0
-  - vc=14.2=h21ff451_1
-  - virtualenv=20.16.2=py310haa95532_0
-  - vs2015_runtime=14.27.29016=h5e58377_2
-  - wheel=0.37.1=pyhd3eb1b0_0
-  - win_inet_pton=1.1.0=py310haa95532_0
-  - wincertstore=0.2=py310haa95532_2
-  - xz=5.2.6=h8cc25b3_0
-  - yaml=0.2.5=he774522_0
-  - zlib=1.2.12=h8cc25b3_3
-  - pip:
-    - setuptools==65.5.0
-    - coverage
-    - detect-secrets==1.0.3
-    - myst-parser
-    - pre-commit
-    - pytest
-    - python-dotenv
-    - Sphinx
-    - toml
-prefix: C:\Users\westwj1\Anaconda3\envs\resdev310
+- python=3
+- coverage
+- pyyaml
+- requests
+- sphinx
+- pip
+- pip:
+  - arrow
+  - cookiecutter
+  - detect-secrets
+  - myst-parser
+  - pre-commit==2.17.0
+  - python-dotenv
+  - table_logger
+  - pandas==1.1.5
+  - numpy
+  - pydoop
+  - setuptools
+  - pytest

logs/logs.md

@@ -0,0 +1,5 @@
+# Folder for all log files
+
+Do not commit and log files to the repository.
+
+These will be excluded with .gitignore and pre-commits

main.py

@@ -0,0 +1,10 @@
+from importlib import reload
+import time
+
+import src.main as src
+
+# reload the pipeline module to implement any changes
+reload(src)
+
+start = time.time()
+src.run_pipeline(start)

previous-environment.yml

@@ -0,0 +1,69 @@
+name: resdev310
+dependencies:
+  - arrow=1.2.2=pyhd3eb1b0_0
+  - binaryornot=0.4.4=pyhd3eb1b0_1
+  - brotlipy=0.7.0=py310h2bbff1b_1002
+  - bzip2=1.0.8=he774522_0
+  - ca-certificates=2022.10.11=haa95532_0
+  - certifi=2022.9.24=py310haa95532_0
+  - cffi=1.15.1=py310h2bbff1b_0
+  - cfgv=3.3.1=py310haa95532_0
+  - chardet=4.0.0=py310haa95532_1003
+  - charset-normalizer=2.0.4=pyhd3eb1b0_0
+  - click=8.0.4=py310haa95532_0
+  - colorama=0.4.5=py310haa95532_0
+  - cookiecutter=1.7.3=pyhd3eb1b0_0
+  - cryptography=37.0.1=py310h21b164f_0
+  - distlib=0.3.2=pyhd3eb1b0_0
+  - filelock=3.6.0=pyhd3eb1b0_0
+  - identify=2.5.5=py310haa95532_0
+  - idna=3.4=py310haa95532_0
+  - jinja2=3.0.3=pyhd3eb1b0_0
+  - jinja2-time=0.2.0=pyhd3eb1b0_3
+  - libffi=3.4.2=hd77b12b_4
+  - markupsafe=2.1.1=py310h2bbff1b_0
+  - nodeenv=1.7.0=py310haa95532_0
+  - openssl=1.1.1q=h2bbff1b_0
+  - pip=22.2.2=py310haa95532_0
+  - platformdirs=2.5.2=py310haa95532_0
+  - poyo=0.5.0=pyhd3eb1b0_0
+  - pre-commit=2.20.0=py310haa95532_0
+  - pre_commit=2.20.0=hd3eb1b0_0
+  - pycparser=2.21=pyhd3eb1b0_0
+  - pyopenssl=22.0.0=pyhd3eb1b0_0
+  - pysocks=1.7.1=py310haa95532_0
+  - python=3.10.6=hbb2ffb3_0
+  - python-dateutil=2.8.2=pyhd3eb1b0_0
+  - python-slugify=5.0.2=pyhd3eb1b0_0
+  - pyyaml=6.0=py310h2bbff1b_0
+  - requests=2.28.1=py310haa95532_0
+  - six=1.16.0=pyhd3eb1b0_1
+  - sqlite=3.39.3=h2bbff1b_0
+  - text-unidecode=1.3=pyhd3eb1b0_0
+  - tk=8.6.12=h2bbff1b_0
+  - toml=0.10.2=pyhd3eb1b0_0
+  - typing_extensions=4.3.0=py310haa95532_0
+  - tzdata=2022e=h04d1e81_0
+  - ukkonen=1.0.1=py310h59b6b97_0
+  - unidecode=1.2.0=pyhd3eb1b0_0
+  - urllib3=1.26.11=py310haa95532_0
+  - vc=14.2=h21ff451_1
+  - virtualenv=20.16.2=py310haa95532_0
+  - vs2015_runtime=14.27.29016=h5e58377_2
+  - wheel=0.37.1=pyhd3eb1b0_0
+  - win_inet_pton=1.1.0=py310haa95532_0
+  - wincertstore=0.2=py310haa95532_2
+  - xz=5.2.6=h8cc25b3_0
+  - yaml=0.2.5=he774522_0
+  - zlib=1.2.12=h8cc25b3_3
+  - pip:
+    - setuptools==65.5.0
+    - coverage
+    - detect-secrets==1.0.3
+    - myst-parser
+    - pre-commit
+    - pytest
+    - python-dotenv
+    - Sphinx
+    - toml
+prefix: C:\Users\westwj1\Anaconda3\envs\resdev310

pyproject.toml

@@ -17,7 +17,8 @@ profile = "black"
 [tool.pytest.ini_options]
 addopts = [
     "-vv",
-    "--doctest-modules"
+    "--doctest-modules",
+    "--ignore=tests/test_utils/test_hdfs_mods.py"
 ]
 doctest_optionflags = "NORMALIZE_WHITESPACE"
 testpaths = [

src/_version.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"