DEVELOPERS.md

OpenCodelists for developers

Repo structure

There are the usual config files in the root of the repo.

opencodelists/

The "project root", containing settings.py etc. Also an app, with models for users, organisations, and projects.

codelists/

The app with models and views for viewing, downloading, creating, and updating codelists.

coding_systems/

A directory containing one app per coding system. Each of these apps contains a coding_system.py which defines a common interface, and an import_data.py which contains code for importing new data.

Each directory contains a README with information about the underlying data.

coding_systems/versioning

An app that holds information about coding system releases.

Each release of a coding system is imported into a separate sqlite database, and has a CodingSystemRelease instance associated with it which allows us to identify the specific release database to use to retrieve codes and terms for a codelist version..

mappings/

A directory containing one app per mapping between coding systems. Each of these apps will have a common structure, which has not yet been codified.

templates/

Templates.

deploy/

Resources for deployment. Deployment is via dokku (see deployment notes).

scripts/

A place to put scripts to be run via runscript.

Notes for developers

Production database and backups

Production data is stored on dokku3 at /storage/ within the container layer file system. This maps to /var/lib/dokku/data/storage/opencodelists in the host operating system's file system. See also deployment notes).

/storage/db.sqlite3 is the core Django database.

/storage/coding_systems contains the coding system databases. These are read-only. Refer to their README files for information on the source data and creation process.

The core database is fully backed up daily on the local file system. Coding system databases are not backed up locally but can be recreated from source. Weekly backups of the droplets allow a restore of the file system.

The core database backups are located at /storage/backup/db. They are created by deploy/bin/backup.sh scheduled via cron as configured in app.json. Backups are taken via the sqlite3 .backup command . These are effectively copies of the database file. They are compressed to save space.

To restore from a backup, use the command-line tool to create a fresh temporary backup of the current state of the database (in case anything gones wrong), then restore from the decompressed backup file. On the production server:

dokku enter opencodelists
sqlite3 /storage/db.sqlite3 ".backup /storage/backup/previous-db.sqlite3"

zstd -d /storage/backup/db/{PATH_TO_BACKUP_ZST} -o /storage/backup/restore-db.sqlite3
sqlite3 /storage/db.sqlite3 ".restore /storage/backup/restore-db.sqlite3

When all is confirmed working with the restore, you can delete previous-db.sqlite3 and restore-db.sqlite3.

The latest backup is available via symlink at /storage/backup/db/latest-db.sqlite3.zst. You can use scp, zstd -d and `sqlite3 ".restore" to bring your local database into the same state as the production database. You may also wish to retrieve the coding systems databases, otherwise you will not be able to interact with codelists that require them.

Local development

Prerequisites:

• Python v3.12.x
• Pip
• fnm
• Just

Install just

# macOS
brew install just

# Linux
# Install from https://github.com/casey/just/releases

# Add completion for your shell. E.g. for bash:
source <(just --completions bash)

# Show all available commands
just #  shortcut for just --list

Install fnm

See https://github.com/Schniz/fnm#installation.

Run local development server

The development server can be run locally, as described below, or in docker.

To use Django Debug Toolbar in development, set DJANGO_DEBUG_TOOLBAR. It is not enabled by default because it adds tens of seconds to the load time of some pages.

Set up/update local dev environment

just dev-setup

Run local django server

just run

Access at http://localhost:7000

Run tests

# python tests and coverage
just test-py

# run specific test with usual pytest syntax
just test-py <path/to/test>::<test name>

# js tests
just assets-test

# all tests
just test

Check/Fix formatting, linting, sorting

# check python files (black, isort, flake8)
just check

# fix black and isort
just fix

# js linting
just assets-lint

Using docker for development and tests

Run a local development server in docker:

just docker-serve

Run the python tests in docker

just docker-test-py

To run named test(s) or pass additional args, pass paths and args as you normally would to pytest:

just docker-test-py tests/reports/test_models.py::test_report_model_validation -k some-mark --pdb

Run the JS tests in docker

just docker-test-js

Run a command in the dev docker container

just docker-run <command>

Build and run a prod container locally (for testing/QA)

just docker-build prod
docker run -it --entrypoint /bin/bash  opencodelists

Other production maintenance tasks

Manually creating and updating codelist versions

Currently OpenCodelists handles multiple coding system releases, however it can't always handle new versions of codelists when the coding system has changed significantly since the original version was created. When a user creates a new version of a codelist, the searches from the original codelist are re-run. If these searches bring back any new codes, the version will fail to be created. Users will see a message prompting them to contact tech-support:

Codes with unknown status found when creating new version; this is likely due to an update in
the coding system. Please contact tech-support for assistance.

The codelist version can be manually created by running the following command on dokku3:

dokku$ dokku run opencodelists \
    python manage.py create_and_update_draft <original version hash> --author <username>

The command will create a new draft version, and will assign statuses to each unknown code where possible. i.e. if a new code has an ancestor code that is included, it will be implicitly included ((+) status), and if it has an ancestor code that is excluded, it will be implictly excluded ((-) status). The command will output a list of codes that were returned as "unknown" (?) status from the re-run searches, and have been assigned an assumed status. These should be passed on to the user to confirm.

Deleting published codelists

⚠️ We should not delete published codelists, except in very rare circumstances. It is difficult to guarantee that codelists are not being anywhere at all — there may be uses other than for OpenSAFELY projects.

However, on rare occasions, users may request deletion of published codelists or their versions.

Note that users can delete unpublished versions of codelists, or entirely unpublished codelists themselves. Users cannot delete published versions of codelists.

These requests should be limited to user-owned codelists, not organisation codelists.

Process for deleting a single version of a codelist

First, visit the codelist URL of interest and note down the version ID hash on the page.

On dokku3:

1. Start shell_plus which loads the database models for you:

   $ dokku run opencodelists python manage.py shell_plus

2. Access the specific version of the codelist:

   >>> version = CodelistVersion.objects.get_by_hash("<hash>")

3. Delete the codelist version:

   >>> version.delete()

Process for deleting a codelist entirely

First, visit the codelist URL of interest and note down the version hash on the page.

On dokku3:

1. Start shell_plus which loads the database models for you:

   $ dokku run opencodelists python manage.py shell_plus

2. Access the codelist through the specified version:

   >>> version = CodelistVersion.objects.get_by_hash("<hash>")
   >>> codelist = version.codelist

3. Delete the codelist and all of its versions:

   >>> codelist.delete()

Import codelists from an xlsx file

See codelists/scripts/README.md

Updating mappings

See the relevant README in each of the subdirectories inside mappings/.