Add CRUTS4.05 data to flamingo

[agstephens]

NOTES about gzipped CRU TS data:

1. xarray can read .nc.gz - so we don't need to gunzip or change anything!

Tasks:

• AS: Extract WPS input info from Flamingo process module/utils:
  • Move to ceda-wps-assets
  • Make WPS dependent on ceda-wps-assets
• Update the mini-ceda-archive to add: /badc/cru/data/cru_ts/cru_ts_4.05/data/wet/cru_ts4.05.1901.2020.wet.dat.nc.gz
• AS/FE to discuss procedure and document the process:
  • Doc URL: https://secure.helpscout.net/docs/59b25bca042863033a1caf91/article/5d9c7a1b04286364bc901df0/
  • Process is:
    • Check data can be read by Xarray
    • Check data can be read by clisops package
    • Create test file(s) in the mini-ceda-archive
    • Update datasets.yml file with information about new data
    • Write additional tests in flamingo/tests
    • Run on test server with test UI
    • Connect test server to production UI (for quick look) - then disconnect
    • Deploy on production WPS cluster
    • Connect to production UI
    • Test, test, test
• AS: Put config content for each dataset in to ceda-wps-assets
• AS/FE: Add tests for each new dataset.
• FE/AS: Deploy on to test server:
  • Can we run WPS on test server and connect to production UI for testing (temporarily) - only admin access?
• FE/AS: Deploy into production (using updated Ansible playbook - pulling info from ceda-wps-assets)
• Update MOLES records with a script to add process info: https://catalogue.ceda.ac.uk/uuid/c26a65020a5e4b80b20018f148556681

[agstephens]

NOTES ABOUT PROCESS:

1. Gunzipping uses lots of memory - we only have 4GB RAM on ceda-wps-staging - so testing fails with real data. Solution could be:

• Set up to use test data only (rather than /badc/). The playbook can do this by cloning to /tmp/mini-ceda-archive/... and updating the /etc/roocs.ini file from the template. Just need to change the base path in it.

[agstephens]

Related to: #13

[agstephens]

To check:

• /etc/roocs.ini should not use ~/.mini-ceda-archive/ on ceda-wps-staging.ceda.ac.uk
  • Also, cannot use /root/.mini-ceda-archive/ because the cwps user CANNOT read the root directory!
  • So, you need to either use the real archive (see memory error below), or, make sure the test data is visible at /tmp/mini-ceda-archive/.... so that it can be read. - only test with variable wet.
• ceda-wps-staging is hitting memory allocation errors (in Flamingo jobs) - so we are asking to double to RAM from 4GB to 8GB.
• Edit requirements.txt to pywps==4.4
• Update the production service so that it allows: SubsetHadUKGrid to be run.
• This requires the roocs-utils module of fixed-path-modifiers.
  • Check that there are no breaking changes in fixed-path-modifiers branch and then merge it to Master, and add it to history file. Then change requirements to use latest from Github in recipe.

[agstephens]

Try to re-deploy, with HadUK-Grid switched off, on ceda-wps-staging.

NOTE: must use pywps==4.4 at the moment; 4.5 is broken on github.

[agstephens]

Combine roll-out with: #13
They both appear to work.