Test notebooks with CI

[agstephens]

See Richard's example:
https://github.com/cedadev/cci-api-demo/blob/main/.github/workflows/test.yml

[agstephens]

(How) would we mount the archive in the CI pipeline?

[Mahir-Sparkess]

(How) would we mount the archive in the CI pipeline?

@agstephens
I've tabulated all the files accessed in the archive and their size if you have any ideas on the best approach to this.

file	size (bytes)
/badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/abrupt-4xCO2/r1i1p1f1/Amon/tas/gn/v20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_095001-099912.nc	75
/gws/nopw/j04/cedaproc/public-data/uk_max_temp.txt	13877
/badc/cru/data/cru_ts/cru_ts_4.02/data/tmp/cru_ts4.02.1901.2017.tmp.dat.nc	2911345672
/badc/cru/data/cru_ts/cru_ts_4.02/data/pre/cru_ts4.02.1901.2017.pre.dat.nc	2911345652
/neodc/baci_ssv/data/v1.0/regional_sites/13_europe/optical/rho_h18v04_2015_7day.nc	2946270820
/neodc/baci_ssv/data/v1.0/regional_sites/13_europe/lst/lst_h18v04_2015_7day.nc	284298031
/neodc/baci_ssv/data/v1.0/regional_sites/13_europe/sar/sentinel-1_descending_h18v04_2015_7day_vv.nc	161935270
/neodc/baci_ssv/data/v1.0/regional_sites/13_europe/sar/sentinel-1_ascending_h18v04_2015_7day_vv.nc	180899439
/neodc/sentinel5p/data/L2_NO2/v1.3/2020/01/01/S5P_OFFL_L2__NO2____20200101T110146_20200101T124316_11493_01_010302_20200103T041218.nc	453013609
/neodc/esacci/sst/data/lt/Analysis/L4/v01.1/1991/09/01/19910901120000-ESACCI-L4_GHRSST-SSTdepth-OSTIA-GLOB_LT-v02.0-fv01.1.nc	19813423

total	9.2 GB

I am not sure what the limit is in the Github action environment, or can not find a clear answer, but I imagine this is too big.

above mini-archive, a subset of the netCDF files: https://github.com/Mahir-Sparkess/mini-ceda-archive/tree/ceda-notebooks
As of writing: 364 KB

[agstephens]

@Mahir-Sparkess I like the fact that there aren't too many of these.

For other projects, we have done the following:

1. Create a mini version of the relevant files (cut-down in time and/or space) in their own repos, see:
   https://github.com/roocs/mini-esgf-data
   https://github.com/cedadev/mini-ceda-archive

2. Set up pytest to include a fixture that downloads the relevant mini repo to the $HOME directory:

• Fixture to download data: https://github.com/cedadev/midas-extract/blob/master/tests/conftest.py#L25-L38
• This gets attached to all tests that need the data, e.g.: https://github.com/cedadev/midas-extract/blob/master/tests/test_midas_extract.py#L117-L124

We could do a similar thing with the above files and then just symlink them to "/badc" and "/neodc" when running the tests.

[Mahir-Sparkess]

I've created an issue on the jasmin notebooks repository
This issue can be closed when that issue is resolved.

[Mahir-Sparkess]

The link to the notebook CI dockerfile: https://breezy.badc.rl.ac.uk/mrahman/jasmin-notebooks/-/blob/notebooks-test/images/ceda-notebooks-ci/Dockerfile