Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jupyter Notebook (Use Case 3) for the EOBS dataset - to be reviewed and optimised w/ eqc toolbox #161

Open
Figas23 opened this issue Oct 21, 2024 · 23 comments
Labels

Comments

@Figas23
Copy link

Figas23 commented Oct 21, 2024

Notebook description

Insitu wind and precipitation completeness for climate monitoring

Use Case: Impact of climate change on precipitation over the Iberian Peninsula.

User Question: How consistent are the estimates of precipitation trends from E-OBS compared to those from reanalysis products?

Method: ERA5 and E-OBS daily precipitation sum (RR) from the latest available versions available, shall be used to calculate: (i) the probability density function of each alternative 30-year period (i.e., 1951-1980, 1961-1990, 1971-2000, 1981-2010, 1991-2020); (ii) the amount of days accounting for the tails of the distribution, i.e., annual count of days when RR >=10mm, (as defined by the WMO Expert Team on Climate Change Detection Indices, ETCCDI) and (iii) the corresponding maps. These maps will be compared to disclose if RR extremes are shifting in terms of mean and the distribution's tails. These results shall be compared in the two datasets.

Notebook link or upload

C3S2_D520.5.3.14b_Quality_Assessment_User_Questions_EOBS_UQ3_v1.6.zip

Anything else we need to know?

No response

Environment

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
accessible-pygments 0.0.4 pyhd8ed1ab_0 conda-forge
affine 2.4.0 pyhd8ed1ab_0 conda-forge
alabaster 0.7.13 pyhd8ed1ab_0 conda-forge
alsa-lib 1.2.8 h166bdaf_0 conda-forge
antlr-python-runtime 4.7.2 py310hff52083_1003 conda-forge
arrow-cpp 11.0.0 ha770c72_13_cpu conda-forge
astroid 2.15.4 py310hff52083_0 conda-forge
asttokens 2.0.5 pyhd3eb1b0_0
attr 2.5.1 h166bdaf_1 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
aws-c-auth 0.6.26 hdca2abe_0 conda-forge
aws-c-cal 0.5.21 h48707d8_2 conda-forge
aws-c-common 0.8.14 h0b41bf4_0 conda-forge
aws-c-compression 0.2.16 h03acc5a_5 conda-forge
aws-c-event-stream 0.2.20 h00877a2_4 conda-forge
aws-c-http 0.7.5 hf342b9f_5 conda-forge
aws-c-io 0.13.19 h094d3b0_2 conda-forge
aws-c-mqtt 0.8.6 h337b09f_11 conda-forge
aws-c-s3 0.2.7 hde0a405_0 conda-forge
aws-c-sdkutils 0.1.8 h03acc5a_0 conda-forge
aws-checksums 0.1.14 h03acc5a_5 conda-forge
aws-crt-cpp 0.19.8 h85f3b07_11 conda-forge
aws-sdk-cpp 1.10.57 h17c43bd_8 conda-forge
babel 2.12.1 pyhd8ed1ab_1 conda-forge
backcall 0.2.0 pyhd3eb1b0_0
beautifulsoup4 4.12.2 pyha770c72_0 conda-forge
blosc 1.21.3 hafa529b_0 conda-forge
bokeh 2.4.3 pyhd8ed1ab_3 conda-forge
boost-cpp 1.78.0 h6582d0a_3 conda-forge
branca 0.6.0 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
brotlipy 0.7.0 py310h5764c6d_1005 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
c3s-eqc-automatic-quality-control 0.1.2.dev44+g6f65bb5 pypi_0 pypi
c3s-eqc-toolbox-template 0.2.2.dev110+gf4e6b34 pypi_0 pypi
ca-certificates 2023.7.22 hbcca054_0 conda-forge
cacholote 0.3.0 pypi_0 pypi
cads-api-client 1.4.3 pypi_0 pypi
cads-toolbox 0.0.2b1.dev3+gecb63ae pypi_0 pypi
cairo 1.16.0 h35add3b_1015 conda-forge
cartopy 0.21.1 py310h7eb24ba_1 conda-forge
cdsapi 0.7.3 pypi_0 pypi
certifi 2023.7.22 pyhd8ed1ab_0 conda-forge
cf-units 3.2.0 py310h0a54255_0 conda-forge
cf_xarray 0.8.0 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
cfgrib 0.9.10.3 pyhd8ed1ab_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
cfitsio 4.2.0 hd9d235c_0 conda-forge
cftime 1.6.2 py310hde88566_1 conda-forge
cgul 0.0.4 pypi_0 pypi
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
comm 0.1.2 py310h06a4308_0
contourpy 1.0.7 py310hdf3cbec_0 conda-forge
coucal 0.0.1b3 pypi_0 pypi
coverage 7.2.5 py310h2372a71_0 conda-forge
cryptography 39.0.2 py310h34c0648_0 conda-forge
curl 8.0.1 h588be90_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.12.0 py310h5764c6d_1 conda-forge
dask 2023.3.1 pyhd8ed1ab_0 conda-forge
dask-core 2023.3.1 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
debugpy 1.5.1 py310h295c915_0
decorator 5.1.1 pyhd3eb1b0_0
distlib 0.3.6 pyhd8ed1ab_0 conda-forge
distributed 2023.3.1 pyhd8ed1ab_0 conda-forge
docutils 0.19 py310hff52083_1 conda-forge
eccodes 2.30.0 h95353b9_0 conda-forge
emohawk 0.0.4b0 pypi_0 pypi
entrypoints 0.4 py310h06a4308_0
esmf 8.4.1 nompi_h5b4e5e4_2 conda-forge
esmpy 8.4.1 pyhc1e730c_0 conda-forge
exceptiongroup 1.1.1 pyhd8ed1ab_0 conda-forge
executing 0.8.3 pyhd3eb1b0_0
expat 2.5.0 h27087fc_0 conda-forge
fftw 3.3.10 nompi_hf0379b8_106 conda-forge
filelock 3.12.0 pyhd8ed1ab_0 conda-forge
findlibs 0.0.2 pyhd8ed1ab_0 conda-forge
fiona 1.9.2 py310ha325b7b_0 conda-forge
folium 0.14.0 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.2 py310h1fa729e_0 conda-forge
freeglut 3.2.2 h9c3ff4c_1 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
freexl 1.0.6 h166bdaf_1 conda-forge
fsspec 2023.4.0 pyh1a96a4e_0 conda-forge
gdal 3.6.4 py310hf0ca374_1 conda-forge
geographiclib 1.52 pyhd8ed1ab_0 conda-forge
geopandas 0.12.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.12.2 pyha770c72_0 conda-forge
geopy 2.3.0 pyhd8ed1ab_0 conda-forge
geos 3.11.2 hcb278e6_0 conda-forge
geotiff 1.7.1 h480ec47_8 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
gflags 2.2.2 he1b5a44_1004 conda-forge
giflib 5.2.1 h0b41bf4_3 conda-forge
glib 2.74.1 h6239696_1 conda-forge
glib-tools 2.74.1 h6239696_1 conda-forge
glog 0.6.0 h6f12383_0 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
greenlet 2.0.2 py310heca2aa9_0 conda-forge
gst-plugins-base 1.22.0 h4243ec0_2 conda-forge
gstreamer 1.22.0 h25f0c4b_2 conda-forge
gstreamer-orc 0.4.33 h166bdaf_0 conda-forge
harfbuzz 6.0.0 h3ff4399_1 conda-forge
hdf4 4.2.15 h501b40f_6 conda-forge
hdf5 1.14.0 nompi_hb72d44e_103 conda-forge
heapdict 1.0.1 py_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
identify 2.5.24 pyhd8ed1ab_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
imagesize 1.4.1 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.1.0 pyha770c72_0 conda-forge
iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge
ipykernel 6.19.2 py310h2f386ee_0
ipython 8.10.0 py310h06a4308_0
jack 1.9.22 h11f4161_0 conda-forge
jasper 4.0.0 h32699f2_1 conda-forge
jedi 0.18.1 py310h06a4308_1
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
json-c 0.16 hc379101_0 conda-forge
jupyter_client 7.4.9 py310h06a4308_0
jupyter_core 5.2.0 py310h06a4308_0
kaleido 0.2.1 pypi_0 pypi
kealib 1.5.0 he7a6254_1 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py310hbf28c38_1 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lazy-object-proxy 1.9.0 py310h1fa729e_0 conda-forge
lcms2 2.15 haa2dc70_1 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20230125.0 cxx17_hcb278e6_1 conda-forge
libaec 1.0.6 hcb278e6_1 conda-forge
libarrow 11.0.0 h93537a5_13_cpu conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcap 2.67 he9d0100_0 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libclang 16.0.3 default_h83cc7fd_0 conda-forge
libclang13 16.0.3 default_hd781213_0 conda-forge
libcrc32c 1.1.2 h9c3ff4c_0 conda-forge
libcups 2.3.3 h36d4200_3 conda-forge
libcurl 8.0.1 h588be90_0 conda-forge
libdb 6.2.32 h9c3ff4c_0 conda-forge
libdeflate 1.18 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 h28343ad_4 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.4.2 h27087fc_0 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgcrypt 1.10.1 h166bdaf_0 conda-forge
libgdal 3.6.4 h7239d12_1 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.74.1 h606061b_1 conda-forge
libglu 9.0.0 he1b5a44_1001 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libgoogle-cloud 2.8.0 h0bc5f78_1 conda-forge
libgpg-error 1.46 h620e276_0 conda-forge
libgrpc 1.52.1 hcf146ea_1 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge
libkml 1.3.0 h37653c0_1015 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm11 11.1.0 he0ac6c6_5 conda-forge
libllvm15 15.0.7 hadd5161_1 conda-forge
libllvm16 16.0.3 hbf9e925_1 conda-forge
libnetcdf 4.9.2 nompi_hf3f8848_103 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libnuma 2.0.16 h0b41bf4_1 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.2 hb675445_0 conda-forge
libprotobuf 3.21.12 h3eb15da_0 conda-forge
librttopo 1.1.0 h0d5128d_13 conda-forge
libsndfile 1.2.0 hb75c966_0 conda-forge
libsodium 1.0.18 h7b6447c_0
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.0.1 h7d1ca68_25 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libsystemd0 253 h8c4010b_1 conda-forge
libthrift 0.18.1 h5e4af38_0 conda-forge
libtiff 4.5.0 ha587672_6 conda-forge
libtool 2.4.7 h27087fc_0 conda-forge
libudev1 253 h0b41bf4_1 conda-forge
libutf8proc 2.8.0 h166bdaf_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.5.0 h79f4944_1 conda-forge
libxml2 2.10.4 hfdac1af_0 conda-forge
libzip 1.9.2 hc929e4a_1 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
llvmlite 0.39.1 py310h58363a5_1 conda-forge
locket 1.0.0 pyhd8ed1ab_0 conda-forge
lz4 4.3.2 py310h0cfdcf0_0 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
make 4.3 hd18ef5c_1 conda-forge
mapclassify 2.5.0 pyhd8ed1ab_1 conda-forge
markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.2 py310h1fa729e_0 conda-forge
matplotlib 3.7.1 py310hff52083_0 conda-forge
matplotlib-base 3.7.1 py310he60537e_0 conda-forge
matplotlib-inline 0.1.6 py310h06a4308_0
mdit-py-plugins 0.3.5 pyhd8ed1ab_0 conda-forge
mdurl 0.1.0 pyhd8ed1ab_0 conda-forge
mpg123 1.31.3 hcb278e6_0 conda-forge
msgpack-python 1.0.5 py310hdf3cbec_0 conda-forge
multiurl 0.3.1 pypi_0 pypi
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mypy 1.2.0 py310h1fa729e_0 conda-forge
mypy_extensions 1.0.0 pyha770c72_0 conda-forge
mysql-common 8.0.32 ha901b37_1 conda-forge
mysql-libs 8.0.32 hd7da12d_1 conda-forge
myst-parser 1.0.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nest-asyncio 1.5.6 py310h06a4308_0
netcdf-fortran 4.6.0 nompi_h3142581_105 conda-forge
netcdf4 1.6.3 nompi_py310h2d0b64f_102 conda-forge
networkx 3.0 pyhd8ed1ab_0 conda-forge
nodeenv 1.7.0 pyhd8ed1ab_0 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numba 0.56.4 py310ha5257ce_0 conda-forge
numpy 1.23.5 py310h53a5b5f_0 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.2 hd590300_0 conda-forge
orc 1.8.3 hfdbbad2_0 conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 2.0.1 py310h7cbd5c2_0 conda-forge
pandas-stubs 2.0.1.230501 pyhd8ed1ab_0 conda-forge
parquet-cpp 1.5.1 2 conda-forge
parso 0.8.3 pyhd3eb1b0_0
partd 1.3.0 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pexpect 4.8.0 pyhd3eb1b0_3
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 9.4.0 py310h065c6d2_2 conda-forge
pip 23.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
platformdirs 3.1.1 pyhd8ed1ab_0 conda-forge
plotly 5.14.1 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 pyhd8ed1ab_5 conda-forge
ply 3.11 py_1 conda-forge
pooch 1.7.0 pyha770c72_3 conda-forge
poppler 23.04.0 hf052cbe_1 conda-forge
poppler-data 0.4.12 hd8ed1ab_0 conda-forge
postgresql 15.2 h3248436_0 conda-forge
pre-commit 3.3.1 pyha770c72_0 conda-forge
proj 9.2.0 h8ffa02c_0 conda-forge
prompt-toolkit 3.0.36 py310h06a4308_0
psutil 5.9.4 py310h5764c6d_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3eb1b0_2
pulseaudio 16.1 hcb278e6_3 conda-forge
pulseaudio-client 16.1 h5195f5e_3 conda-forge
pulseaudio-daemon 16.1 ha8d29e2_3 conda-forge
pure_eval 0.2.2 pyhd3eb1b0_0
pyarrow 11.0.0 py310h633f555_13_cpu conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pydantic 1.10.7 py310h1fa729e_0 conda-forge
pydata-sphinx-theme 0.13.3 pyhd8ed1ab_0 conda-forge
pygments 2.14.0 pyhd8ed1ab_0 conda-forge
pymannkendall 1.4.3 pyha21a80b_0 conda-forge
pyopenssl 23.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyproj 3.5.0 py310hb814896_1 conda-forge
pyqt 5.15.7 py310hab646b1_3 conda-forge
pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge
pyshp 2.3.1 pyhd8ed1ab_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
pytest 7.3.1 pyhd8ed1ab_0 conda-forge
pytest-cov 4.0.0 pyhd8ed1ab_0 conda-forge
python 3.10.9 he550d4f_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-dotenv 1.0.0 pyhd8ed1ab_0 conda-forge
python-eccodes 1.5.1 py310h0a54255_0 conda-forge
python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2022.7.1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py310h5764c6d_5 conda-forge
pyzmq 23.2.0 py310h6a678d5_0
qt-main 5.15.8 h5c52f38_9 conda-forge
rasterio 1.3.6 py310h30d39c3_1 conda-forge
re2 2023.02.02 hcb278e6_0 conda-forge
readline 8.2 h8228510_1 conda-forge
regionmask 0.10.0 pyhd8ed1ab_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
rich 13.3.5 pyhd8ed1ab_0 conda-forge
rioxarray 0.14.1 pyhd8ed1ab_0 conda-forge
rtree 1.0.1 py310hbdcdc62_1 conda-forge
s2n 1.3.40 h3358134_0 conda-forge
scikit-learn 1.2.2 py310h209a8ca_0 conda-forge
scipy 1.10.1 py310h8deb116_0 conda-forge
setuptools 67.6.0 pyhd8ed1ab_0 conda-forge
shapely 2.0.1 py310h056c13c_1 conda-forge
shellingham 1.5.1 pyhd8ed1ab_0 conda-forge
sip 6.7.7 py310heca2aa9_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
snowballstemmer 2.2.0 pyhd8ed1ab_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge
sparse 0.14.0 pyhd8ed1ab_0 conda-forge
sphinx 6.2.1 pyhd8ed1ab_0 conda-forge
sphinx-autoapi 2.1.0 pyhd8ed1ab_0 conda-forge
sphinxcontrib-applehelp 1.0.4 pyhd8ed1ab_0 conda-forge
sphinxcontrib-devhelp 1.0.2 py_0 conda-forge
sphinxcontrib-dotnetdomain 0.4 py_0 conda-forge
sphinxcontrib-golangdomain 0.2.0.dev0 py_0 conda-forge
sphinxcontrib-htmlhelp 2.0.1 pyhd8ed1ab_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.3 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.5 pyhd8ed1ab_2 conda-forge
sqlalchemy 2.0.12 py310h2372a71_0 conda-forge
sqlite 3.40.0 h4ff8645_0 conda-forge
stack_data 0.2.0 pyhd3eb1b0_0
structlog 22.3.0 pypi_0 pypi
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
tenacity 8.2.2 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tiledb 2.13.2 hd532e3d_0 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
tornado 6.2 py310h5764c6d_1 conda-forge
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
traitlets 5.7.1 py310h06a4308_0
typer 0.9.0 pyhd8ed1ab_0 conda-forge
types-pytz 2023.3.0.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
tzcode 2023a h0b41bf4_0 conda-forge
tzdata 2022g h191b570_0 conda-forge
ucx 1.14.0 ha0ee010_0 conda-forge
udunits2 2.2.28 hc3e0081_0 conda-forge
ukkonen 1.0.1 py310hbf28c38_3 conda-forge
unicodedata2 15.0.0 py310h5764c6d_0 conda-forge
unidecode 1.3.6 pyhd8ed1ab_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
virtualenv 20.21.1 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyhd3eb1b0_0
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
wrapt 1.15.0 py310h1fa729e_0 conda-forge
xarray 2023.4.2 pyhd8ed1ab_0 conda-forge
xcb-util 0.4.0 h166bdaf_0 conda-forge
xcb-util-image 0.4.0 h166bdaf_0 conda-forge
xcb-util-keysyms 0.4.0 h166bdaf_0 conda-forge
xcb-util-renderutil 0.3.9 h166bdaf_0 conda-forge
xcb-util-wm 0.4.1 h166bdaf_0 conda-forge
xerces-c 3.2.4 h8d71039_2 conda-forge
xesmf 0.7.1 pyhd8ed1ab_0 conda-forge
xkeyboard-config 2.38 h0b41bf4_0 conda-forge
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.8.4 h0b41bf4_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge
xorg-libxi 1.7.10 h7f98852_0 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xyzservices 2023.2.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zeromq 4.3.4 h2531618_0
zict 2.2.0 pyhd8ed1ab_0 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge

@malmans2
Copy link
Member

malmans2 commented Oct 22, 2024

Hi @Figas23,

It looks like you haven't been using the VM. More importantly, you're running an outdated version of the EQC software. The latest version now relies on earthkit-data, the official ECMWF library, whereas you're still using emohawk, which has been deprecated by ECMWF.

When we migrated to the new VM a few months ago, we ensured that all existing notebooks were cached to use earthkit. I recommend either updating your environment and clearing the cache or switching to the VM. One of the VM's benefits is that it provides the correct environment, so you don't have to deal with your own installations.

I plan to start working on this notebook later today or tomorrow. I immagine the data is not yet cached on the VM, so downloading everything could take some time.

@Figas23
Copy link
Author

Figas23 commented Oct 22, 2024

Hello @malmans2

Thank you for the suggestions. I will follow them.

@malmans2
Copy link
Member

Hi @Figas23,

I'm going through the code before I start working on it, and I noticed there's some interpolation of ERA5 data. Is that actually necessary? It seems like you’re only comparing time series and not calculating biases. Could we just cache the reduced ERA5 time series without interpolating?

@malmans2
Copy link
Member

Also, the notebook downloads era5 wind speed data, but those data are never used.
Can we remove those variables?

@Figas23
Copy link
Author

Figas23 commented Oct 23, 2024

Hey @malmans2

We change our approach and we won't be using wind speed data in this notebook. You can remove those variables

@malmans2
Copy link
Member

What about the interpolation?

I have a question about this plot:
image

The title mentions 'Minimum precipitation,' but the code is actually using the mean precipitation. Is this the intended behavior?

@malmans2
Copy link
Member

malmans2 commented Oct 23, 2024

One more questions. The maps compared are using different colorbars.
Is this the intended behaviour, or should they use the same colorbar such as the example below?
image

@Figas23
Copy link
Author

Figas23 commented Oct 23, 2024

Hello @malmans2
Regarding the minimum precipitation plot. In the code above that plot we extract the year with the minimum precipitation of the dataset (in this case its 2005) and then we compare the E-OBS and ERA5 precipitation values for that year.

Regarding the maps, yes, they are supposed to be using the same colorbar (this way is easier to compare for different seasons)

@malmans2
Copy link
Member

Regarding the minimum precipitation plot. In the code above that plot we extract the year with the minimum precipitation of the dataset (in this case its 2005) and then we compare the E-OBS and ERA5 precipitation values for that year.

OK, maybe the plot title is a little misleading as it's not obvious you are comparing the mean values, not the max/min values.

Regarding the maps, yes, they are supposed to be using the same colorbar (this way is easier to compare for different seasons)

OK, I assume the same applies to the bins used to build the PDF? At the moment, your code uses different bins as it depends on the max/min values of each single plot.

@Figas23
Copy link
Author

Figas23 commented Oct 23, 2024

Yes we still have to work on titles, descriptions and comments.

Yes the same applies to the bins for the PDF

@malmans2
Copy link
Member

malmans2 commented Oct 23, 2024

OK, I think I have everything I need.
I'll send you a first version later today with everything but the ERA5 comparison.
I will try to download ERA5 overnight.

I still need a couple of info about the ERA5 processing though:

  1. Can we get rid of the interpolation? Interpolations are computationally expensive, but it looks like you don't need it as you are comparing timeseries only.
  2. Can you please confirm that you need ERA5 at 3h frequency as in your NB? Looks like the E-OBS dataset has daily frequency.

@malmans2
Copy link
Member

Actually, another question. Some of the analyses use spatial weighted means, whereas others use regular means (e.g., the last section). Is this the intended behavior? I.e., do we need to cache both unweighted and weighted means?

@malmans2
Copy link
Member

Also, the section where the data is organised in periods uses data from 1951 to 2020, whereas the timeseries used after is from 1950 to 2020. Is this done on purpose? Should they use the same time periods? I.e., can we parametrise it instead of hardcoding it?

@malmans2
Copy link
Member

I have completed the first draft of the template. A few comments:

  1. The ERA5 data currently in the template is a placeholder, not the actual data. Before proceeding with downloading and processing the real ERA5 data, could you please confirm whether interpolation should be skipped and what ERA5 frequency we should use?
  2. If you notice any discrepancies compared to your results, it's because I’ve applied spatial weighted means and ensured consistent time periods (e.g., starting from 1951). It's quite clear in the bar plot.

The notebook is already cached on the VM:

Let me know!

@Figas23
Copy link
Author

Figas23 commented Oct 24, 2024

Hey @malmans2

Regarding your questions:

  1. Sorry but I didn't fully understand the question but we need daily ERA5 frequency to match that of E-OBS, and if this feasible before downloading it would be perfect. Regarding interpolation: we need to match the ERA5 grid to that of EOBS as well to compute the climatology maps.
    Would this be ok?

  2. Ok, thanks.

Meanwhile I took a look at the notebook and it looks great and all the analysis we wanted to do were there.

Let me know.

@malmans2
Copy link
Member

malmans2 commented Oct 24, 2024

we need daily ERA5 frequency to match that of E-OBS

Why do you have this line in your code? "time": [f"{hour:02d}:00" for hour in range(0, 24, 3)]
Can we just use "time": "00:00"? This would reduce the amount of data downloaded by a factor of 8.

we need to match the ERA5 grid to that of EOBS as well to compute the climatology maps

This is not implemented in your notebook yet, right?

@Figas23
Copy link
Author

Figas23 commented Oct 24, 2024

Well @malmans2 our idea was to download all available timestamps and then resample by day in order to get sum of the precipitation for a given day. We do it using this code: ds_ERA5 = ds_ERA5['tp'].resample(time='1D').sum(dim='time')

It is already implemented. Its just an interpolation so that the 2 datasets E-OBS and ERA5 have matching resolution:
ds_ERA5_interpolation = ds_ERA5.interp_like(ds_EOBS_rr)

@malmans2
Copy link
Member

We do it using this code: ds_ERA5 = ds_ERA5['tp'].resample(time='1D').sum(dim='time')

I see. Are you aware of this dataset? https://cds.climate.copernicus.eu/datasets/derived-era5-single-levels-daily-statistics
However, it's not obvious to me if accumulated fields are available. I will check with ECMWF, otherwise I will use your implementation.

It is already implemented. Its just an interpolation so that the 2 datasets E-OBS and ERA5 have matching resolution:
ds_ERA5_interpolation = ds_ERA5.interp_like(ds_EOBS_rr)

Yes, I’ve seen the interpolation, but my question is: can we avoid it?
Why is it necessary to interpolate the data if you’re not plotting ERA5 maps or computing biases?
Unless I’m missing something, aren’t you only using ERA5 for plotting the time series and bars? Wouldn’t it be computationally more efficient and accurate to compute the spatially weighted means of ERA5 without interpolating it to the E-OBS grid?

For example, where in your code do you need climatology maps? If there’s another reason for it, that’s fine, and I’ll try to optimise it. Otherwise, I think we can simply cache the spatially weighted means of ERA5 without interpolation.

@malmans2
Copy link
Member

ECMWF informed me that daily sums haven't been implemented yet, but they are working on it. In the meantime, let’s compute them ourselves.

@malmans2
Copy link
Member

Hi @Figas23,

I've finalised the template. Here is a few comments:

  • Since our software works well with GRIB format, I recommend using GRIB over NetCDF wherever possible. GRIB is the native format of ERA5, and requesting NetCDF can significantly slow down the CDS processing. Additionally, converting to NetCDF may cause technical issues with some datasets. The template is using GRIB.
  • I've removed the interpolation step. I wasn't sure if you were OK with this, so please let me know if you need the interpolation and I'll update the template.
  • ERA5 provides both land and ocean data, while it seems E-OBS only covers land. I don't see any masking in your code, so I haven’t implemented it in the template.

Everything is already cached on the VM!

@malmans2
Copy link
Member

malmans2 commented Nov 5, 2024

Hi @Figas23,

This issue has been inactive for a while. Do you need any further assistance, or should I go ahead and close it?

@anolive
Copy link

anolive commented Jan 2, 2025

Hi @malmans2 , we are revising this notebook now a implementing it also for e-obs and era5 wind, so maybe leave it open a while longer to keep the thread, if ok. Thank you, and happy new year! Ana and @Figas23

@malmans2
Copy link
Member

malmans2 commented Jan 8, 2025

Hi there! Happy new year!

OK, no problem. The reason I’m keen on closing issues is that we use GitHub issue stats to track progress and submit reports for our deliverables.

Sometimes, issues become stale simply because evaluators don’t confirm they can be closed after the work is done. But we’ve successfully closed most issues so far, so we’re happy to leave this one open a bit longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants