Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pattern #187

Merged
merged 53 commits into from
Oct 17, 2024
Merged
Changes from 1 commit
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
4486426
Merge pull request #164 from ACCESS-Community-Hub/prerelease
paolap Aug 23, 2024
55edbf5
Update meta.yaml to hardcode version
paolap Aug 23, 2024
98d3818
Update meta.yaml
paolap Aug 23, 2024
5d99fdb
Update mopper-conda-release.yaml
paolap Aug 23, 2024
95dfb70
Merge pull request #165 from ACCESS-Community-Hub/paolap-patch-1
paolap Aug 23, 2024
560d243
started adding geopotential calculation and transects
paolap Aug 28, 2024
11d663a
fixed tables all functionality issue #169 and #168
paolap Aug 28, 2024
ba18e7a
Merge branch 'tables' into transect
paolap Aug 28, 2024
caf0df4
added new pressure level and progress on geopot height calculation
paolap Aug 30, 2024
a3c15d3
added plev17 and new AM3_Amon.json table
paolap Aug 30, 2024
1e32dbf
added new version AM3 with intial table and mappings
paolap Aug 30, 2024
cbd35fd
fixing bug #171
paolap Aug 30, 2024
1474ce9
added calculation to var on model elvels for AM3 mappings
paolap Aug 30, 2024
7599b23
calculations docs added, fixed #171, #172, #173
paolap Sep 9, 2024
37bcd9d
fixing autodoc
paolap Sep 9, 2024
b0a3195
fixing autodoc 2
paolap Sep 9, 2024
f12662d
fixing autodoc 3
paolap Sep 9, 2024
5380a10
fixing autodoc 4
paolap Sep 9, 2024
04a66f5
fixing autodoc 5
paolap Sep 9, 2024
61c3d31
fixing autodoc 6
paolap Sep 9, 2024
45535b5
fixing autodoc 7
paolap Sep 9, 2024
fa98fd6
fixing autodoc 8
paolap Sep 9, 2024
47c0b10
fixing autodoc 9
paolap Sep 9, 2024
7d89062
update to calculations docstrings
paolap Sep 9, 2024
709f283
solved #176
paolap Sep 10, 2024
ef07c9e
fixed #177 #167
paolap Sep 17, 2024
fa19895
added test_setup_utils
paolap Sep 18, 2024
5ffeec5
added more tests to test_setuo_utils.py
paolap Sep 18, 2024
8f90822
added more tests to setup_utils, changed define_files as in #179
paolap Sep 19, 2024
5381d0d
merge with tables
paolap Sep 23, 2024
4f1a136
added test for define_file, possibly fixed issue with missing data su…
paolap Sep 23, 2024
20287f1
more adjustment to date ranges and pyproject data packaging
paolap Sep 24, 2024
9855688
fixed small bug
paolap Sep 24, 2024
9ba8160
fixed detection of time ranges and handling of dates, added tests and…
paolap Sep 26, 2024
745c82a
solved issue #153
paolap Sep 26, 2024
fcd3eca
reviewed an updated docs
paolap Sep 27, 2024
97b2cd4
fixed issue with data packaging
paolap Oct 2, 2024
f0cdb0a
updated docs for calculations
paolap Oct 2, 2024
f9beff1
added test for sum timeshot
paolap Oct 2, 2024
0e05387
solved #170 and progress on #180, rebuilt db after cleaning mappings …
paolap Oct 3, 2024
3eac399
solved #185 #182 #180
paolap Oct 8, 2024
678960d
checked that #91 is completely fixed
paolap Oct 8, 2024
97e6062
close: #183, #180, #185
paolap Oct 10, 2024
7fe1aca
updated docs for calculations
paolap Oct 10, 2024
5cbd519
adjuted calculations import in tests
paolap Oct 10, 2024
eff45de
fixed bugs after testing CM2 monthly
paolap Oct 11, 2024
81d624b
more fixes of small bugs
paolap Oct 14, 2024
c8d1530
minor adjustments
paolap Oct 16, 2024
3cf0cb5
change conda git tag
paolap Oct 16, 2024
451ccd2
weirdly it was getting very old xarray from defaults channel
paolap Oct 16, 2024
12f66d1
weirdly it was getting very old xarray from defaults channel 2
paolap Oct 16, 2024
2c4ddb2
weirdly it was getting very old xarray from defaults channel 3
paolap Oct 16, 2024
e177801
weirdly it was getting very old xarray from defaults channel 4
paolap Oct 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
more fixes of small bugs
paolap committed Oct 14, 2024
commit 81d624b9266edfb8759e6e4921402c0ac69da292
9 changes: 7 additions & 2 deletions docs/calculations.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Derived variables calculations
==============================
Calculating derived variables
=============================

Calculations are used to derive a variable from one or multiple inputs, to resample a variable to a new frequency or generally to modify a variable so it will match fully the corresponding definition in a CMOR table.

@@ -34,6 +34,7 @@ Contributing
------------
TBA


Available functions
-------------------

@@ -44,27 +45,31 @@ Atmosphere and aerosol
:undoc-members:
:show-inheritance:


Ocean
^^^^^
.. automodule:: mopper.calc_ocean
:members:
:undoc-members:
:show-inheritance:


SeaIce
^^^^^^
.. automodule:: mopper.calc_seaice
:members:
:undoc-members:
:show-inheritance:


Land
^^^^
.. automodule:: mopper.calc_land
:members:
:undoc-members:
:show-inheritance:


Other
^^^^^
.. automodule:: mopper.calc_utils
27 changes: 27 additions & 0 deletions docs/map_exp1.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
cmor_var;input_vars;calculation;units;dimensions;axes;frequency;realm;cell_methods;positive;cmor_table;version;vtype;size;nsteps;fpattern;long_name;standard_name
amdry;fld_s30i403;;kg m-2;time lat lon;longitude latitude time1;10minPt;atmos;area: time: point;;AUS2200_A10min;AUS2200;float32;22048000;720;umnsa_spec_;TOTAL COLUMN DRY MASS RHO GRID;
cl;fld_s00i265;level_to_height(var[0],levs=(0,66));%;time model_theta_level_number lat lon;longitude latitude alevel time1;1hrPt;atmos;area: time: point;;AUS2200_A1hr;AUS2200;float32;1543360000;120;umnsa_cldrad_;AREA CLOUD FRACTION IN EACH LAYER;cloud_area_fraction_in_atmosphere_layer
cli;fld_s00i012;level_to_height(var[0],levs=(0,66));kg kg-1;time model_theta_level_number lat lon;longitude latitude alevel time1;1hrPt;atmos;area: time: point;;AUS2200_A1hr;AUS2200;float32;1543360000;120;umnsa_cldrad_;QCF AFTER TIMESTEP;mass_fraction_of_cloud_ice_in_air
eow;fld_s03i232;;kg m-2 s-1;time_0 lat lon;longitude latitude time;1hr;land;area: time: mean;;AUS2200_A1hr;AUS2200;float32;22048000;120;umnsa_slv_;Evaporation flux from open sea;
rsds;fld_s01i235;;W m-2;time_0 lat lon;longitude latitude time;1hr;atmos;area: time: mean;down;AUS2200_A1hr;AUS2200;float32;22048000;120;umnsa_slv_;TOTAL DOWNWARD SURFACE SW FLUX;surface_downwelling_shortwave_flux_in_air
...
# Derived variables with matching version and frequency: Use with caution!;;;;;;;;;;;;;;;;;
hus24;fld_s00i010 fld_s00i408;plevinterp(var[0], var[1], 24);1;time model_theta_level_number lat lon_0;longitude latitude plev24 time1;1hrPt;atmos;area: time: point;;AUS2200_A1hrPlev;AUS2200;float32;1543360000;120;umnsa_mdl_;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
prw;fld_s30i404 fld_s30i403 fld_s30i405 fld_s30i406;var[0]-sum_vars(var[1:]);kg m-2;time lat lon;longitude latitude time1;10minPt;atmos;area: time: point;;AUS2200_A10min;AUS2200;float32;22048000;720;umnsa_spec_;TOTAL COLUMN DRY MASS RHO GRID;
...
# Variables definitions coming from different version;;;;;;;;;;;;;;;;;
ps;fld_s00i409;;Pa;time_1 lat lon;longitude latitude time1;3hrPt;atmos;area: time: point;;CMIP6_E3hrPt;CM2;float32;22048000;40;umnsaa_pvera;SURFACE PRESSURE AFTER TIMESTEP;surface_air_pressure
# Variables with different frequency: Use with caution!;;;;;;;;;;;;;;;;;
ps;fld_s00i409;;Pa;time_1 lat lon;longitude latitude time1;3hrPt;atmos;area: time: point;;CMIP6_E3hrPt;CM2;float32;22048000;40;umnsaa_pvera;SURFACE PRESSURE AFTER TIMESTEP;surface_air_pressure
...
# Variables matched using standard_name: complete mapping or discard!;;;;;;;;;;;;;;;;;
['rsut-AUS2200_A1hr', 'rsutaf-CMIP6_AERmon', 'rsut-CMIP6_Amon', 'rsut-CMIP6_CF3hr', 'rsut-CMIP6_CFday', 'rsut4co2-CMIP6_CFmon', 'rsut-CMIP6_CFsubhr', 'rsut-CMIP6_E1hrClimMon', 'rsut-CMIP6_E3hr', 'rsut-CMIP6_Esubhr'];fld_s01i208;;W m-2;time_4 lat lon;time_4 lat lon;1hrPt;atmos;area: time: point;;;;float32;22048000;120;umnsaa_pvera;OUTGOING SW RAD FLUX (TOA);toa_outgoing_shortwave_flux
...
# Derived variables: Use with caution!;;;;;;;;;;;;;;;;;
prw;fld_s30i404 fld_s30i403 fld_s30i405 fld_s30i406;var[0]-(var[1]+var[2]+var[3]);kg m-2;time lat lon;longitude latitude time;10minPt;atmos;area: time: point;;CMIP6_Amon;CM2;float32;22048000;720;umnsa_spec_;TOTAL COLUMN DRY MASS RHO GRID;
zg24;fld_s15i101 fld_s00i408;height_gpheight(var[0], pmod=var[1], levnum=24);m;time model_theta_level_number lat lon_0;longitude latitude plev24 time1;1hrPt;atmos;area: time: point;;AUS2200_A6hr;AUS2200;float32;1543360000;120;umnsa_mdl_;PRESSURE AT THETA LEVELS AFTER TS;air_pressure
...
# Variables without mapping;;;;;;;;;;;;;;;;;
fld_s03i051;fld_s03i051;;;time pseudo_level lat lon;time pseudo_level lat lon;10minPt;atmos;area: time: point;;;;float32;110240000;720;umnsa_spec_;COMBINED RESIST. TO EVAP ON TILES;
fld_s03i052;fld_s03i052;;;time pseudo_level lat lon;time pseudo_level lat lon;10minPt;atmos;area: time: point;;;;float32;110240000;720;umnsa_spec_;COEFFS RATIO FOR 1.5M T ON TILES;
...
30 changes: 3 additions & 27 deletions docs/mopdb_command.rst
Original file line number Diff line number Diff line change
@@ -82,33 +82,9 @@ The other groups of records require checking, as either the version or the frequ
.. _template example:
.. dropdown:: Example output of template

cmor_var;input_vars;calculation;units;dimensions;frequency;realm;cell_methods;positive;cmor_table;version;vtype;size;nsteps;filename;long_name;standard_name
agesno;fld_s03i832;;day;time pseudo_level_1 lat lon;mon;landIce land;area: time: mean;;CMIP6_LImon;CM2;float32;1880064;12;cw323a.pm;CABLE SNOW AGE ON TILES;age_of_surface_snow
amdry;fld_s30i403;;kg m-2;time lat lon;mon;atmos;area: time: mean;;CM2_mon;CM2;float32;110592;12;cw323a.pm;TOTAL COLUMN DRY MASS RHO GRID;
amwet;fld_s30i404;;kg m-2;time lat lon;mon;atmos;area: time: mean;;CM2_mon;CM2;float32;110592;12;cw323a.pm;TOTAL COLUMN WET MASS RHO GRID;atmosphere_mass_per_unit_area
ci;fld_s05i269;;1;time lat lon;mon;atmos;area: time: mean;;CMIP6_Amon;CM2;float32;110592;12;cw323a.pm;deep convection indicator;
...
# Derived variables with matching version and frequency: Use with caution!;;;;;;;;;;;;;;;;
baresoilFrac;fld_s03i317 fld_s03i395;extract_tilefrac(var[0],14,landfrac=var[1]);1;time pseudo_level_1 lat lon;mon;land;area: time: mean;;CMIP6_Lmon;CM2;float32;1880064;12;cw323a.pm;SURFACE TILE FRACTIONS;
c3PftFrac;fld_s03i317 fld_s03i395;extract_tilefrac(var[0],[1,2,3,4,5,6,8,9,11],landfrac=var[1]);1;time pseudo_level_1 lat lon;mon;land;area: time: mean;;CMIP6_Lmon;CM2;float32;1880064;12;cw323a.pm;SURFACE TILE FRACTIONS;
# Variables definitions coming from different version;;;;;;;;;;;;;;;;
rlntds;fld_s02i203;;W m-2;time lat lon;mon;ocean;area: time: mean; time: mean;;CMIP6_Omon;float32;110592;12;cw323a.pm;NET DN LW RAD FLUX:OPEN SEA:SEA MEAN;surface_net_downward_longwave_flux
rssntds;fld_s01i203;;W m-2;time lat lon;mon;ocean;area: time: mean; time: mean;;CM2_mon;float32;110592;12;cw323a.pm;NET DN SW RAD FLUX:OPEN SEA:SEA MEAN;surface_net_downward_shortwave_flux
# Variables with different frequency: Use with caution!;;;;;;;;;;;;;;;;
rlntds;fld_s02i203;;W m-2;time lat lon;mon;ocean;area: time: mean; time: mean;;CMIP6_Omon;float32;110592;12;cw323a.pm;NET DN LW RAD FLUX:OPEN SEA:SEA MEAN;surface_net_downward_longwave_flux
rssntds;fld_s01i203;;W m-2;time lat lon;mon;ocean;area: time: mean; time: mean;;CM2_mon;float32;110592;12;cw323a.pm;NET DN SW RAD FLUX:OPEN SEA:SEA MEAN;surface_net_downward_shortwave_flux
# Variables matched using standard_name: Use with caution!;;;;;;;;;;;;;;;;
['huss-CMIP6_3hr', 'hus-CMIP6_6hrLev', 'hus4-CMIP6_6hrPlev', 'hus27-CMIP6_6hrPlevPt', 'hus7h-CMIP6_6hrPlevPt', 'huss-CMIP6_6hrPlevPt', 'hus-CMIP6_Amon', 'huss-CMIP6_Amon', 'hus-CMIP6_CFday', 'hus-CMIP6_CFmon', 'hus-CMIP6_CFsubhr', 'huss-CMIP6_CFsubhr', 'hus-CMIP6_day', 'huss-CMIP6_day', 'hus-CMIP6_E3hrPt', 'hus7h-CMIP6_E3hrPt', 'hus-CMIP6_Eday', 'hus850-CMIP6_Eday', 'hus-CMIP6_EdayZ', 'hus-CMIP6_Emon', 'hus27-CMIP6_Emon', 'hussLut-CMIP6_Emon', 'hus-CMIP6_Esubhr', 'huss-CMIP6_Esubhr', 'huss-AUS2200_A10min', 'hus-AUS2200_A1hr', 'huss-AUS2200_A1hr', 'hus24-AUS2200_A1hrPlev', 'hus3-AUS2200_A1hrPlev'];;;1;time model_theta_level_number lat lon;mon;;area: time: mean;;CMIP6_Amon;;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
...
# Derived variables: Use with caution!;;;;;;;;;;;;;;;;
hus24;fld_s00i010 fld_s00i408;plevinterp(var[0], var[1], 24);1;time model_theta_level_number lat lon;mon;atmos;area: time: mean;;AUS2200_A1hrPlev;AUS2200;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
sifllatstop;fld_s03i234 fld_s00i031;maskSeaIce(var[0],var[1]);1;time lat lon;mon;seaIce;area: time: mean;up;AUS2200_A1hr;AUS2200;float32;110592;12;cw323a.pm;FRAC OF SEA ICE IN SEA AFTER TSTEP;sea_ice_area_fraction
theta24;fld_s00i004 fld_s00i408;plevinterp(var[0], var[1], 24);K;time model_theta_level_number lat lon;mon;atmos;area: time: mean;;AUS2200_A1hrPlev;AUS2200;float32;9400320;12;cw323a.pm;THETA AFTER TIMESTEP;air_potential_temperature
# Variables without mapping;;;;;;;;;;;;;;;;
fld_s00i211;;;1;time model_theta_level_number lat lon;mon;;area: time: mean;;;;float32;9400320;12;cw323a.pm;Convective cloud amount with anvil;
fld_s00i253;;;;time model_rho_level_number lat lon;mon;;area: time: mean;;;;float32;9400320;12;cw323a.pm;DENSITY*R*R AFTER TIMESTEP;
fld_s00i413;;;1;time pseudo_level lat lon;mon;;area: time: mean;;;;float32;552960;12;cw323a.pm;Sea ice concentration by categories;
...
.. literalinclude:: map_ex1.csv
:language: csv



Create an intake catalogue
6 changes: 4 additions & 2 deletions docs/output.rst
Original file line number Diff line number Diff line change
@@ -33,6 +33,7 @@ This folder will contain the following files:
* frequency - output variable frequency
* realm - output variable realm
* timeshot - cell_methods value for time: point, mean, sum, max, min
* axes - The cmor names of the axes used in variable definition
* tstart - datetime stamp for time range start
* tend - datetime stamp for time range end
* sel_start - datetime stamp to use for input file selection (start)
@@ -72,10 +73,11 @@ This folder will contain the following files:
| # for a list of packages
|
| module use /g/data/hh5/public/modules
| module load conda/analysis3-23.04
| module load conda/analysis3
| source mopper_env/bin/activate # if using conda option
|
| cd /g/data/ua8/Working/packages/ACCESS-MOPPeR
| mopper.py -i ashwed1980_config.yaml run
| mop run -c ashwed1980_config.yaml # --debug (uncomment to run in debug mode)
| echo 'APP completed for exp ashwed1980.'

* **experiment-id.json**
1 change: 1 addition & 0 deletions mappings/map_atmos_CM2.csv
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@ hur19uvgrid;fld_s30i206;;%;time pressure lat_v lon_u;longitude latitude plev19 t
hus;fld_s00i010;level_to_height(var[0]);1;time model_theta_level_number lat lon;longitude latitude alevel time;mon;atmos;area: time: mean;;CMIP6_CFmon;CM2;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev19 time;mon;atmos;area: time: mean;;CMIP6_Amon;CM2;float32;2101248;12;cw323a.pm;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev19 time;day;atmos;area: time: mean;;CMIP6_Eday;CM2;float32;884736;74772;cm000a.pd;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev8 time;day;atmos;area: time: mean;;CMIP6_day;CM2;float32;884736;74772;cm000a.pd;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;mon;atmos;area: time: mean;;CM2_mon;CM2;float32;110592;12;cw323a.pm;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;3hrPt;atmos;area: time: point;;CM2_3hr;CM2;float32;110592;578880;cm000a.p8;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;day;atmos;area: time: mean;;CM2_day;CM2;float32;110592;74772;cm000a.pd;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
1 change: 1 addition & 0 deletions mappings/map_atmos_ESM1.5.csv
Original file line number Diff line number Diff line change
@@ -27,6 +27,7 @@ hur19uvgrid;fld_s30i206;;%;time pressure lat_v lon_u;longitude latitude plev19 t
hus;fld_s00i010;level_to_height(var[0]);1;time model_theta_level_number lat lon;longitude latitude alevel time;mon;atmos;area: time: mean;;CMIP6_CFmon;ESM1.5;float32;9400320;12;cw323a.pm;SPECIFIC HUMIDITY AFTER TIMESTEP;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev19 time;mon;atmos;area: time: mean;;CMIP6_Amon;ESM1.5;float32;2101248;12;cw323a.pm;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev19 time;day;atmos;area: time: mean;;CMIP6_Eday;ESM1.5;float32;884736;74772;cm000a.pd;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
hus;fld_s30i295;;1;time pressure lat lon;longitude latitude plev8 time;day;atmos;area: time: mean;;CMIP6_day;ESM1.5;float32;884736;74772;cm000a.pd;SPECIFIC HUMIDITY ON P LEV/T GRID;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;mon;atmos;area: time: mean;;CM2_mon;ESM1.5;float32;110592;12;cw323a.pm;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;3hrPt;atmos;area: time: point;;CM2_3hr;ESM1.5;float32;110592;578880;cm000a.p8;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
huss;fld_s03i237;;1;time lat lon;longitude latitude time height1.5m;day;atmos;area: time: mean;;CM2_day;ESM1.5;float32;110592;74772;cm000a.pd;SPECIFIC HUMIDITY AT 1.5M;specific_humidity
Binary file modified src/mopdata/access.db
Binary file not shown.
Loading