Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Leap seconds #10

Merged
merged 57 commits into from
Sep 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
68e29a5
Draft PR for lossy compression with @cofinoa
czender Apr 11, 2024
4e2e097
Finish most draft text in section. Change family from required to opt…
czender Apr 12, 2024
7fe0c23
Start formatting
czender Apr 12, 2024
e50a1d8
Finish per-variable and description sections. Add two examples.
czender Apr 17, 2024
cd122e7
Conformance doc
czender Apr 17, 2024
e165e95
Update to replace "lossy compression" by "quantization" container var…
czender Jun 26, 2024
743519f
Include all quantization algorithms in Appendix A.
czender Jun 27, 2024
4d0730e
Address/merge textual suggestions from JGs PR review
czender Jun 27, 2024
3a8ebfa
Merge branch 'main' into csz_qnt
czender Jun 27, 2024
d92e51d
Fix fonts/wording in figure captions
czender Jun 27, 2024
1ce9eef
Update implementation attributes in example to realistic version numbers
czender Jun 27, 2024
75b4f49
Address second round of JGs suggestions
czender Jun 28, 2024
010e87a
implement third round of JG suggestions
czender Jul 2, 2024
03fb0a7
Update conformance.adoc
czender Jul 26, 2024
6f138ad
Update ch08.adoc
czender Jul 26, 2024
6b6c329
Update ch08.adoc
czender Jul 26, 2024
761ba8b
Update ch08.adoc
czender Jul 26, 2024
5a08a6a
Update ch08.adoc
czender Jul 26, 2024
03e3771
Update ch08.adoc
czender Jul 26, 2024
0bc82a1
Update ch08.adoc
czender Jul 26, 2024
05ac07a
Update ch08.adoc
czender Jul 26, 2024
de29a87
Update conformance.adoc
czender Jul 26, 2024
0d8354e
Update ch08.adoc
czender Jul 26, 2024
7cc0412
Update ch08.adoc
czender Jul 26, 2024
9ccd62a
Update ch08.adoc
czender Jul 26, 2024
dd47793
Update ch08.adoc
czender Jul 26, 2024
91bb467
Update ch08.adoc
czender Jul 26, 2024
93ad109
Update ch08.adoc
czender Jul 26, 2024
5abbb1b
Update ch08.adoc
czender Jul 26, 2024
afb0fe5
Update ch08.adoc
czender Jul 26, 2024
e767aae
Update ch08.adoc
czender Jul 26, 2024
ef91344
Update ch08.adoc
czender Jul 26, 2024
ac96f72
Update ch08.adoc
czender Jul 26, 2024
f4f80a4
Update ch08.adoc
czender Jul 30, 2024
9215cde
Update ch08.adoc
czender Jul 30, 2024
b950ad2
Remove overly meta sentence flagged by DH.
czender Jul 30, 2024
e6b7982
Merge branch 'main' into csz_qnt
czender Aug 14, 2024
b467fcf
Implement latest suggestions by JG to resolve DH review points regard…
czender Aug 14, 2024
85a57c1
Fix two small typos
czender Aug 14, 2024
c0bc8ac
clarification of time coordinate requirements
JonathanGregory Sep 3, 2024
41972f7
remove Mountain Daylight Time
JonathanGregory Sep 4, 2024
3424905
leap seconds
davidhassell Sep 4, 2024
8ebd988
leap seconds
davidhassell Sep 4, 2024
0f51b7f
leap seconds
davidhassell Sep 4, 2024
356cb8b
leap seconds (2)
davidhassell Sep 5, 2024
9d3c007
Merge branch 'main' into csz_qnt
czender Sep 5, 2024
2539e66
Merge pull request #519 from czender/csz_qnt
JonathanGregory Sep 6, 2024
ef4d99e
appendix A, and some text moving
davidhassell Sep 6, 2024
b8bb755
Merge branch 'leap-seconds' of https://github.com/davidhassell/cf-con…
JonathanGregory Sep 7, 2024
a0fffb4
Merge branch 'issue166' of https://github.com/JonathanGregory/cf-conv…
JonathanGregory Sep 7, 2024
de06f38
David's leap-second branch and my issue 166
JonathanGregory Sep 7, 2024
66bc6d8
updates wrt David's versioN
JonathanGregory Sep 8, 2024
b5f0e91
Merge pull request #539 from JonathanGregory/leap-seconds
JonathanGregory Sep 8, 2024
5a19e97
Update .gitignore
JonathanGregory Sep 8, 2024
33b8014
Update .gitignore
JonathanGregory Sep 8, 2024
76a4710
reverse b5f0e91 at 1134 UTC today, done by accident
JonathanGregory Sep 8, 2024
eab7d70
Merge pull request #540 from JonathanGregory/recover240908
JonathanGregory Sep 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 33 additions & 1 deletion appa.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ See <<appendix-grid-mappings>> for the grid mapping attributes, and <<appendix-m
The "Type" values are **S** for string, **N** for numeric, and **D** for the type of the data variable.
Each attribute may be used in any of the ways shown in its "Use" entry.
**G** indicates it can appear as a global attribute, and **Gr** as a group attribute.
For variable attributes, the possible values of "Use" are: **C** for variables containing coordinate data, **D** for data variables, **M** for geometry container variables, **Do** for domain variables, **BI** and **BO** for boundary variables (see <<cell-boundaries>> for the distinction between **BI** and **BO**), and **-** for variables with some other purpose.
For variable attributes, the possible values of "Use" are: **C** for variables containing coordinate data, **D** for data variables, **M** for geometry container variables, **Q** for quantization container variables, **Do** for domain variables, **BI** and **BO** for boundary variables (see <<cell-boundaries>> for the distinction between **BI** and **BO**), and **-** for variables with some other purpose.
CF does not prohibit any of these attributes from being attached to variables of different kinds from those listed as their "Use" in this table, but their meanings are not defined by CF if they are used in these other ways.
"Links" indicates the location of the attribute"s original definition (first link) and sections where the attribute is discussed in this document (additional links as necessary).

Expand Down Expand Up @@ -38,6 +38,12 @@ Attribute
If both **`scale_factor`** and **`add_offset`** attributes are present, the data are first scaled before the offset is added.
In cases where there is a strong constraint on dataset size, it is allowed to pack the coordinate variables (using add_offset and/or scale_factor), but this is not recommended in general.

| **`algorithm`**
| S
| Q
| <<quantization-variables>>, and <<quantization-algorithms-description>>
| Name of the quantization algorithm employed.

| **`ancillary_variables`**
| S
| D
Expand Down Expand Up @@ -200,6 +206,12 @@ Use in conjunction with **`flag_meanings`**.
| link:$$https://www.unidata.ucar.edu/software/netcdf/docs/attribute_conventions.html$$[NUG Appendix A, "Attribute Conventions"]
| List of the applications that have modified the original data.

| **`implementation`**
| S
| Q
| <<quantization-variables>>, and <<quantization-algorithms-description>>
| The name and version of the library or client software that performed the quantization with **`algorithm`**.

| **`instance_dimension`**
| S
| -
Expand Down Expand Up @@ -300,6 +312,26 @@ Allowed for auxiliary coordinate variables but not allowed for coordinate variab
| <<COARDS>>
| Direction of increasing vertical coordinate value.

| **`quantization`**
| S
| D
| <<quantization-variables>>
| Identifies a variable that defines a quantization algorithm and its provenance.

| **`quantization_nsb`**
| N
| D
| <<per-variable-quantization-attributes>>, and <<quantization-algorithms-description>>
| Specifies the number of significant bits retained in the IEEE mantissa of data quantized with the BitRound algorithm.
Use in conjunction with **`quantization`**.

| **`quantization_nsd`**
| N
| D
| <<per-variable-quantization-attributes>>, and <<quantization-algorithms-description>>
| Specifies the number of significant base-10 digits retained in the IEEE mantissa of data quantized with base-10 quantization algorithms.
Use in conjunction with **`quantization`**.

| **`references`**
| S
| G, D
Expand Down
8 changes: 6 additions & 2 deletions bibliography.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,15 @@
[bibliography]
=== References

- [[[CFDM]]] link:$$https://doi.org/10.5194/gmd-10-4619-2017$$[A data model of the Climate and Forecast metadata conventions (CF-1.6) with a software implementation (cf-python v2.1)]. Hassell, D., Gregory, J., Blower, J., Lawrence, B. N., and Taylor, K. E.: _Geosci. Model Dev._, 10, 4619-4646, 2017.
- [[[COARDS]]] link:$$https://ferret.pmel.noaa.gov/Ferret/documentation/coards-netcdf-conventions$$[Conventions for the standardization of NetCDF Files].
Sponsored by the "Cooperative Ocean/Atmosphere Research Data Service," a NOAA/university cooperative for the sharing and distribution of global atmospheric and oceanographic research data sets. May 1995.
- [[[DCG19]]] link:$$https://doi.org/10.5194/gmd-12-4099-2019$$[Evaluation of lossless and lossy algorithms for the compression of scientific datasets in netCDF-4 or HDF5 files]. Delaunay, X., A. Courtois, and F. Gouillon: _Geosci. Model Dev._, 12, 4099-4113, 2019.
- [[[FGDC]]] link:$$https://www.fgdc.gov/standards/projects/FGDC-standards-projects/metadata/base-metadata/v2_0698.pdf$$[Content Standard for Digital Geospatial Metadata].
Federal Geographic Data Committee, FGDC-STD-001-1998.
- [[[IEEE_754]]] link:$$https://doi.org/10.1109/IEEESTD.2019.8766229$$[IEEE Standard for Floating-Point Arithmetic], in _IEEE Std 754-2019 (Revision of IEEE 754-2008)_, 22 July 2019.
- [[[Kou21]]] link:$$https://doi.org/10.5194/gmd-14-377-2021$$[A note on precision-preserving compression of scientific data]. Kouznetsov, R.: _Geosci. Model Dev._, 14, 377-389, 2021.
- [[[KRD21]]] link:$$https://doi.org/10.1038/s43588-021-00156-2$$[Compressing atmospheric data into its real information content]. Klöwer, M., Razinger, M., Dominguez, J. J., Düben, P., and Palmer, T. N.: _Nat. Comput. Sci._, 1, 713-724, 2021.
- [[[NetCDF]]] link:$$https://doi.org/10.5065/D6H70CW6$$[NetCDF Software Package]. UNIDATA Program Center of the University Corporation for Atmospheric Research.
- [[[NUG]]] link:$$https://docs.unidata.ucar.edu/nug/current/index.html$$[The NetCDF User's Guide].
- [[[OGC_WKT-CRS]]] link:$$https://www.opengeospatial.org/standards/wkt-crs$$[OGC Well-known text representation of coordinate reference systems].
Expand All @@ -16,7 +20,7 @@ OGC document 12-063. 1st May 2015.
- [[[SCH02]]] link:$$https://doi.org/10.1175/1520-0493(2002)130<2459:ANTFVC>2.0.CO;2$$[A new terrain-following vertical coordinate formulation for atmospheric prediction models]. C Schaer, D Leuenberger, and O Fuhrer. 2002. _Monthly Weather Review_. 130. 2459-2480.
- [[[Snyder]]] link:$$https://doi.org/10.3133/pp1395$$[Map Projections: A Working Manual]. USGS Professional Paper 1395.
- [[[UDUNITS]]] link:$$https://doi.org/10.5065/D6KD1WN0$$[UDUNITS Software Package]. UNIDATA Program Center of the University Corporation for Atmospheric Research.
- [[[UGRID]]] link:$$https://ugrid-conventions.github.io/ugrid-conventions$$[UGRID Conventions for storing unstructured (or flexible mesh) data in netCDF files]
- [[[W3C]]] link:$$https://www.w3.org/$$[World Wide Web Consortium (W3C)].
- [[[XML]]] link:$$https://www.w3.org/TR/1998/REC-xml-19980210$$[Extensible Markup Language (XML) 1.0]. T. Bray, J. Paoli, and C.M. Sperberg-McQueen. 10 February 1998.
- [[[CFDM]]] link:$$https://doi.org/10.5194/gmd-10-4619-2017$$[A data model of the Climate and Forecast metadata conventions (CF-1.6) with a software implementation (cf-python v2.1)]. Hassell, D., Gregory, J., Blower, J., Lawrence, B. N., and Taylor, K. E.: _Geosci. Model Dev._, 10, 4619-4646, 2017.
- [[[UGRID]]] link:$$https://ugrid-conventions.github.io/ugrid-conventions$$[UGRID Conventions for storing unstructured (or flexible mesh) data in netCDF files]
- [[[Zen16]]] link:$$https://doi.org/10.5194/gmd-9-3199-2016$$[Bit Grooming: Statistically accurate precision-preserving quantization with compression, evaluated in the netCDF Operators (NCO, v4.4.8+)]. Zender, C. S.: _Geosci. Model Dev._, 9, 3199-3211, 2016.
4 changes: 3 additions & 1 deletion ch01.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,8 @@ out-of-group reference:: A reference to a variable or dimension that is not cont

path:: Paths must follow the UNIX style path convention and may begin with either a '/', '..', or a word.

quantization variable:: A variable used as a container for attributes that define a specific quantization algorithm. The type of the variable is arbitrary since it contains no data.

recommendation:: Recommendations in this convention are meant to provide advice that may be helpful for reducing common mistakes.
In some cases we have recommended rather than required particular attributes in order to maintain backwards compatibility with COARDS.
An application must not depend on a dataset's adherence to recommendations.
Expand Down Expand Up @@ -226,4 +228,4 @@ The UGRID conventions description is referenced from, rather than rewritten into
A summary indicating how UGRID relates to other parts of the CF conventions, and which features of UGRID are excluded from CF, can be found in <<mesh-topology-variables>>.
To reduce the chance of ambiguities arising from their accidental re-use, all of the UGRID standardized attributes are specified in <<appendix-mesh-topology-attributes>> and <<attribute-appendix>>.

The UGRID conventions have their own conformance document, which should be used in conjunction with the CF conformance document when checking the validity of datasets.
The UGRID conventions have their own conformance document, which should be used in conjunction with the CF conformance document when checking the validity of datasets.
Loading
Loading