Relax nanosecond datetime restriction in CF time decoding #9618
Conversation
|
Nice, mypy 1.12 is out and breaks our typing, 😭. |
Can we pin it in the CI temporarily? |
Yes, 1.11.2 was the last version. |
kmuehlbauer
force-pushed
the
any-time-resolution-2
branch
from
ca5050d to
f7396cf
Compare
|
This is now ready for a first round of review. I think this is already in a quite usable state. But no rush, this should be thoroughly tested. |
kmuehlbauer
added
topic-CF conventions
topic-cftime
run-upstream
|
Sounds good @kmuehlbauer! I’ll try and take an initial look this weekend. |
…ore/variable.py to use any-precision datetime/timedelta with autmatic inferring of resolution
…ocessing, raise now early
…t resolution, fix code and tests to allow this
Co-authored-by: Spencer Clark <[email protected]>
Co-authored-by: Spencer Clark <[email protected]>
# Conflicts: # xarray/coding/times.py # xarray/tests/test_backends.py
|
I think I've fixed up all issues due to manually merging. Hopefully this is now in a consistent state. I'm not sure if this is good enough, or if I should just rebase/squash this on latest master? Any thoughts appreciated. |
| # proleptic_gregorian and standard/gregorian are only equivalent | ||
| # if reference date and date range is >= 1582-10-15 | ||
| if calendar != "proleptic_gregorian": | ||
| if date < type(date)(1582, 10, 15): |
There was a problem hiding this comment.
Are you trying to handle subclasses of Timestamp here? If not, I would just write:
| if date < type(date)(1582, 10, 15): | |
| if date < pd.Timestamp(1582, 10, 15): |
| if date < type(date)(1582, 10, 15): | ||
| raise OutOfBoundsDatetime( | ||
| f"Dates before 1582-10-15 cannot be decoded " | ||
| f"with pandas using {calendar!r} calendar." |
There was a problem hiding this comment.
It's always good form to include the bad value in an error message:
| f"with pandas using {calendar!r} calendar." | |
| f"with pandas using {calendar!r} calendar: {date}" |
| new_time_unit: PDDatetimeUnitOptions = next(iter_unit) | ||
| if (np.unique(flat_num_dates % 1) > 0).any() and new_time_unit != "ns": | ||
| flat_num_dates, new_time_unit = _check_higher_resolution( | ||
| flat_num_dates * 1000, | ||
| iter_unit=iter_unit, | ||
| ) |
There was a problem hiding this comment.
The design of this function is a little strange, because it modifies the iterator object in-place and has a return value. The convention for Python is for all functions to only either modify inputs or have a return value.
Why not re-write it as a loop instead? e.g.,
for time_unit in time_units:
if good_enough(time_unit, flat_num_dates):
break
flat_num_dates = ...
return flat_num_dates, time_unit| # When receiving objects which pd.Series can't resolve by its own | ||
| # we try astype-conversion to "ns"-resolution for datetimes and pd.Timestamp. | ||
| if ( | ||
| values.dtype.kind == "O" | ||
| and as_series.dtype.kind == "O" | ||
| and as_series.size > 0 | ||
| and ( | ||
| isinstance(as_series[0], datetime | pd.Timestamp) | ||
| or pd.api.types.is_datetime64_dtype(as_series[0]) | ||
| ) | ||
| ): | ||
| as_series = as_series.astype("=M8[ns]") |
There was a problem hiding this comment.
This part feels a little strange to me and possibly unnecessary. When did this come up?
No need to rebase/squash manually! As long as this PR can be merged in GitHub, it will be squashed automatically |
whats-new.rstThis is another attempt to resolve #7493. This goes a step further than #9580.
The idea of this PR is to automatically infer the needed resolutions for decoding/
encodingand only keep the constraints pandas imposes ("s" - lowest resolution, "ns" - highest resolution). There is still the idea of adefault resolution, but this should only take precedence if it doesn't clash with the automatic inference. This can be discussed, though. Update: I've implementedtime-unit-kwarga first try to have default resolutionon decode, which will override the current inferred resolution only to higher resolution (eg.'s'->'ns'). To work towards #4490 the time decoding options (decode_timeanduse_cftimeare bundled withinCFDatetimeCoderwhich is distributed viadecode_timeskwarg.use_cftime-kwarg is deprecated.For sanity checking, and also for my own good, I've created a documentation page on time-coding in the internal dev section. Any suggestions (especially grammar) or ideas for enhancements are much appreciated.
There still might be room for consolidation of functions/methods (mostly in coding/times.py), but I have to leave it alone for some days. I went down that rabbit hole and need to relax, too 😬.
Looking forward to get your insights here, @spencerkclark, @ChrisBarker-NOAA, @pydata/xarray.
Todo:
time_units(where appropriate)CFDatetimeCoderas input fordecode_timeskwarg