Use GPS ref. time to sync profiles

[fpavogt]

Describe the change
@thaifoon makes the following comment:

I would urge to switch the profile synchronization, whenever possible, to GPS time-based synchronization. I am becoming more and more convinced, that profile synchronization based on the analysis of the measurands is biasing the results in the end. As far as I have seen, every radiosonde is providing relative time referencing with available conversion to a universal timeline, be it GPS time, UTC or local.

To implement this in dvas, we need to include those reference times in the metadata.

Open questions:

1. For GDPs, what are the attributes we have to extract ?
2. For constructor data, will this reference time be included in the data file, or should it be added to the metadata .yml ?

[thaifoon]

GDP attributes:
RS41GDP - :g.Measurement.StartTime
RS92GDP - :g.Ascent.StartTime
iMS100GDP - :g.Measurement.StartTime
M10GDP - :g.Measurement.StartTime

NB! If any of the manufacturers is using GPS time, rather than UTC, leap seconds have to be added to the timeline, following this procedure:

def leap_seconds_generator(launch_date):
leap_seconds = 0

#if launch_date > dt.datetime(1981, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1982, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1983, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1985, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1988, 01, 01):
#    leap_seconds = leap_seconds + 1
    
#if launch_date > dt.datetime(1990, 01, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1991, 01, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1992, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1993, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1994, 07, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1996, 01, 01):
#    leap_seconds = leap_seconds + 1
#if launch_date > dt.datetime(1997, 07, 01):
#    leap_seconds = leap_seconds + 1
if launch_date > dt.datetime(1999, 1, 1):
    leap_seconds = leap_seconds + 13
    
if launch_date > dt.datetime(2006, 1, 1):
    leap_seconds = leap_seconds + 1
if launch_date > dt.datetime(2009, 1, 1):
    leap_seconds = leap_seconds + 1

if launch_date > dt.datetime(2012, 7, 1):
    leap_seconds = leap_seconds + 1
if launch_date > dt.datetime(2015, 7, 1):
    leap_seconds = leap_seconds + 1
if launch_date > dt.datetime(2016, 12, 31):
    leap_seconds = leap_seconds + 1
    
return leap_seconds

[fpavogt]

Brief update for the record: things are not as straightforward as one may have hoped. Using GPS times to sync GDP profiles leads to clear profile mismatches for all variables. This strongly suggests a problem with the reference GPS times, rather than some sensor lag (which ought not to be the same for all variables).

Before a dvas implementation can be completed, the exact sync algorithm must be clarified/verified.

[thaifoon]

I tested the consistency of RS92 and RS41 seconds to GPS seconds in the raw data of the soundings. I confirm the both radiosondes consistently representing the correct GPS/UTC seconds in the GDP.

[fpavogt]

Hey @thaifoon, thanks for the update ! This is interesting, because as far as I can tell, the shifts between RS41 and/or RS92 GDP profiles only (i.e. no other GDPs) still remains problematic if dvas uses the GPS times.

I'm specifically looking at the LIN flight 154639 from 27.04.2022 @ 08:00 UTC. The rig contained 2xRS41, 1xRS92, 2xM10, and 2xiMS-100.

If I just look at the RS41 & RS92, I obtain the following shifts from cross-correlating the gph:

Sync. shifts from alt (just aligning the point at 5000.0 m): [13, 17, 1] = [12, 16, 0]
Sync. shifts from "gph": [14, 16, 2] = [12, 14, 0]
Sync. shifts from starttime: [7, 3, 6] = [4, 0, 3]

Mids: ['RS41[GDP]', 'RS41[GDP]', 'RS92[GDP]']

If we ignore the RS92 and look just at the RS41, the first two methods suggest shifts of [0, +4] and [0, +2], whereas the last one gives [0, -4]. This is a difference of 6-8s ... too large to be ok.

Conclusion: something is happening between the raw data and the GDP stage ?

[fpavogt]

@thaifoon: just realized that there may be something odd with this flight 154639 - I'm seeing some residual shifts for the RS92. The values I quoted above seem ok for GPH, but not for the other variables ... Maybe not the best example to look at closely at first ?

[fpavogt]

Brief update: looking more closely at the same flight, the GPS time sync appears to work fine for the iMS-100 GDP. Specifically, both the GPS time and the cross-correlation approaches give the same shifts:

Sync. shifts from alt (5000.0): [0, 7]
Sync. shifts from "gph": [0, 7]
Sync. shifts from starttime: [0, 7]

The problem is thus restricted to the RS41 and RS92 GDPs. A formal bug report has been submitted to the GRUAN LC.

[fpavogt]

Adding the wontfix tag. GPS time synchronization does not work for several UAII participants and the RS41 GDP. This will be documented in the final report. gph synchronization is the only viable option.