Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ZTD bufr2ioda python tool and "obs space" yaml in rrfsv2-workflow #299

Open
wants to merge 19 commits into
base: develop
Choose a base branch
from

Conversation

HuiLiu-NOAA
Copy link
Collaborator

@HuiLiu-NOAA HuiLiu-NOAA commented Feb 27, 2025

This PR adds a bufr2ioda python tool and an "obs space" of jedivar yaml for ZTD obs.

The ZTD bufr2ioda python tool converts RAP hourly gpsipw Bufr data into the ZTD IODA files for use in the RRFSv2 workflow.  The .json file is generic with analysis time only and should work for all bufr2ioda python converters.
The initial "obs space" part is for assimilation of ZTD observations using the existing UKMO local operator. More QC filters would be added later.

@HuiLiu-NOAA HuiLiu-NOAA self-assigned this Feb 27, 2025
@HuiLiu-NOAA HuiLiu-NOAA changed the title Add bufr2ioda python tool and "obs space" yaml for ZTD Add bufr2ioda python tool and "obs space" yaml for ZTD in rrfsv2 workflow Feb 27, 2025
@HuiLiu-NOAA HuiLiu-NOAA changed the title Add bufr2ioda python tool and "obs space" yaml for ZTD in rrfsv2 workflow Add bufr2ioda python tool and "obs space" yaml for ZTD in rrfsv2-workflow Feb 27, 2025
Copy link
Contributor

@SamuelDegelia-NOAA SamuelDegelia-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add distribution and obs localizations sections to gnss_zenithTotalDelay.yaml? Those will be needed for LETKF. See here and here for examples.

Copy link
Collaborator

@delippi delippi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, @HuiLiu-NOAA! It is my understanding that this PR is meant to get the yaml established in RDASApp and that there will be more updates to come. I think there are more QC filters that will need to go in, but this is a great starting point! I do have a few comments/suggestions for its current form.

  • Can you refer to some of the validated yamls in RDASApp and follow that style? This is just my own personal preference, but I would like to see a single line (possibly at most two lines) comment before each filter. For example, you can call your step 1 # Pre-online regional domain check (latitude) and # Pre-online regional domain check (longitude) respectively.
  • I believe we're going to stop (as of yesterday or so) naming any netcdf files with the .nc4 extension. We should just use .nc.
  • The jdiag name will be renamed, but we need to hash that out with PR284.

SamuelDegelia-NOAA and others added 5 commits February 28, 2025 12:28
This json file should be generic for all bufr2ioda python converters.
add vertical coordinate option for FV3-JEDI.
@HuiLiu-NOAA HuiLiu-NOAA changed the title Add bufr2ioda python tool and "obs space" yaml for ZTD in rrfsv2-workflow Add ZTD and SATWND bufr2ioda python tool and "obs space" yaml in rrfsv2-workflow Mar 1, 2025
@HuiLiu-NOAA HuiLiu-NOAA changed the title Add ZTD and SATWND bufr2ioda python tool and "obs space" yaml in rrfsv2-workflow Add ZTD and SATWND bufr2ioda python tool and ZTD "obs space" yaml in rrfsv2-workflow Mar 1, 2025
@HuiLiu-NOAA HuiLiu-NOAA changed the title Add ZTD and SATWND bufr2ioda python tool and ZTD "obs space" yaml in rrfsv2-workflow Add ZTD and AMV bufr2ioda python tool and ZTD "obs space" yaml in rrfsv2-workflow Mar 1, 2025
@ShunLiu-NOAA
Copy link

@HuiLiu-NOAA It appears the goal of this PR is not clear enough. It is trying to solve a couple of issues in one PR. Let's discuss this and find a better way to commit these changes.

@HuiLiu-NOAA
Copy link
Collaborator Author

@HuiLiu-NOAA It appears the goal of this PR is not clear enough. It is trying to solve a couple of issues in one PR. Let's discuss this and find a better way to commit these changes.

Moved the slightly revised SATWND python tool to a new separated PR.

@HuiLiu-NOAA HuiLiu-NOAA changed the title Add ZTD and AMV bufr2ioda python tool and ZTD "obs space" yaml in rrfsv2-workflow Add ZTD bufr2ioda python tool and "obs space" yaml in rrfsv2-workflow Mar 2, 2025
remove these lines not necessary for ZTD obs.
Copy link
Collaborator

@delippi delippi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you planning to add more QC filters? I can think of a couple that you may want to try.

@HuiLiu-NOAA
Copy link
Collaborator Author

HuiLiu-NOAA commented Mar 7, 2025

Are you planning to add more QC filters? I can think of a couple that you may want to try.

Yes, more QC filters will be added as the evaluation continues. Your suggestions are very welcome.

@delippi
Copy link
Collaborator

delippi commented Mar 10, 2025

Are you planning to add more QC filters? I can think of a couple that you may want to try.

Yes, more QC filters will be added as the evaluation continues. Your suggestions are very welcome.

You don't have to add these right now since you're just getting the yaml established and you plan to do more testing, but here are a few of my thoughts. These are based on what I have in the conventional obs yamls which I think should still apply.

  • What kind of QualityMarker info comes with this data? I see you reject all with QM > 10. For the other data we reject QM > 3.
  • How is this data spaced in time? You might consider some time window filter that selects data only within say 15min of the analysis time.
  • Duplicate checking
  • Error inflation
  • A sanity check to make sure the data falls within some expected range
  • More robust gross error checks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants