Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add missing arch_tars dependencies #3319

Open
wants to merge 11 commits into
base: develop
Choose a base branch
from
Open

Add missing arch_tars dependencies #3319

wants to merge 11 commits into from
+107 −25

Conversation

AntonMFernando-NOAA
Copy link
Contributor

@AntonMFernando-NOAA AntonMFernando-NOAA commented Feb 12, 2025 •
edited
Loading

Description

  • The gfs_arch_tars job currently does not depend on gempak jobs, even though it archives data produced by them. This PR will introduce that dependency. Additionally, there are several missing dependencies for cleanup when the arch_tar job is not executed. Nearly all of the job's dependencies need to be replicated for cleanup in case arch_tar doesn't run. This PR will address this problem as well.

    Resolves Missing gfs_arch_tars dependencies #3294

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

  • Cycled test on WCOSS2

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

WalterKolczynski-NOAA
WalterKolczynski-NOAA requested changes Feb 13, 2025
View reviewed changes
env/HERA.env Show resolved Hide resolved
workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved
@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Feb 13, 2025
Open
22 tasks
@AntonMFernando-NOAA AntonMFernando-NOAA self-assigned this Feb 13, 2025
@AntonMFernando-NOAA AntonMFernando-NOAA marked this pull request as ready for review February 13, 2025 23:20
@DavidHuber-NOAA
Copy link
Contributor

I ran a pair of tests on Hera, both C96_atm3dvar_extended with DO_GEMPAK=NO (there are some issues with gempak jobs on Hera). One of the tests ran with HPSSARCH=YES and one with HPSSARCH=NO to test the dependencies. Visual inspection of the resulting XMLs show that the arch_vrfy, arch_tars, and cleanup jobs all have the correct dependencies.

I then made some adjustments to the XML to open the throttle to test if I overlooked any dependencies. This was achieved by changing the parameters cyclethrottle to 8 and taskthrottle to 125. All jobs succeeded on their first pass, the vrfy jobs all ran after all of the product jobs completed and before the metp and tars jobs, and the cleanup jobs all ran last for their associated RUN.

It would be good to repeat this operation with the other CI tests (especially the hybvar tests). When CI testing for this PR is launched on Hercules and Hera, I will inspect the XMLs to verify all jobs have the correct dependencies.

@WalterKolczynski-NOAA WalterKolczynski-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Feb 14, 2025
@WalterKolczynski-NOAA
Copy link
Contributor

It would be good to repeat this operation with the other CI tests (especially the hybvar tests). When CI testing for this PR is launched on Hercules and Hera, I will inspect the XMLs to verify all jobs have the correct dependencies.

I'm doing Hercules first, as that seems to be the machine where the cleanup was running prematurely the most often.

@emcbot emcbot added CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress and removed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Hercules-Building **Bot use only** CI testing is cloning/building on Hercules labels Feb 14, 2025
@DavidHuber-NOAA
Copy link
Contributor

Looking through the C96C48_hybatmDA, C48mx500_3DVarAOWCDA, and C48mx500_hybAOWCDA XMLs generated by the Hercules CI, I think all of the dependencies are correct.

@emcbot emcbot added CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Feb 15, 2025
@emcbot
Copy link

emcbot commented Feb 15, 2025

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3319


Experiment C48_ATM_342a7861 Completed 1 Cycles: *SUCCESS* at Fri Feb 14 15:09:22 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_342a7861 Completed 1 Cycles: *SUCCESS* at Fri Feb 14 15:46:10 CST 2025
Experiment C48mx500_hybAOWCDA_342a7861 Completed 2 Cycles: *SUCCESS* at Fri Feb 14 15:52:01 CST 2025
Experiment C96C48_hybatmDA_342a7861 Completed 3 Cycles: *SUCCESS* at Fri Feb 14 16:46:41 CST 2025
Experiment C96_atm3DVar_342a7861 Completed 3 Cycles: *SUCCESS* at Fri Feb 14 16:52:36 CST 2025
Experiment C48_S2SW_342a7861 Completed 1 Cycles: *SUCCESS* at Fri Feb 14 17:10:53 CST 2025
Experiment C48mx500_3DVarAOWCDA_342a7861 Completed 2 Cycles: *SUCCESS* at Fri Feb 14 18:17:46 CST 2025
Experiment C48_S2SWA_gefs_342a7861 Completed 1 Cycles: *SUCCESS* at Fri Feb 14 18:24:25 CST 2025

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul aerorahul is a code owner

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA is a code owner

Assignees

@AntonMFernando-NOAA AntonMFernando-NOAA

Labels
CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Missing gfs_arch_tars dependencies
4 participants
@AntonMFernando-NOAA @DavidHuber-NOAA @WalterKolczynski-NOAA @emcbot