Remove Direct Linking to COM from DATA for wavepostsbs Job
#3303
Conversation
AntonMFernando-NOAA
requested review from
aerorahul and
WalterKolczynski-NOAA
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
AntonMFernando-NOAA
changed the title
wavepostsbs Jobwavepostsbs Job
AntonMFernando-NOAA
requested review from
sbanihash
and removed request for
sbanihash
Co-authored-by: Rahul Mahajan <[email protected]>
Co-authored-by: Rahul Mahajan <[email protected]>
Co-authored-by: Rahul Mahajan <[email protected]>
AntonMFernando-NOAA
requested review from
JessicaMeixner-NOAA and
sbanihash
as code owners
|
@aerorahul Some checks of the last commit were not successful because the |
|
@aerorahul Ran two CI tests on HERCULES with the current branch. |
ush/wave_grib2_sbs.sh
Outdated
| cpfs "${outfile}.idx" "${com_dir}/${outfile}.idx" | ||
| echo "Copied ${outfile} and ${outfile}.idx from ${GRIBDATA} to COM" | ||
| else | ||
| echo "${outfile} and ${outfile}.idx not found in ${GRIBDATA} to copy to COM" |
There was a problem hiding this comment.
Should this be an error?
There was a problem hiding this comment.
@JessicaMeixner-NOAA What this does is copying the ${outfile} to the COM directory, after the GRIB packing program is run. We are trying to first run the GRIB packing program in the working directory and then move the grib and .idx files to the COM dir. Is this what you are concerned about?
There was a problem hiding this comment.
That's great! Thanks for these updates. What I'm concerned with is if the grib and .idx files do not exist, then perhaps that should be an error message and not just simply a " echo "${outfile} and ${outfile}.idx not found" ?
There was a problem hiding this comment.
@JessicaMeixner-NOAA "${outfile} and ${outfile}.idx not found in ${GRIBDATA} to copy to COM Currently, this is the error message. What changes would you recommend for this?
There was a problem hiding this comment.
Shouldn't it say something like "ERROR: explanation" and then exit instead of continuing on?
There was a problem hiding this comment.
@JessicaMeixner-NOAA I will revise the script based on your suggestion. Thank you!
There was a problem hiding this comment.
@JessicaMeixner-NOAA I reviewed the script again, and it turns out the issue is addressed in the if blocks (L233-256). However, it only checks the COM files. So, I think your suggestion is correct.
| @@ -232,16 +255,14 @@ if [[ ! -s "${com_dir}/${outfile}.idx" ]]; then | |||
|
|
|||
| if [[ "${SENDDBN}" = 'YES' ]] && [[ ${outfile} != *global.0p50* ]]; then | |||
| set +x | |||
| echo " Alerting GRIB file as ${com_dir}/${outfile}" | |||
| echo " Alerting GRIB index file as ${com_dir}/${outfile}.idx" | |||
| echo " Alerting GRIB file as ${outfile}" | |||
There was a problem hiding this comment.
Should the com_dir be removed from this?
There was a problem hiding this comment.
@JessicaMeixner-NOAA After the L255 the grib and the .idx files should be in both the COM directory and the working directory, since the working directory is not removed yet. So, this should run without an error if those files exist in both dirs. If @aerorahul thinks this is not correct I can change it. But I think this should be correct because we need an alert to send for the files available in COM dir, while checking the files in the working directory.
There was a problem hiding this comment.
this is just an echo. its fine to be either. the crucial line is the dbn_alert that alerts the correct path of this file from com_dir.
ush/wave_grib2_sbs.sh
Outdated
| cpfs "${subfnam}.idx" "${com_dir}/${subfnam}.idx" | ||
| echo "Copied ${subfnam} and ${subfnam}.idx from ${GRIBDATA} to COM" | ||
| else | ||
| echo "${subfnam} and ${subfnam}.idx not found in ${GRIBDATA} to copy to COM" |
There was a problem hiding this comment.
Should this be an error?
There was a problem hiding this comment.
same above.
Description
In the current GFS version, some jobs directly write to COMOUT via links in the working directories. This approach is risky, as downstream jobs might receive incomplete files and fail. In order to resolve this issue direct linking to COM from DATA should be removed. This PR addresses this issue in the
wavepostsbsjob.Resolves Remove direct linking to
COMfromDATAforwavepostsbsjob #3302Refs [NCO Bug] Replace links with copy/move/rsync in workflow scripts #712
Refs Replace link w/ copy in the forecast job #3273
Type of change
Change characteristics
How has this been tested?
HERAChecklist