Skip to content

Commit

Permalink
Fix returned status code in case of partial data availability
Browse files Browse the repository at this point in the history
Consider the case of downloading data for 6 months. This script does it month by
month. If data does not exist for the last month the return code was `-10`
before this fix, indicating that no data at all is available.

Now we only get a `-10` return code if indeed no data at all is available in the
requested period
  • Loading branch information
cpaulik committed May 2, 2022
1 parent 133bbd7 commit a4784e3
Showing 1 changed file with 8 additions and 1 deletion.
9 changes: 8 additions & 1 deletion src/ecmwf_models/era5/download.py
Original file line number Diff line number Diff line change
Expand Up @@ -237,6 +237,9 @@ def download_and_move(
c = cdsapi.Client(
error_callback=cds_status_tracker.handle_error_function)

all_status_codes = [] # Since we download month/month or day/day we need to
# collect all the status codes to return a valid
# status code for the whole time period
pool = multiprocessing.Pool(1)
while curr_start <= enddate:
status_code = -1
Expand Down Expand Up @@ -326,6 +329,8 @@ def download_and_move(
),
)

all_status_codes.append(status_code)

curr_start = curr_end + timedelta(days=1)
pool.close()
pool.join()
Expand All @@ -341,7 +346,9 @@ def download_and_move(
if os.path.exists(weightspath):
os.unlink(weightspath)

return status_code
# if any of the sub-periods was successful we want the function to return 0
consolidated_status_code = max(all_status_codes)
return consolidated_status_code


def parse_args(args):
Expand Down

0 comments on commit a4784e3

Please sign in to comment.