Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[5pt] PR: Incorporate Lidar-Informed elevations for bridges #1406

Open
wants to merge 55 commits into
base: dev
Choose a base branch
from

Conversation

AliForghani-NOAA
Copy link
Collaborator

@AliForghani-NOAA AliForghani-NOAA commented Jan 18, 2025

This PR closes the issue #1242.
This PR incorporates lidar-derived elevations for OSM bridges into the FIM. The workflow consists of:

  • Extracting lidar-derived elevations for bridges
  • Calculating elevation differences between 3DEP DEMs and lidar elevations within bridge areas
  • Applying the elevation differences into the final REM

Additions

  • data/bridges/make_rasters_using_lidar.py. Generates bridge elevation raster files using lidar data, as described here. This script has to be run on Windows machines. This script must be run twice, each time with a different OSM bridge-lines GPKG file: once for the CONUS and separately for Alaska. The raster files are generated using EPSG:5070 for CONUS bridges and EPSG:3338 for Alaska bridges.

  • data/bridges/make_dem_dif_for_bridges.py Compares lidar-elevation raster files with USGS 3DEP DEMs and generates DEM difference raster files at HUC levels, as described here. Non-bridge locations (which make up most of the HUC domain) have zero values, while OSM bridge locations typically show positive values. This script also needs to be run twice: once for the CONUS and a separate time specifically for Alaska.

  • data/bridges/conda_fim_bridges_enviro.yml A conda environment file to install conda env needed for running data/bridges/make_rasters_using_lidar.py

  • data/bridges/setup_conda_for_make_rasters.txt Provides instructions for running the data/bridges/make_rasters_using_lidar.py script on a Windows machine using a Conda environment.

Changes

  • data/bridges/pull_osm_bridges.py The updates include:

    • This script now generates two separate OSM bridge centerline files: one for CONUS and another for Alaska
    • ?
  • src/bash_variables.env - Ingests links to two new VRT files for DEM difference rasters (CONUS and Alaska) and the new Alaska OSM bridges GPKG file.

  • data/wbd/generate_pre_clip_fim_huc8.py Pre-clips the new Alaska OSM bridges GPKG file.

  • src/run_unit_wb.sh Crops the bridge elevation difference VRT file to the buffered HUC boundary and create a copy for branch 0.

  • src/run_by_branch.sh Clips HUC-level bridge elevation difference rasters for branches.

  • src/heal_bridges_osm.py Now implements two distinct workflows for healing the REM at OSM bridge locations, depending on the availability of lidar data:

    • If lidar data is unavailable: The existing workflow is followed, creating a 10m buffer around the bridge centerline and applying the maximum REN value within this buffer to the entire area.
    • If lidar data is available: The script reads elevation difference rasters and incorporates their values into the REM, effectively healing it at bridge locations. Additionally, it reports the median healed REM values within a 1.5m buffer around bridge centerlines for use in determining bridge risk status.
  • src/delineate_hydros_and_produce_HAND.sh Updated to align with the revised input arguments of src/heal_bridges_osm.py

  • src/aggregate_by_huc.py Applies 'threshold' HAND/discharge terminology instead of 'max' HAND/discharge terminology for determining bridges risk status.

  • tools/bridge_inundation.py. Same as src/aggregate_by_huc.py

  • data/usgs/acquire_and_preprocess_3dep_dems.py ??

Removals

Testing

Deployment Plan (For developer use)

How does the changes affect the product?

  • Code only?
  • If applicable, has a deployment plan be created with the deployment person/team?
  • Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
  • If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
  • Require new pre-clip set?
  • Has new or updated python packages?

Current files / folders to be deployed

  • WBD_National_HUC8_EPSG_5070_HAND_domain.gpkg, replaces
    WBD_National_EPSG_5070_WBDHU8_clip_dem_domain.gpkg
  • Wbd_national_South_Alaska.gpkg ( new but from other PR from Oct.. see if it is already up to date in all enviros)
  • New OSM Files - .../fim-data/inputs/osm/bridges/20250207/
  • New pre-clips (coming)

(updates pending)
Diff raster files for each HUC6 (AK huc8)
udpated osm_bridges after lidar processing (each address below, adj for latest editions)

-/inputs/osm/bridges/250207/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg

  • /inputs/osm/bridges/250207/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg
  • /inputs/osm/bridges/250207/CanBeDeleted_alaska_lidar_osm_rasters/ (let's keep the "CanBeDeleted" just on EFS, no copy to other enviros)
  • /inputs/osm/bridges/250102/CanBeDeleted_conus_lidar_osm_rasters/ (let's keep the "CanBeDeleted" just on EFS, no copy to other enviros)

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

  • Informative and human-readable title, using the format: [_pt] PR: <description>
  • Links are provided if this PR resolves an issue, or depends on another other PR
  • If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
  • Changes are limited to a single goal (no scope creep)
  • The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
  • pre-commit hooks were run locally
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • CHANGELOG updated with template version number, e.g. 4.x.x.x
  • Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

  • Update CHANGELOG with latest version number and merge date
  • Update the Citation.cff file to reflect the latest version number in the CHANGELOG
  • If applicable, update README with major alterations

@RobHanna-NOAA
Copy link
Contributor

We have a fix we need to do.. but not quite yet:

data/inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg
and a conus version
/data//inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg

but bash variables is looking only for:

osm_bridges=${inputsDir}/osm/bridges/250102/huc6_dem_diff/osm_all_bridges_modified.gpkg
and
input_bridge_elev_diff=${inputsDir}/osm/bridges/250102/huc6_dem_diff/bridge_elev_diff.vrt

Shouldn't there be two sets of these bash variables? one for conus and one for ak?
which would affect run_unit_wb.sh and also run_by_branch.sh

Keep an eye on your heal_bridges_osm.py file too.

It would also affect generate_pre_clip_fim_huc8.py and clip_vectors_to_wbd.py. I can help with that if you like. Just do small subset tests on pre-clip as I am going to be merging in some Matt's fixes for pre-clips.

To do a full re-gen of conus pre-commits, I need both branches.

I think I will approve and get Matt's 1178 merged to dev first, but hold on doing a pre-clip on his, then we can merge dev into yours and update your pre-clip. Those new pre-clips can now be used to do a full clean test on fim-pipeline and your code chagnes. I will do the new pre-clips when we get there on my prod machines.

When we merge dev back to yours.. we will need to watch the delinate_hydros...sh as both of you have changes to the exact same heal calls.

We will have to put this on hold until Matt's is merged in, then we can fix yours up, including pre-clips. That will affect the files being given / loaded for fim_pipeline to use.

@AliForghani-NOAA
Copy link
Collaborator Author

AliForghani-NOAA commented Jan 24, 2025

Rob,
I could understand two topics:
1- Regarding bash variables links for Alaska.... Yes, this is correct and I am still working on that --have not performed an Alaska run yet.

2- Regarding pre-clipping of osm vector data.... we need to pre-clip below two files which are just some new input data and they should not have any conflict with anything else. I could not follow if there is any limitation for this data pre-clipping or not.

  • CONUS: /inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg
  • Alaska: /inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg

Please let me know if I missed anything else.

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Jan 24, 2025

Rob, I could understand two topics: 1- Regarding bash variables links for Alaska.... Yes, this is correct and I am still working on that --have not performed an Alaska run yet.

2- Regarding pre-clipping of osm vector data.... we need to pre-clip below two files which are just some new input data and they should not have any conflict with anything else. I could not follow if there is any limitation for this data pre-clipping or not.

  • CONUS: /inputs/osm/bridges/250102/huc6_dem_diff/conus/conus_osm_bridges_modified.gpkg
  • Alaska: /inputs/osm/bridges/250102/huc6_dem_diff/alaska/alaska_osm_bridges_modified.gpkg

Please let me know if I missed anything else.

Yes.

  1. absolutely correct
  2. We know we need to fix pre-clip for this which we can do after Matt's merge. We will need to pre-clip before we can accurately test the full lidar system. I can help with the pre-clip adjustments and testing as I know it pretty well. Well.. I can talk you through it if you want to do it too. Matt's pending PR has pre-clip changes we need first.

@RobHanna-NOAA
Copy link
Contributor

RobHanna-NOAA commented Jan 28, 2025

  • Next steps updated Jan 28
    • Rob: make all new DEMS (conus and AK) at the current HUC6 Conus / HUC8 AK. Something seems wrong with Matt's earlier PR for dems from earlier for layer name changes in AK. I need to check this. Just to make sure. Seeing as the WBD appears to have changed ???
      - Update: Jan 31: DEMS are good but I am changing the name and updating one of the WBD files to make it more/less have only the FIM domain WBD_National_EPSG_5070_WBDHU8_clip_dem_domain.gpkg will be replace with WBD_National_HUC8_EPSG_5070_HAND_domain.gpkg. It is a smaller version including our domain HUC8s but a few other strays. ak) removed North Alaska, some Pacific ones, a few non US ones. I didn't' fully reconcile it to our included_huc8_withAlaska.lst but much closer the the previous one.
    • With new DEM's/vrt's being created, run/test Ali's tool for doing dem diff rasters. Run the dem diff regen at scale.
      • Updated: Jan 31: No longer needed
    • Update Feb 7, 2025: - new osm bridge data completed.
    • Update Feb 7, 2025: Repull lidar data
    • Update Feb 7, 2025: - Re-run make dem differences for lidar.
    • Test pre-clips: Ali is testing one conus and one AK huc for pre-clip and will take it through fim-pipeline. (Done)
    • Run pre-clip at scale.
    • Approve my part of Ali's PR (the data/bridges code). Have someone else test / approve fim-pipeline more in depth with this full new dataset. ie) testing changes to the files in the repo "src" directory.
    • Copy new PR inputs to all enviros. PS. some already copied, but reconcile it in all 5 enviros.

This lets me test Matt's DEM, WBD change, Ali's new LIDAR DEMs tool, and pre-clip which may/may not have an error. The result of that chain gives us valid exact data going into production for this PR and some retro, already merged, PR's.

In a separate PR, make the fixes for get_sample_data ensuring the new DEMS, pre-clips and LIDAR DEMS are coming in correctly. Look for other changed / new inputs. Might only need the new LIDAR DEMs. TBD.

This was linked to issues Feb 4, 2025
@AliForghani-NOAA AliForghani-NOAA marked this pull request as ready for review February 7, 2025 18:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[5pt] OSM "proposed" bridges [21pt] Lidar bridges
2 participants