Redo url file #150

jacquelynsmale · 2023-11-14T19:22:59Z

No description provided.

hjkristenson · 2023-11-16T02:30:47Z

image_services/opera/metadata/metadata_text/OPERA_RTC_HH_metadata.txt

+OPERA Level-2 Radiometric Terrain Corrected (RTC) products in HH polarization generated from Sentinel-1 SAR imagery, processed by OPERA (JPL). Values for the HH polarization are commonly driven by surface roughness and/or soil moisture, with rougher surfaces and higher soil moisture returning higher backscatter values. Surface water appears very dark under calm conditions, as the signal bounces off the surface away from the sensor.
+
+Credits:
+OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.


Suggested change

OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.

OPERA Level-2 Radiometric Terrain Corrected SAR Backscatter from Sentinel-1. NASA Alaska Satellite Facility Distributed Active Archive Center. doi: https://doi.org/10.5067/SNWG/OPERA_L2_RTC-S1_V1. Contains modified Copernicus Sentinel data, processed by ESA.

hjkristenson · 2023-11-16T02:31:08Z

image_services/opera/metadata/metadata_text/OPERA_RTC_HV_metadata.txt

+OPERA Level-2 Radiometric Terrain Corrected (RTC) products in HV polarization generated from Sentinel-1 SAR imagery, processed by OPERA (JPL). Values for the HV polarization are commonly driven by surface roughness and/or soil moisture, with rougher surfaces and higher soil moisture returning higher backscatter values. Surface water appears very dark under calm conditions, as the signal bounces off the surface away from the sensor.
+
+Credits:
+OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.


Suggested change

OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.

OPERA Level-2 Radiometric Terrain Corrected SAR Backscatter from Sentinel-1. NASA Alaska Satellite Facility Distributed Active Archive Center. doi: https://doi.org/10.5067/SNWG/OPERA_L2_RTC-S1_V1. Contains modified Copernicus Sentinel data, processed by ESA.

image_services/opera/metadata/metadata_text/OPERA_RTC_VH_metadata.txt

hjkristenson · 2023-11-16T02:31:46Z

image_services/opera/metadata/metadata_text/OPERA_RTC_VV_metadata.txt

+OPERA Level-2 Radiometric Terrain Corrected (RTC) products in VV polarization generated from Sentinel-1 SAR imagery, processed by OPERA (JPL). Values for the VV polarization are commonly driven by surface roughness and/or soil moisture, with rougher surfaces and higher soil moisture returning higher backscatter values. Surface water appears very dark under calm conditions, as the signal bounces off the surface away from the sensor.
+
+Credits:
+OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.


Suggested change

OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.

OPERA Level-2 Radiometric Terrain Corrected SAR Backscatter from Sentinel-1. NASA Alaska Satellite Facility Distributed Active Archive Center. doi: https://doi.org/10.5067/SNWG/OPERA_L2_RTC-S1_V1. Contains modified Copernicus Sentinel data, processed by ESA.

hjkristenson · 2023-11-16T02:31:58Z

image_services/opera/metadata/templates/rtc_metadata.txt.j2

+OPERA Level-2 Radiometric Terrain Corrected (RTC) products in {{ polarization }} polarization generated from Sentinel-1 SAR imagery, processed by OPERA (JPL). Values for the {{ polarization }} polarization are commonly driven by surface roughness and/or soil moisture, with rougher surfaces and higher soil moisture returning higher backscatter values. Surface water appears very dark under calm conditions, as the signal bounces off the surface away from the sensor.
+
+Credits:
+OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.


Suggested change

OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.

OPERA Level-2 Radiometric Terrain Corrected SAR Backscatter from Sentinel-1. NASA Alaska Satellite Facility Distributed Active Archive Center. doi: https://doi.org/10.5067/SNWG/OPERA_L2_RTC-S1_V1. Contains modified Copernicus Sentinel data, processed by ESA.

Automate OPERA URLs

…ata.txt Co-authored-by: Heidi Kristenson <[email protected]>

jhkennedy

This all looks good to me! A couple of exceedingly minor suggestions you can take or leave, and a couple of questions for parts I wasn't sure about that look deliberate -- just curious why; not blocking.

jhkennedy · 2023-12-14T20:45:20Z

image_services/opera/make_opera_services.py

+    with open(filename, newline='') as urlfile:
+        records = urlfile.read().split('\n')[:-1]


Interesting -- is this a windows thing? I would have expected:

with open(filename) as urlfile: records = urlfile.read().splitlines()[:-1]

or with pathlib:

Path(filename).read_text().splitlines()[:-1]

image_services/opera/update_opera_urls.py

image_services/opera/update_opera_urls.sh

jhkennedy · 2023-12-14T20:58:28Z

image_services/opera/make_opera_services.py

+    filename = f'{dataset_name}_vsis3_urls.csv'
+    with open(filename, newline='') as urlfile:
+        records = urlfile.read().split('\n')[:-1]
+    return [f'{record[:-1]}' for record in records]


What character are you stripping here?

@jhkennedy an empty space :/ Kinda hacky. I don't know why it keeps putting it there?

@jacquelynsmale is it a space or a carriage return? If there's a space in the file that's why it'd be there, but windows does line endings with a \r\n (carriage return and new line). The with block has newline='' which does not translate \r\n into a single \n character, so your split would mean there's potentially a trailing \r if the csv was written on a windows box

@jacquelynsmale if you're stripping whitespace from the end of the line, you could also just do:

return [record.rstrip() for record in records]

Or from both ends with strip()

Co-authored-by: Joseph H Kennedy <[email protected]>

jacquelynsmale added 2 commits November 14, 2023 10:20

update script to pull from csv file

80e3742

add in vsis3 file

a655195

jacquelynsmale requested a review from a team November 14, 2023 19:23

jacquelynsmale marked this pull request as draft November 14, 2023 19:23

jacquelynsmale added 13 commits November 14, 2023 10:40

udpate url bucket

bf2f844

change raster path to url

178e85b

move around some things

dae889f

switch download url

0a20b83

switch back to raster path

a2a9be5

add in region

7481c7f

add in environs

50d2466

reformat

bae41c7

add in gdal config option

ed53451

remove extra s3_prefixes

04d4731

remove unnecessary environ

03bba84

add in boto3

ef8851c

get rid of asf_search dependency

1af729f

hjkristenson reviewed Nov 16, 2023

View reviewed changes

jacquelynsmale added 10 commits November 15, 2023 22:17

get rid of the environment setting

b7744d7

add in edc set up

1652583

remove old cookie files

d32cbd1

add back in gdal exceptions

4cebdaa

AWS profile update

69e5fa4

add in another profile statement

eb54a0a

swap permissions

cacf139

add back in hyp3 permissions???

c8a61ee

update the max um per mosaic

7afd87c

update the max um per mosaic

9471169

hjkristenson and others added 18 commits December 1, 2023 16:36

add metadata file for populating the MD for the test CA VV dataset

f09f386

add metadata file for populating the MD for the test CA VH dataset

5dacc2e

change link formatting to markdown

eb697b0

incorporate metadata edits from JPL

94e130a

remove space in title

f66b72d

more minor updates

1c3a437

adjust date range

eb71e57

adjust date range

a2a07bf

remove s3 bucket raster file pull

3a9e6d6

change around naming convention

3a52f3e

udpate path and remove boto3 import

4704a4a

remove unneccary dataset varaible

775b357

iron ut naming between makeoperaservices and update_opera_urls

1492ade

add union for s3 bucket upload

6730c64

switch dataset name and service name

013405f

rename uri to urls to be consistent

71a29c2

Merge branch 'redo-url-file' into automate-urls

1ccef60

Merge pull request #151 from ASFHyP3/automate-urls

801a3d2

Automate OPERA URLs

jacquelynsmale changed the base branch from develop to opera-egis-updates December 14, 2023 00:30

jacquelynsmale and others added 3 commits December 13, 2023 15:33

Update image_services/opera/metadata/metadata_text/OPERA_RTC_VH_metad…

458cd5e

…ata.txt Co-authored-by: Heidi Kristenson <[email protected]>

Delete image_services/opera/vsis3_urls.txt

7d04962

Merge branch 'opera-egis-updates' into redo-url-file

a9cdc47

jacquelynsmale marked this pull request as ready for review December 14, 2023 00:35

jhkennedy approved these changes Dec 14, 2023

View reviewed changes

jacquelynsmale and others added 4 commits December 14, 2023 12:01

finally incorporate changes in from Heidi

7fb63bb

Update image_services/opera/update_opera_urls.py

4432d90

Co-authored-by: Joseph H Kennedy <[email protected]>

tone down line splitting shenanigans

06a7183

Update image_services/opera/update_opera_urls.sh

db4211e

Co-authored-by: Joseph H Kennedy <[email protected]>

jacquelynsmale merged commit d0b5bd8 into opera-egis-updates Dec 14, 2023
1 of 2 checks passed

jacquelynsmale deleted the redo-url-file branch December 14, 2023 21:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redo url file #150

Redo url file #150

jacquelynsmale commented Nov 14, 2023

hjkristenson Nov 16, 2023 •

edited

Loading

hjkristenson Nov 16, 2023 •

edited

Loading

hjkristenson Nov 16, 2023

hjkristenson Nov 16, 2023

jhkennedy left a comment

jhkennedy Dec 14, 2023

jhkennedy Dec 14, 2023

jacquelynsmale Dec 14, 2023

jhkennedy Dec 14, 2023 •

edited

Loading

jhkennedy Dec 14, 2023

	OPERA Level-2 Radiometric Terrain Corrected Static Layers from Sentinel-1. Contains modified Copernicus Sentinel data, processed by ESA.
	OPERA Level-2 Radiometric Terrain Corrected SAR Backscatter from Sentinel-1. NASA Alaska Satellite Facility Distributed Active Archive Center. doi: https://doi.org/10.5067/SNWG/OPERA_L2_RTC-S1_V1. Contains modified Copernicus Sentinel data, processed by ESA.

		with open(filename, newline='') as urlfile:
		records = urlfile.read().split('\n')[:-1]

Redo url file #150

Redo url file #150

Conversation

jacquelynsmale commented Nov 14, 2023

hjkristenson Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

hjkristenson Nov 16, 2023 • edited Loading

Choose a reason for hiding this comment

hjkristenson Nov 16, 2023

Choose a reason for hiding this comment

hjkristenson Nov 16, 2023

Choose a reason for hiding this comment

jhkennedy left a comment

Choose a reason for hiding this comment

jhkennedy Dec 14, 2023

Choose a reason for hiding this comment

jhkennedy Dec 14, 2023

Choose a reason for hiding this comment

jacquelynsmale Dec 14, 2023

Choose a reason for hiding this comment

jhkennedy Dec 14, 2023 • edited Loading

Choose a reason for hiding this comment

jhkennedy Dec 14, 2023

Choose a reason for hiding this comment

hjkristenson Nov 16, 2023 •

edited

Loading

hjkristenson Nov 16, 2023 •

edited

Loading

jhkennedy Dec 14, 2023 •

edited

Loading