Merge pull request #120 from ASFHyP3/release

Release: add browse images to s3

.github/pull_request_template.md

@@ -1,6 +1,6 @@
 <!--
 If this is a pull request for a new release, please use the release template:
-   https://github.com/asfadmin/hyp3-rtc-gamma/compare/master...develop?template=release.md
+   https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/master...develop?template=release.md
  
 NOTE: Pull requests should only be opened for merges to protected branches (required) and any 
 changes which you'd like reviewed. Do not open a pull request to update a feature or personal

.github/workflows/release.yml

@@ -16,12 +16,11 @@ jobs:
           token: ${{ secrets.TOOLS_BOT_PAK }}
 
       - name: Create Release
-        uses: actions/create-release@v1
+        uses: docker://antonyurchenko/git-release:latest
         env:
           GITHUB_TOKEN: ${{ secrets.TOOLS_BOT_PAK }}
-        with:
-          tag_name: ${{ github.ref }}
-          release_name: HyP3 RTC GAMMA ${{ github.ref }}
+          ALLOW_TAG_PREFIX: "true"
+          RELEASE_NAME_PREFIX: HyP3 RTC GAMMA
 
       - name: Attempt fast-forward develop from master
         run: |
@@ -41,4 +40,5 @@ jobs:
           pr_assignee: ${{ github.actor }}
           pr_label: tools-bot
           pr_draft: false
+          pr_allow_empty: true
           github_token: ${{ secrets.TOOLS_BOT_PAK }}

.github/workflows/test-and-build.yml

@@ -64,7 +64,7 @@ jobs:
         with:
           query: .github/queries/asssociated-pr.query.yml
           outputFile: pr.json
-          owner: asfadmin
+          owner: ASFHyP3
           name: hyp3-rtc-gamma
           sha: ${{ github.sha }}
 
@@ -76,7 +76,7 @@ jobs:
         with:
           query: .github/queries/pr-labels.query.yml
           outputFile: labels.json
-          owner: asfadmin
+          owner: ASFHyP3
           name: hyp3-rtc-gamma
 
       - name: Upload a Build Artifact

CHANGELOG.md

@@ -6,29 +6,36 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/) 
 and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [v2.0.3](https://github.com/asfadmin/hyp3-rtc-gamma/compare/v2.0.2...v2.0.3)
+## [2.0.4](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.3...v2.0.4)
+
+### Changed
+* The v2 entrypoint will now upload browse images and thumnail images in addition to the zip file.
+* Upgrade to [hyp3-lib 1.3.0](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#130).  In particular, geotiff products no longer include overviews.
+* Eliminated seprate "low-res" (1024x) and "high-res" (2048x) browse image resolutions in favor of a single 2048x image.
+
+## [2.0.3](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.2...v2.0.3)
 
 ## Fixed
 * Updates the minimum required `hyp3lib` and `hyp3proclib` version to bring in the
-  [`get_dem.py` NoData bugfix](https://github.com/asfadmin/hyp3-lib/pull/175) and
+  [`get_dem.py` NoData bugfix](https://github.com/ASFHyP3/hyp3-lib/pull/175) and
   the [`default_rtc_resolution` bugfix](https://github.com/asfadmin/hyp3-proc-lib/pull/4),
   respectively
 
-## [v2.0.2](https://github.com/asfadmin/hyp3-rtc-gamma/compare/v2.0.1...v2.0.2)
+## [2.0.2](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.1...v2.0.2)
 
 ### Changed
 * The v2 entrypoint will now make up to three retry attempts if it fails to download the input granule from the ASF archive.
 * Changed the name of the product README file to `<product_name>.README.txt`, e.g. `S1A_IW_RT30_20170708T161200_G_gpn.README.txt`
 * Calls to mk_geo_radcal will no longer include the `-j do not use layover-shadow map in the calculation of pixel_area` flag.  The layover-shadow map will now be consistently applied to all products.
-* Upgraded to [hyp3-lib v1.2.2](https://github.com/asfadmin/hyp3-lib/blob/develop/CHANGELOG.md#v122)
+* Upgraded to [hyp3-lib v1.2.2](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#v122)
 * Removed custom blank_bad_data.py from mk_geo_radcal processing.  Border pixels for older GRD products are now cleaned using the default `make_edge` setting of `par_S1_GRD`.
 
-## [v2.0.1](https://github.com/asfadmin/hyp3-rtc-gamma/compare/v2.0.0...v2.0.1)
+## [2.0.1](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.0...v2.0.1)
 
 ### Changed
 * hyp3-v2 output products are now packaged in a .zip file similar to hyp3-v1 products
 
-## [v2.0.0](https://github.com/asfadmin/hyp3-rtc-gamma/compare/v1.3.1...v2.0.0)
+## [2.0.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v1.3.1...v2.0.0)
 
 This is a significant refactor of `hyp3-rtc-gamma` into:
 * A `pip` installable package called `hyp3_rtc_gamma`

Dockerfile

@@ -7,8 +7,8 @@ LABEL org.opencontainers.image.description="HyP3 plugin for radiometric terrain
 LABEL org.opencontainers.image.vendor="Alaska Satellite Facility"
 LABEL org.opencontainers.image.authors="ASF APD/Tools Team <[email protected]>"
 LABEL org.opencontainers.image.licenses="BSD-3-Clause"
-LABEL org.opencontainers.image.url="https://github.com/asfadmin/hyp3-rtc-gamma"
-LABEL org.opencontainers.image.source="https://github.com/asfadmin/hyp3-rtc-gamma"
+LABEL org.opencontainers.image.url="https://github.com/ASFHyP3/hyp3-rtc-gamma"
+LABEL org.opencontainers.image.source="https://github.com/ASFHyP3/hyp3-rtc-gamma"
 # LABEL org.opencontainers.image.documentation=""
 
 # Dynamic lables to define at build time via `docker build --label`

conda-env.yml

@@ -36,5 +36,5 @@ dependencies:
     # For running
     - --trusted-host hyp3-pypi.s3-website-us-east-1.amazonaws.com
       --extra-index-url http://hyp3-pypi.s3-website-us-east-1.amazonaws.com
-    - hyp3lib>=1.2.3,<2
+    - hyp3lib>=1.3.0,<2
     - hyp3proclib>=1.0.1,<2

hyp3_rtc_gamma/__main__.py

@@ -1,7 +1,7 @@
 """
 rtc_gamma processing for HyP3
 """
-
+import glob
 import os
 import shutil
 import sys
@@ -135,7 +135,9 @@ def main_v2():
     output_zip = make_archive(base_name=product_name, format='zip', base_dir=product_name)
     if args.bucket:
         upload_file_to_s3(output_zip, args.bucket, args.bucket_prefix)
-
+        browse_images = glob.glob(f'{product_name}/*.png')
+        for image in browse_images:
+            upload_file_to_s3(image, args.bucket, args.bucket_prefix + '/browse')
 # end v2 functions
 
 
@@ -144,15 +146,15 @@ def find_png(dir_):
     for subdir, dirs, files in os.walk(dir_):
         for file in files:
             filepath = os.path.join(subdir, file)
-            if filepath.endswith(".png") and 'rgb' in filepath and 'large' not in filepath:
+            if filepath.endswith(".png") and 'rgb' in filepath:
                 log.info('Browse image: ' + filepath)
                 return filepath
 
     # No RGB, see if we can find a grayscale browse
     for subdir, dirs, files in os.walk(dir_):
         for file in files:
             filepath = os.path.join(subdir, file)
-            if filepath.endswith(".png") and 'large' not in filepath:
+            if filepath.endswith(".png"):
                 log.info('Browse image: ' + filepath)
                 return filepath
 

hyp3_rtc_gamma/create_metadata.py

@@ -186,11 +186,6 @@ def create_arc_xml(infile, outfile, input_type, gamma_flag, pwr_flag, filter_fla
                 scale = 'grayscale'
                 encoded_jpg = pngtothumb("{}.png".format(outfile))
 
-            if "large" in myfile:
-                res = "medium"
-            else:
-                res = "low"
-
             with open("{}/RTC_GAMMA_Template_{}_png.xml".format(etc_dir, scale), "rb") as f:
                 for line in f:
                     line = line.replace(b"[DATE]", bytes(date, 'utf-8'))
@@ -202,7 +197,6 @@ def create_arc_xml(infile, outfile, input_type, gamma_flag, pwr_flag, filter_fla
                     line = line.replace(b"[FULL_TYPE]", bytes(full_type, 'utf-8'))
                     line = line.replace(b"[THUMBNAIL_BINARY_STRING]", encoded_jpg)
                     line = line.replace(b"[GRAN_NAME]", bytes(granulename, 'utf-8'))
-                    line = line.replace(b"[RES]", bytes(res, 'utf-8'))
                     line = line.replace(b"[SPACING]", bytes("{}".format(spacing), 'utf-8'))
                     line = line.replace(b"[DEM]", bytes("{}".format(dem_type), 'utf-8'))
                     line = line.replace(b"[FORMAT]", bytes(format_type, 'utf-8'))

hyp3_rtc_gamma/etc/README_RTC_GAMMA.txt

@@ -65,13 +65,13 @@ Each RTC image [FILT] been speckle filtered. The default is to not apply a speck
 -------------
 ## 2. Browse images in grayscale and color
 
-PNG files are generated in two different resolutions for quick visualization of the backscatter data. Each png browse image is accompanied by an aux file containing the projection and geocoding information for the file.
+PNG files are generated for quick visualization of the backscatter data. Each png browse image is accompanied by an aux file containing the projection and geocoding information for the file.
 
-All products will include a grayscale png browse image in both resolutions. It is a rendering of the primary polarization data, scaled to an ASF standard to display nicely in grayscale. The low-resolution image is designated by a simple .png extension, while the tag _large.png indicates the medium-resolution image. 
+All products will include a grayscale png browse image. It is a rendering of the primary polarization data, scaled to an ASF standard to display nicely in grayscale. The image is designated by a simple .png extension. 
 
-For dual-pol products, a false-color png browse image is generated in both resolutions. It is a rendering of the primary and cross-polarization data, scaled to an ASF standard to display nicely in color. These files are additionally tagged with _rgb, but otherwise have the same tags/extensions as the grayscale browse images.
+For dual-pol products, a false-color png browse image is generated. It is a rendering of the primary and cross-polarization data, scaled to an ASF standard to display nicely in color. These files are additionally tagged with _rgb, but otherwise have the same tags/extensions as the grayscale browse images.
 
-KMZ files are generated in the higher resolution for use in Google Earth and other compatible applications. All products will include a grayscale kmz image, and dual-pol products will also include a color browse kmz image.
+KMZ files are generated for use in Google Earth and other compatible applications. All products will include a grayscale kmz image, and dual-pol products will also include a color browse kmz image.
 
 -------------
 ## 3. DEM used to correct the data

hyp3_rtc_gamma/etc/RTC_GAMMA_Template_color_png.xml

@@ -19,8 +19,8 @@
         <ModTime>[TIME]</ModTime>
     </Esri>
     <dataIdInfo>
-        <idPurp>This is a [RES]-resolution browse image of a radiometric terrain corrected granule of [TYPE] SAR data from the Sentinel-1 mission, processed using GAMMA software. The values are scaled from a combination of the primary and cross-polarization values to maximize color contrast in the appearance of the image. To access actual [FORMAT] values, refer to the VV/VH/HV/HH TIFF files included in the same folder as this image.</idPurp>
-        <idAbs>&lt;DIV STYLE="text-align:Left;"&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;P&gt;&lt;SPAN&gt;This file is a [RES]-resolution browse image of a radiometric terrain corrected granule of [FULL_TYPE] ([TYPE]) Synthetic Aperture Radar data from the Copernicus Sentinel-1 mission (European Space Agency). It is provided in [PCS] coordinates and is corrected for terrain using the [DEM] Digital Elevation Model.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The actual [FORMAT] data is contained in the VV/HH (primary polarization) and/or VH/HV (cross-polarization) TIFF files included in the same downloaded folder as this file. This image is a [RES]-resolution rendering of a combination of the primary and cross-polarization data, scaled to an ASF standard to display nicely in color for visualization purposes. &lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The name of the SAR granule used to generate the files in this folder is: [GRAN_NAME]. For a detailed description of the Sentinel-1 file naming convention, refer to &lt;/SPAN&gt;&lt;A href="https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1/sentinel-1-data-and-imagery/#Naming_Convention"&gt;&lt;SPAN&gt;https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1/sentinel-1-data-and-imagery/#Naming_Convention&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For areas where there is not a publicly-available digital elevation model (i.e. sea ice), geocoded products without terrain correction are also available from &lt;/SPAN&gt;&lt;A href="http://hyp3.asf.alaska.edu"&gt;&lt;SPAN&gt;http://hyp3.asf.alaska.edu&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The Sentinel-1A satellite was launched April 3, 2014, and the Sentinel-1B satellite was launched April 25, 2016. The satellites each have a 12-day repeat cycle. More information about the mission is available at &lt;/SPAN&gt;&lt;A href="https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/sentinel-1"&gt;&lt;SPAN&gt;https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/sentinel-1&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Additional information about Sentinel-1 data, imagery, tools and applications is available at &lt;/SPAN&gt;&lt;A href="https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1"&gt;&lt;SPAN&gt;https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;
+        <idPurp>This is a browse image of a radiometric terrain corrected granule of [TYPE] SAR data from the Sentinel-1 mission, processed using GAMMA software. The values are scaled from a combination of the primary and cross-polarization values to maximize color contrast in the appearance of the image. To access actual [FORMAT] values, refer to the VV/VH/HV/HH TIFF files included in the same folder as this image.</idPurp>
+        <idAbs>&lt;DIV STYLE="text-align:Left;"&gt;&lt;DIV&gt;&lt;DIV&gt;&lt;P&gt;&lt;SPAN&gt;This file is a browse image of a radiometric terrain corrected granule of [FULL_TYPE] ([TYPE]) Synthetic Aperture Radar data from the Copernicus Sentinel-1 mission (European Space Agency). It is provided in [PCS] coordinates and is corrected for terrain using the [DEM] Digital Elevation Model.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The actual [FORMAT] data is contained in the VV/HH (primary polarization) and/or VH/HV (cross-polarization) TIFF files included in the same downloaded folder as this file. This image is a rendering of a combination of the primary and cross-polarization data, scaled to an ASF standard to display nicely in color for visualization purposes. &lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The name of the SAR granule used to generate the files in this folder is: [GRAN_NAME]. For a detailed description of the Sentinel-1 file naming convention, refer to &lt;/SPAN&gt;&lt;A href="https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1/sentinel-1-data-and-imagery/#Naming_Convention"&gt;&lt;SPAN&gt;https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1/sentinel-1-data-and-imagery/#Naming_Convention&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;For areas where there is not a publicly-available digital elevation model (i.e. sea ice), geocoded products without terrain correction are also available from &lt;/SPAN&gt;&lt;A href="http://hyp3.asf.alaska.edu"&gt;&lt;SPAN&gt;http://hyp3.asf.alaska.edu&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;The Sentinel-1A satellite was launched April 3, 2014, and the Sentinel-1B satellite was launched April 25, 2016. The satellites each have a 12-day repeat cycle. More information about the mission is available at &lt;/SPAN&gt;&lt;A href="https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/sentinel-1"&gt;&lt;SPAN&gt;https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/sentinel-1&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;P&gt;&lt;SPAN&gt;Additional information about Sentinel-1 data, imagery, tools and applications is available at &lt;/SPAN&gt;&lt;A href="https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1"&gt;&lt;SPAN&gt;https://asf.alaska.edu/data-sets/sar-data-sets/sentinel-1&lt;/SPAN&gt;&lt;/A&gt;&lt;SPAN&gt;.&lt;/SPAN&gt;&lt;/P&gt;&lt;/DIV&gt;&lt;/DIV&gt;&lt;/DIV&gt;
         </idAbs>
         <idCredit>ASF DAAC [YEARPROCESSED] using GAMMA software. Contains modified Copernicus Sentinel data [YEARACQUIRED], processed by ESA.</idCredit>
         <idCitation xmlns="">