Skip to content

Commit

Permalink
Merge pull request #191 from ASFHyP3/develop
Browse files Browse the repository at this point in the history
Release hyp3-rtc-gamma --> hyp3-gamma
  • Loading branch information
jhkennedy authored Dec 11, 2020
2 parents 9c4bac6 + 8c8baa3 commit be2f89b
Show file tree
Hide file tree
Showing 18 changed files with 115 additions and 80 deletions.
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<!--
If this is a pull request for a new release, please use the release template:
https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/master...develop?template=release.md
https://github.com/ASFHyP3/hyp3-gamma/compare/master...develop?template=release.md
NOTE: Pull requests should only be opened for merges to protected branches (required) and any
changes which you'd like reviewed. Do not open a pull request to update a feature or personal
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.TOOLS_BOT_PAK }}
ALLOW_TAG_PREFIX: "true"
RELEASE_NAME_PREFIX: "HyP3 RTC GAMMA "
RELEASE_NAME_PREFIX: "HyP3 GAMMA "

- name: Attempt fast-forward develop from master
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/static-analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ jobs:
- name: Lint with flake8
run: |
flake8 --max-line-length=120 --import-order-style=pycharm --statistics \
--application-import-names hyp3_rtc_gamma
--application-import-names hyp3_gamma
gitleaks:
Expand Down
25 changes: 13 additions & 12 deletions .github/workflows/test-and-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,12 @@ on:
- develop

env:
HYP3_REGISTRY: 626226570674.dkr.ecr.us-east-1.amazonaws.com
S3_PYPI_HOST: hyp3-pypi.s3-website-us-east-1.amazonaws.com
AWS_REGION: us-east-1
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
HYP3_REGISTRY: 845172464411.dkr.ecr.us-west-2.amazonaws.com
S3_PYPI_HOST: hyp3-pypi-west.s3-website-us-west-2.amazonaws.com
S3_PYPI_BUCKET: hyp3-pypi-west
AWS_REGION: us-west-2
AWS_ACCESS_KEY_ID: ${{ secrets.V2_AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.V2_AWS_SECRET_ACCESS_KEY }}

jobs:
pytest:
Expand All @@ -27,14 +28,14 @@ jobs:
with:
auto-update-conda: true
python-version: 3.7
activate-environment: hyp3-rtc-gamma
activate-environment: hyp3-gamma
environment-file: conda-env.yml

- name: Pytest in conda environment
shell: bash -l {0}
run: |
python -m pip install .[develop]
pytest --cov=hyp3_rtc_gamma
pytest --cov=hyp3_gamma
- name: Safety analysis of conda environment
shell: bash -l {0}
Expand Down Expand Up @@ -64,7 +65,7 @@ jobs:
query: .github/queries/asssociated-pr.query.yml
outputFile: pr.json
owner: ASFHyP3
name: hyp3-rtc-gamma
name: hyp3-gamma
sha: ${{ github.sha }}

- name: Get PR labels
Expand All @@ -76,7 +77,7 @@ jobs:
query: .github/queries/pr-labels.query.yml
outputFile: labels.json
owner: ASFHyP3
name: hyp3-rtc-gamma
name: hyp3-gamma

- name: Upload a Build Artifact
if: github.ref == 'refs/heads/master'
Expand Down Expand Up @@ -128,8 +129,8 @@ jobs:
export SDIST_VERSION=$(python setup.py --version)
echo "::set-output name=version::${SDIST_VERSION}"
python setup.py sdist bdist_wheel
echo "Uploading version ${SDIST_VERSION} to S3-PyPI"
s3pypi --bucket hyp3-pypi --force --verbose
echo "Uploading version ${SDIST_VERSION} to ${S3_PYPI_BUCKET}"
s3pypi --bucket ${S3_PYPI_BUCKET} --private --force --verbose
dockerize:
Expand All @@ -149,7 +150,7 @@ jobs:

- name: Build, tag, and push image to Amazon ECR
run: |
aws s3 cp s3://hyp3-docker/software/GAMMA_SOFTWARE-20191203-PATCHED_MSP_ISP_DIFF_LAT.linux64_ubuntu1804.tar.gz .
aws s3 cp s3://hyp3-software/GAMMA_SOFTWARE-20191203-PATCHED_MSP_ISP_DIFF_LAT.linux64_ubuntu1804.tar.gz .
tar -zxvf GAMMA_SOFTWARE-20191203-PATCHED_MSP_ISP_DIFF_LAT.linux64_ubuntu1804.tar.gz
export SDIST_VERSION=${{ needs.package.outputs.SDIST_VERSION }}
Expand Down
58 changes: 39 additions & 19 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [PEP 440](https://www.python.org/dev/peps/pep-0440/)
and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.4.1](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.4.0...v2.4.1)
##[3.0.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.4.1...v3.0.0)

**HyP3 RTC GAMMA as been transformed into HyP3 GAMMA; a *processor* based plugin**

### Changed
* `hyp3_rtc_gamma` has been transformed into a `hyp3_gamma` package
* `hyp3_rtc_gamma.__init__` has moved to `hyp3_gamma.__init__`
* `hyp3_rtc_gamma.__main__` has moved to `hyp3_gamma.__main__`
* `hyp3_rtc_gamma.check_coreg` has moved to `hyp3_gamma.rtc.check_coreg`
* `hyp3_rtc_gamma.rtc_sentinel` has moved to `hyp3_gamma.rtc.rtc_sentinel`
* `hyp3_rtc_gamma.smoothem` has moved to `hyp3_gamma.rtc.smoothem`

* `hyp3_gamma` now provides these entrypoints:
* `hyp3_gamma` is the main HyP3 entrypoint
* `rtc` provides the RTC process interface to HyP3
* `rtc_sentinel.py` is the RTC (science code) processor
* `check_coreg.py` for checking the results of the GAMMA coregistration process
* `smoothdem.py` for smoothing and filling holes in DEMs

## [2.4.1](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.4.0...v2.4.1)

### Changed
* `rtc_sentinel.py` now uses GAMMA's updated `mk_geo_radcal2` instead of `mk_geo_radcal`
to remove no-data values in lakes and rivers

* Upgraded to hyp3lib [v1.6.3](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#163) from v1.6.2

## [2.4.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.3.4...v2.4.0)
## [2.4.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.3.4...v2.4.0)

**HyP3 v1 is no longer supported as of this release.**

Expand All @@ -31,25 +51,25 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* `rtc_gamma` package entrypoint will now pass arguments to the `hyp3_rtc_gamma_v2` entrypoint by default.

### Fixed
* `rtc_sentinel.py` now runs successfully when `--nocrosspol` is specified ([#179](https://github.com/ASFHyP3/hyp3-rtc-gamma/issues/179))
* `rtc_sentinel.py` now runs successfully when `--nocrosspol` is specified ([#179](https://github.com/ASFHyP3/hyp3-gamma/issues/179))

## [2.3.4](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.3.3...v2.3.4)
## [2.3.4](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.3.3...v2.3.4)

### Changed
- Upgraded to hyp3lib [v1.6.2](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#162) from v1.6.1

## [2.3.3](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.3.2...v2.3.3)
## [2.3.3](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.3.2...v2.3.3)

### Changed
- Upgraded to hyp3lib [v1.6.1](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#161) from v1.6.0

## [2.3.2](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.3.1...v2.3.2)
## [2.3.2](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.3.1...v2.3.2)

### Changed
- Upgraded to hyp3lib [v1.6.0](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#160) from v1.5.0
- Upgraded to hyp3_metadata [v0.1.2](https://github.com/ASFHyP3/hyp3-metadata-templates/blob/develop/CHANGELOG.md#012) from v0.1.1

## [2.3.1](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.3.0...v2.3.1)
## [2.3.1](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.3.0...v2.3.1)

### Changed
* `README.txt` has been renamed to `README.md.txt`
Expand All @@ -59,7 +79,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### Removed
* Removed the all-white placeholder thumbnail from `_ls_map.tif.xml`

## [2.3.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.2.0...v2.3.0)
## [2.3.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.2.0...v2.3.0)

### Added
* Added cursory developer setup instructions in `README.md`
Expand All @@ -71,14 +91,14 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* RTC Gamma products no longer include an ISO 19115-2 .iso.xml metadata file.
* ASF MapReady is no longer a dependency and has been removed.

## [2.2.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.1.1...v2.2.0)
## [2.2.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.1.1...v2.2.0)

### Changed
* Files now get uploaded to S3 with a `file_type` in v2 tag indicating what the file is
* Files no longer get a prefix based on file type in v2
* Removed `lo_flag` from `rtc_sentinel` since it duplicates the functionality of `resolution`

## [2.1.1](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.1.0...v2.1.1)
## [2.1.1](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.1.0...v2.1.1)

### Changed
* Input scenes are now downloaded directly from NGAP distribution endpoint rather than ASF's datapool
Expand All @@ -90,7 +110,7 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
### Removed
* Non-functional support for non-Sentinel-1 products

## [2.1.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.6...v2.1.0)
## [2.1.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.6...v2.1.0)

### Changed
* Implemented new naming convention for output products. The names of output zips, folders, and files now all share a
Expand All @@ -105,33 +125,33 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* Upgrade to [hyp3-lib v1.4.1](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#141) from 1.3.0.
* Install hyp3-lib via conda instead of pip

## [2.0.6](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.5...v2.0.6)
## [2.0.6](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.5...v2.0.6)

### Fixed
* Resolved issue where thumbnail filenames ended in ..png rather than .png

## [2.0.5](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.4...v2.0.5)
## [2.0.5](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.4...v2.0.5)

### Added
* The v2 entrypoint will now create and upload thumbnail images to S3 in addition to browse images

## [2.0.4](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.3...v2.0.4)
## [2.0.4](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.3...v2.0.4)

### Changed
* The v2 entrypoint will now upload browse images and thumnail images in addition to the zip file.
* Upgrade to [hyp3-lib 1.3.0](https://github.com/ASFHyP3/hyp3-lib/blob/develop/CHANGELOG.md#130).
In particular, geotiff products no longer include overviews.
* Eliminated seprate "low-res" (1024x) and "high-res" (2048x) browse image resolutions in favor of a single 2048x image.

## [2.0.3](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.2...v2.0.3)
## [2.0.3](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.2...v2.0.3)

### Fixed
* Updates the minimum required `hyp3lib` and `hyp3proclib` version to bring in the
[`get_dem.py` NoData bugfix](https://github.com/ASFHyP3/hyp3-lib/pull/175) and
the [`default_rtc_resolution` bugfix](https://github.com/asfadmin/hyp3-proc-lib/pull/4),
respectively

## [2.0.2](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.1...v2.0.2)
## [2.0.2](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.1...v2.0.2)

### Changed
* The v2 entrypoint will now make up to three retry attempts if it fails to download the input
Expand All @@ -144,14 +164,14 @@ and uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
* Removed custom blank_bad_data.py from mk_geo_radcal processing. Border pixels for older GRD products
are now cleaned using the default `make_edge` setting of `par_S1_GRD`.

## [2.0.1](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v2.0.0...v2.0.1)
## [2.0.1](https://github.com/ASFHyP3/hyp3-gamma/compare/v2.0.0...v2.0.1)

### Changed
* hyp3-v2 output products are now packaged in a .zip file similar to hyp3-v1 products

## [2.0.0](https://github.com/ASFHyP3/hyp3-rtc-gamma/compare/v1.3.1...v2.0.0)
## [2.0.0](https://github.com/ASFHyP3/hyp3-gamma/compare/v1.3.1...v2.0.0)

This is a significant refactor of `hyp3-rtc-gamma` into:
This is a significant refactor of `hyp3-gamma` into:
* A `pip` installable package called `hyp3_rtc_gamma`
* A stand alone, container-based HyP3 plugin

Expand Down
12 changes: 6 additions & 6 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,13 @@ FROM ubuntu:18.04

# For opencontainers label definitions, see:
# https://github.com/opencontainers/image-spec/blob/master/annotations.md
LABEL org.opencontainers.image.title="HyP3 RTC GAMMA"
LABEL org.opencontainers.image.description="HyP3 plugin for radiometric terrain correction using GAMMA"
LABEL org.opencontainers.image.title="HyP3 GAMMA"
LABEL org.opencontainers.image.description="HyP3 plugin for SAR processing with GAMMA"
LABEL org.opencontainers.image.vendor="Alaska Satellite Facility"
LABEL org.opencontainers.image.authors="ASF APD/Tools Team <[email protected]>"
LABEL org.opencontainers.image.licenses="BSD-3-Clause"
LABEL org.opencontainers.image.url="https://github.com/ASFHyP3/hyp3-rtc-gamma"
LABEL org.opencontainers.image.source="https://github.com/ASFHyP3/hyp3-rtc-gamma"
LABEL org.opencontainers.image.url="https://github.com/ASFHyP3/hyp3-gamma"
LABEL org.opencontainers.image.source="https://github.com/ASFHyP3/hyp3-gamma"
# LABEL org.opencontainers.image.documentation=""

# Dynamic lables to define at build time via `docker build --label`
Expand Down Expand Up @@ -40,7 +40,7 @@ RUN export CPLUS_INCLUDE_PATH=/usr/include/gdal && \
ARG S3_PYPI_HOST
ARG SDIST_SPEC

RUN python3 -m pip install --no-cache-dir hyp3_rtc_gamma${SDIST_SPEC} \
RUN python3 -m pip install --no-cache-dir hyp3_gamma${SDIST_SPEC} \
--trusted-host "${S3_PYPI_HOST}" \
--extra-index-url "http://${S3_PYPI_HOST}"

Expand All @@ -65,5 +65,5 @@ ENV GAMMA_RASTER=BMP

WORKDIR /home/conda/

ENTRYPOINT ["/usr/local/bin/rtc_gamma"]
ENTRYPOINT ["/usr/local/bin/hyp3_gamma"]
CMD ["-h"]
2 changes: 1 addition & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
include LICENSE
include README.md

graft hyp3_rtc_gamma
graft hyp3_gamma

global-exclude *.py[cod] __pycache__ *.so
34 changes: 24 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,37 @@
# HyP3 RTC GAMMA
# HyP3 GAMMA

HyP3 plugin for radiometric terrain correction using GAMMA
HyP3 plugin for SAR processing with GAMMA

## Developer Setup

Ubuntu 18.0.4 is recommended for GAMMA support.

1. Install GAMMA
1. Install [conda](https://docs.conda.io/en/latest/miniconda.html)
1. Install hyp3-rtc-gamma
1. Install `hyp3_gamma`
```
git clone [email protected]:ASFHyP3/hyp3-rtc-gamma.git
cd hyp3-rtc-gamma
git clone [email protected]:ASFHyP3/hyp3-gamma.git
cd hyp3-gamma
conda env create -f conda-env.yml
conda activate hyp3-rtc-gamma
pip install -e .
conda activate hyp3-gamma
pip install -e .[develop]
```
1. Run It!
1. Check hyp3_gamma is installed
```
rtc_sentinel.py --help
rtc_sentinel.py S1A_IW_SLC__1SSV_20150621T120220_20150621T120232_006471_008934_72D8.zip
hyp3_gamma --help
```

### Radiometric Terrain Correction (RTC)

To run the RTC science process:
```
rtc_sentinel.py --help
rtc_sentinel.py S1A_IW_SLC__1SSV_20150621T120220_20150621T120232_006471_008934_72D8.zip
```

To run the RTC process through the HyP3 interface:
```
hyp3_gamma ++process rtc --help
hyp3_gamma ++process rtc --username ${EDL_USERNAME} --password ${EDL_PASSWORD} \
S1A_IW_SLC__1SSV_20150621T120220_20150621T120232_006471_008934_72D8
```
2 changes: 1 addition & 1 deletion conda-env.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: hyp3-rtc-gamma
name: hyp3-gamma
channels:
- conda-forge
- nodefaults
Expand Down
2 changes: 1 addition & 1 deletion hyp3_rtc_gamma/__init__.py → hyp3_gamma/__init__.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""HyP3 plugin for radiometric terrain correction using GAMMA"""
"""HyP3 plugin for processing SAR data with GAMMA"""

# FIXME: Python 3.8+ this should be `from importlib.metadata...`
from importlib_metadata import PackageNotFoundError, version
Expand Down
8 changes: 4 additions & 4 deletions hyp3_rtc_gamma/__main__.py → hyp3_gamma/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,20 +16,20 @@
from hyp3lib.util import string_is_true
from pkg_resources import load_entry_point

from hyp3_rtc_gamma.rtc_sentinel import rtc_sentinel_gamma
from hyp3_gamma.rtc.rtc_sentinel import rtc_sentinel_gamma


def main():
parser = ArgumentParser(prefix_chars='+', formatter_class=ArgumentDefaultsHelpFormatter)
parser.add_argument(
'++entrypoint', choices=['hyp3_rtc_gamma_v2'], default='hyp3_rtc_gamma_v2',
'++process', choices=['rtc'], default='rtc',
help='Select the HyP3 entrypoint version to use'
)
args, unknowns = parser.parse_known_args()

sys.argv = [args.entrypoint, *unknowns]
sys.argv = [args.process, *unknowns]
sys.exit(
load_entry_point('hyp3_rtc_gamma', 'console_scripts', args.entrypoint)()
load_entry_point('hyp3_gamma', 'console_scripts', args.process)()
)


Expand Down
Empty file added hyp3_gamma/rtc/__init__.py
Empty file.
File renamed without changes.
Loading

0 comments on commit be2f89b

Please sign in to comment.