[IT-3230] Move agora data manager CI

.github/CODEOWNERS

@@ -0,0 +1 @@
+* @Sage-Bionetworks/sagebio-it @Sage-Bionetworks/Agora-Admin

.github/workflows/main.yaml

@@ -0,0 +1,30 @@
+name: main
+
+on:
+  push:
+    branches: [ 'develop','staging','prod' ]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-22.04
+    # context environment variables are setup in github to match branch names
+    environment:
+      name: ${{ github.ref_name }}
+
+    steps:
+      - uses: actions/checkout@v4
+      - uses: BSFishy/pip-action@v1
+        with:
+          packages: |
+            synapseclient[pandas,pysftp]
+      - name: Setup mongo tools - mongo-tools and mongo-shell
+        uses: boly38/action-mongo-tools@stable
+      - name: Import Synapse Data
+        run: ./import-data.sh $BRANCH $SYNAPSE_USERNAME $SYNAPSE_PASSWORD $DB_HOST $DB_USER $DB_PASS
+        env:
+          BRANCH: ${{ github.ref_name }}
+          SYNAPSE_USERNAME: ${{ secrets.SYNAPSE_USERNAME }}
+          SYNAPSE_PASSWORD: ${{ secrets.SYNAPSE_PASSWORD }}
+          DB_HOST: ${{ vars.DB_HOST }}
+          DB_USER: ${{ vars.DB_USER }}
+          DB_PASS: ${{ vars.DB_PASS }}

README.md

@@ -15,34 +15,42 @@ self-service update.
 To deploy an updated data version to the Agora development database
 1. Increment `data-version` in `data-manifest.json` on the `develop` branch.
 2. Commit the change
-3. The [CI system](https://travis-ci.org/Sage-Bionetworks/agora-data-manager) automatically updates the dev DB
+3. The Github action CI system automatically updates the dev DB
 
 
 To deploy an updated data version to the Agora staging database:
 1. Merge the data-version update from the dev branch to the staging branch.
-2. The [CI system](https://travis-ci.org/Sage-Bionetworks/agora-data-manager) automatically updates the staging DB
+2. The Github action CI system automatically updates the dev DB
 
 To deploy an updated data version to the Agora production database:
 1. Merge the data-version update from the staging branch to the production branch.
-2. The [CI system](https://travis-ci.org/Sage-Bionetworks/agora-data-manager) automatically updates the production DB
+2. The Github action CI system automatically updates the dev DB
 
 
 # Setup
 
-The following environment variables need to be setup for the scripts to deploy database updates:
+The following secrets need to be setup in Github for the scripts to deploy database updates:
 
-| Variable             | Description                       | Example                                                                   |
-|----------------------|-----------------------------------|---------------------------------------------------------------------------|
-| BASTIAN_HOST_develop | The bastian host                  | ec2-10-11-12-13.compute-1.amazonaws.com                                   |
-| DB_HOST_develop      | The database host                 | dbcluster-mr0a782pfjnk.cluster-ctcayu3de2lt.us-east-1.docdb.amazonaws.com |
-| DB_USER_develop      | The database user                 | dbuser                                                                    |
-| DB_PASS_develop      | The database password             | supersecret                                                               |
-| SYNAPSE_USERNAME     | The Synapse service user          | syn-service-user                                                          |
-| SYNAPSE_PASSWORD     | The Synapse service user password | supersecret                                                               |
+Global secrets:
 
-__Note__: The variables containing `_develop` postfix corresponds to the branch.
-To deploy to a prod environment a prod branch is require along with a variable
-containing a `_prod` prefix (i.e. BASTIAN_HOST_prod)
+| Variable             | Description                       | Example                     |
+|----------------------|-----------------------------------|-----------------------------|
+| SYNAPSE_USERNAME     | The Synapse service user          | syn-service-user            |
+| SYNAPSE_PASSWORD     | The Synapse service user password | supersecret                 |
 
 
-[db_update]: diagram1.png "update diagram"
+Context specific secrets for each environment that corresponds to a git branch (develop/staging/prod):
+
+| Variable  | Description                 | Example                                                                   |
+|-----------|-----------------------------|---------------------------------------------------------------------------|
+| DB_HOST   | The database host           | dbcluster-mr0a782pfjnk.cluster-ctcayu3de2lt.us-east-1.docdb.amazonaws.com |
+| DB_USER   | The database user           | dbuser                                                                    |
+| DB_PASS   | The database password       | supersecret                                                               |
+
+
+![alt text][github_secrets]
+
+
+
+[db_update]: agora-db-update.drawio.png "update diagram"
+[github_secrets]: github_secrets.png "github secrets screen"

import-data.sh

@@ -5,7 +5,7 @@
 #!/bin/bash
 set -e
 
-TRAVIS_BRANCH=$1
+BRANCH=$1
 SYNAPSE_USERNAME=$2
 SYNAPSE_PASSWORD=$3
 DB_HOST=$4
@@ -23,7 +23,7 @@ TEAM_IMAGES_DIR=$DATA_DIR/team_images
 DATA_VERSION=$(cat $WORKING_DIR/data-manifest.json | grep data-version | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | tr -d '[[:space:]]')
 DATA_MANIFEST_ID=$(cat $WORKING_DIR/data-manifest.json | grep data-manifest-id | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | tr -d '[[:space:]]')
 TEAM_IMAGES_ID=$(cat $WORKING_DIR/data-manifest.json | grep team-images-id | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | tr -d '[[:space:]]')
-echo "$TRAVIS_BRANCH branch, DATA_VERSION = $DATA_VERSION, manifest id = $DATA_MANIFEST_ID"
+echo "$BRANCH branch, DATA_VERSION = $DATA_VERSION, manifest id = $DATA_MANIFEST_ID"
 
 # Download the manifest file from synapse
 synapse -u $SYNAPSE_USERNAME -p $SYNAPSE_PASSWORD get --downloadLocation $DATA_DIR -v $DATA_VERSION $DATA_MANIFEST_ID