[IT-3230] Move agora data manager CI #116
Conversation
Move agora data manager from using travis CI to Github actions CI. This move also changes the update workflow to not require an AWS bastion host which will be much more secure. Another benefit is that it's also a more simple setup.
Once we setup DB updates from github instances we will no longer need an AWS bastian host for each environment. depends on Sage-Bionetworks/agora-data-manager#116
|
Is the DB_HOST in a public subnet? In import-data.sh, I see mongoimport connects to it directly from the runner, correct? |
no, the db host is in a private subnet. The GH runner has access to it via PR Sage-Bionetworks-IT/organizations-infra#1060 |
|
The GH runner has the right to make AWS API calls to setup DocumentDB. Should not mongoimport try to connect to the DB_HOST with the DB_USER/PWD creds? If so DB_HOST will resolve to a private IP address and it will not be able to find the host (I assume it works like an RDS setup). |
Argg! you are correct @xschildw. Will need to rethink this whole setup. |
|
I was thinking that an alternative solution would be to setup AWS CLI + aws-session-manager to execute those mongo commands however it appears that AWS does not support running session manager agent on RDS instances, https://repost.aws/questions/QU9-N7fbk-SLOZvegK63XrRw/using-session-manager-to-connect-rds-without-having-ec2-instance The only other alternative i can think of is to use GH custom runners but i’m not a fan of that solution because it requires a bit of manual configuration to setup. |
|
Changed to use GH self hosted runners |
i added some more docs around that topic but mostly just followed instructions from the github documentation. |
The bastian security group allow SSH access. We can now close that access by setting up self hosted runners in GH actions. This will make the hosts more secure. depends on Sage-Bionetworks/agora-data-manager#116
The bastian security group allow SSH access. We can now close that access by setting up self hosted runners in GH actions. This will make the hosts more secure. depends on Sage-Bionetworks/agora-data-manager#116
Revert creation of a role for GH OIDC because our plan shifted to using self-hosted runner insteead of a GH runner. A role is not needed when using self-hosted runners because the self hosted runner is already setup with access to the databases. more info in PR Sage-Bionetworks/agora-data-manager#116 This reverts commit d1f91e1 and commit a6406aa.
We switch to using GH self hosted runner therefore the bastian hosts do not need to be accessible from the internet anymore. depends on Sage-Bionetworks/agora-data-manager#116
We switch to using GH self hosted runner therefore the bastian hosts do not need to be accessible from the internet anymore. depends on Sage-Bionetworks/agora-data-manager#116
Move agora data manager from using travis CI to Github actions CI. This move also changes the update workflow to us github self-hosted runners which will be much more secure.
depends on Sage-Bionetworks-IT/organizations-infra#1060