Creating staging instances on Amazon

To save costs, we don't keep our staging system online all the time. Instead, we make it reasonable to be able to spin it up through scripts.

But the scripts aren't all written. This is what it takes to set up a staging environment.

Prerequisites

You'll need the overview-manage script on your desktop, described at the bottom of Deploying from scratch to Amazon. It's just a wrapper to ssh the manage instance.

You'll also need the prerequisites from Deploying from scratch to Amazon, so you can spin the instances up. (TODO: add spin-up functionality to overview-manage?)

Database

Rather than create a new database, we can create one from our latest backup. As a side effect, this verifies that our backups always work.

1. cd aws-overview-tools/instances

2. AWS_KEYPAIR_NAME=manage ./create_instance.rb database_staging us-east-1a

3. overview-manage add-instance staging/database/10.x.x.x

4. overview-manage ssh staging database

5. On the EC2 Management Console, go to "Snapshots", pick the most recent "[Database] Daily backup" and click "Create volume". Make it "Magnetic" and the same size as the original. (On EC2, it can't be any smaller.) Go to "Volumes", find the new volume (its status will be "Available" or "Loading...", the latter meaning it can be mounted but will initially be a bit slow), and "Attach" it to the database-staging instance under /dev/sdf.

6. Give the volume a Tag: Name database-staging.

7. In the staging-database instance:

    sudo /etc/init.d/postgresql stop && sudo rm -rf /var/lib/postgresql/* && echo '/dev/xvdf1 /var/lib/postgresql auto defaults 0 0' | sudo tee -a /etc/fstab && sudo mount /var/lib/postgresql && sudo /etc/init.d/postgresql start
   

8. Still in the instance: sudo -u postgres psql overview -c "ALTER USER overview PASSWORD 'overview'" (so if there's a security hole in the staging server, it won't expose our database password).

SearchIndex

1. AWS_KEYPAIR_NAME=manage ./create_instance.rb searchindex_staging us-east-1a

2. overview-manage add-instance staging/searchindex/10.x.x.x

3. On the EC2 Management Console, go to "Snapshots", pick the most recent searchindex "Daily backup"and click "Create volume". Make it "Magnetic" and the same size as the original. Go to "Volumes", find the new volume (its status will be "Available" or "Loading...", the latter meaning it can be mounted but will initially be a bit slow), and "Attach" it to the searchindex-staging instance under /dev/sdf.

4. On the EC2 Management Console, go to "Snapshots", pick the most recent apollo "Daily backup" and click "Create volume". Make it "Magnetic" and the same size as the original. Go to "Volumes", find the new volume (its status will be "Available" or "Loading...", the latter meaning it can be mounted but will initially be a bit slow), and "Attach" it to the worker-staging instance under /dev/sdg.

5. overview-manage ssh staging searchindex

     sudo mkdir /etc/elasticsearch
     sudo mkdir /var/lib/elasticsearch && echo '/dev/xvdf1 /var/lib/elasticsearch auto defaults 0 0' | sudo tee -a /etc/fstab && sudo mount /var/lib/elasticsearch
     sudo mkdir /etc/apollo
     sudo mkdir /var/lib/apollo && echo '/dev/xvdg /var/lib/apollo auto defaults 0 0' | sudo tee -a /etc/fstab && sudo mount /var/lib/apollo
     sudo mv /var/lib/elasticsearch/data/overview-search-index /var/lib/elasticsearch/data/overview-search-index-staging
   

Worker

1. AWS_KEYPAIR_NAME=manage ./create_instance.rb worker_staging us-east-1a
2. overview-manage add-instance staging/worker/10.x.x.x
3. overview-manage ssh staging worker and then log out

Web

1. AWS_KEYPAIR_NAME=manage ./create_instance.rb web_staging us-east-1a
2. overview-manage add-instance staging/web/10.x.x.x
3. overview-manage ssh staging web and then log out
4. Attach the Elastic IP for staging.overviewproject.org to the new instance.

All together

1. Run overview-manage deploy aws-overview-config@[treeish] staging
2. overview-manage ssh staging database and sudo /etc/init.d/postgresql restart to pick up the new configuration.
3. Run overview-manage deploy overview-server@[treeish] staging
4. Attach the correct AWS "Elastic IP" to the staging instance.

And test away, at http://staging.overviewproject.org

Differences between staging and production

• The staging database has a different password.
• The cookie domain is different. We don't have any cookies on overviewproject.org: we only have them on www.overviewproject.org and staging.overviewproject.org. That ensures no overlaps.
• Only on production do we redirect from http(?:s)://overviewproject.org to https://www.overviewproject.org.

Similarities between staging and production

• The network topology is the same
• The configuration files are near-identical. The only differences are specified in /opt/overview/config/manage/config.yml and secrets.yml on the manage instance.
• The staging database is a snapshot of the production one.
• The SSL key is the same.

Security concerns

• Our users' sensitive data is on the database-staging instance, so if that database is hacked, we expose real data. Our only comfort is that the hacker can't edit the live database (it's on a different computer, with a different password, in a different security group).

Shut down the staging instance when it isn't in use.

Shutting down

1. Run for instance in $(overview-manage status | grep staging | cut -f2,3,4 | sed -e 's/\t/\//g'); do overview-manage remove-instance $instance; done (that runs overview-manage remove-instance for every staging instance.

2. On the EC2 console, terminate all instances whose tags end in -staging.

3. Delete the database-staging, searchindex-staging, and apollo-staging volumes. (You may "force detach" first, if your instance is taking a while to shut down. But if you're doing "force detach", make sure you aren't detaching the production volume!)

Be extra careful not to terminate non-staging instances. That would lead to downtime (though we wouldn't lose data).

Be extra-extra careful not to delete the non-staging volume. That would lead to up to 24 hours' worth of data loss, since we'd need to recover from a snapshot. (Incidentally, the instructions on this page double as backup-recovery instructions.)