Data backup and restore

We follow a common backup, restore and purging process for all data

Backup Process

Data backup is scheduled via *_Backup jobs in jenkins. These jobs run respective *-backup ansible roles in this repo.
Backup jobs run once every day @midnight
Data backup is compressed and uploaded to Azure blob storage for long term storage

Data stored in following databases are periodically backed up using process mentioned above

Postgresql - Full DB backup us taken using postgresql-backup role which uses pg_dump_all internally.
ElasticSearch - Snapshot based backups are stored in azure using [azure repository plugin]. Uses ansible role es-azure-snapshot (https://www.elastic.co/guide/en/elasticsearch/plugins/5.6/repository-azure.html)
Cassandra - Full DB backups taken using ansible role cassandra-backup
MongoDB - Full DB backups taken using ansible role mongo-backup

Jenkins thin backup plugin is used for jenkins backup.
This plugin is scheduled to run @midnight. It saves a new backup folder for each backup
The Jenkins_Backup job compresses latest backup folder and uploads to azure blob storage. This is scheduled to run every day

Restore from backup can be done via *_Restore jenkins jobs. These jobs run respective *-restore ansible roles in this repo
Restore jobs take backup_name as parameter to decide backup from which time should be restored
Restore job are run on demand

Purging of backup in azure blob storage is scheduled using azure logic app. The steps are listed in Azure blob storage purge setup
Backup data is retained for 30 days by default
Purging has been setup for postgresql, cassandra and jenkins.
Elasticsearch does incremental backup and snapshots, hence no purging is required
Mongodb will removed soon, hence no purging has been setup