Data backup and restore

We follow a common backup, restore and purging process for all data

Backup Process

• Data backup is scheduled via *_Backup jobs in jenkins. These jobs run respective *-backup ansible roles in this repo.
• Backup jobs run once every day @midnight
• Data backup is compressed and uploaded to Azure blob storage for long term storage

Restore Process

• Restore from backup can be done via *_Restore jenkins jobs. These jobs run respective *-restore ansible roles in this repo
• Restore jobs take backup_name as parameter to decide backup from which time should be restored
• Restore job are run on demand

Purging Process

• Purging of backup in azure blob storage is scheduled using azure logic app. The steps are listed in Azure blob storage purge setup
• Backup data is retained for 30 days by default
• Purging has been setup for postgresql, cassandra and jenkins.
• Elasticsearch does incremental backup and snapshots, hence no purging is required
• Mongodb will removed soon, hence no purging has been setup

Databases

Data stored in following databases are periodically backed up using process mentioned above

• Postgresql
• ElasticSearch
• Cassandra
• MongoDB

Jenkins

• Jenkins thin backup plugin is used for jenkins backup.
• This plugin is scheduled to run @midnight. It saves a new backup folder for each backup
• The Jenkins_Backup job compresses latest backup folder and uploads to azure blob storage. This is scheduled to run every day