Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout stream keep alive for Upgrades, Restores and Migrations #173

Merged
merged 4 commits into from
Jan 26, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
Add keep-alive to backup and restore data streaming to avoid timeout
rooftopcellist committed Jan 25, 2024
commit 84e81b5a482d865334feeb5a408b9776164b010d
20 changes: 18 additions & 2 deletions roles/backup/tasks/postgres.yml
Original file line number Diff line number Diff line change
@@ -99,11 +99,27 @@
namespace: "{{ backup_pvc_namespace }}"
pod: "{{ ansible_operator_meta.name }}-db-management"
command: |
bash -c """
bash -c "
function end_keepalive {
rc=$?
rm -f \"$1\"
kill $(cat /proc/$2/task/$2/children 2>/dev/null) 2>/dev/null || true
wait $2 || true
exit $rc
}
keepalive_file=\"$(mktemp)\"
while [[ -f \"$keepalive_file\" ]]; do
echo 'Dumping data from database...'
sleep 60
done &
keepalive_pid=$!
trap 'end_keepalive \"$keepalive_file\" \"$keepalive_pid\"' EXIT SIGINT SIGTERM
echo keepalive_pid: $keepalive_pid
set -e -o pipefail
PGPASSWORD='{{ eda_postgres_pass }}' {{ pgdump }} > {{ backup_dir }}/eda.db
set +e +o pipefail
echo 'Successful'
"""
"
register: data_migration
no_log: "{{ no_log }}"
failed_when: "'Successful' not in data_migration.stdout"
19 changes: 17 additions & 2 deletions roles/restore/tasks/postgres.yml
Original file line number Diff line number Diff line change
@@ -85,11 +85,26 @@
namespace: "{{ backup_pvc_namespace }}"
pod: "{{ ansible_operator_meta.name }}-db-management"
command: |
bash -c """
bash -c "
function end_keepalive {
rc=$?
rm -f \"$1\"
kill $(cat /proc/$2/task/$2/children 2>/dev/null) 2>/dev/null || true
wait $2 || true
exit $rc
}
keepalive_file=\"$(mktemp)\"
while [[ -f \"$keepalive_file\" ]]; do
echo 'Migrating data from old database...'
sleep 60
done &
keepalive_pid=$!
trap 'end_keepalive \"$keepalive_file\" \"$keepalive_pid\"' EXIT SIGINT SIGTERM
echo keepalive_pid: $keepalive_pid
set -e -o pipefail
cat {{ backup_dir }}/eda.db | PGPASSWORD='{{ eda_postgres_pass }}' {{ pg_restore }}
echo 'Successful'
"""
"
register: data_migration
no_log: "{{ no_log }}"
failed_when: "'Successful' not in data_migration.stdout"