Add/lifecycle heartbeat #1116

hyeong01 · 2025-01-10T03:47:10Z

Issue #, if available:
#493 NTH should issue lifecycle heartbeats

Description of changes:

Added the option to issue heartbeats

Configurable interval and length

Added heartbeat configuration validity check
Added automatic closure when lifecycle action becomes invalid
Added unit and E2E testing
Added explanation of the feature to readme

How you tested your changes:
Environment (Linux / Windows): Linux
Kubernetes Version: v1.31.3

Unit testing:

Checked heartbeat signals and closure due to heartbeat expiration
Checked heartbeat signals and closure due to drain completion
Checked heartbeat failures due to invalid lifecycle hook target
Checked heartbeat failures due to not invalid lifecycle hook target

E2E testing:

E2E tested heartbeat signal and closure due to heartbeat expiration with kind and localstack.

Manual testing with real ASG and K8s:

Tested all 32 possible configuration cases
Tested interval > timeout case
Graceful termination 5->3, 5->4, 10->3, 103->3 (corresponding number of workers for terminating instances) with varying intervals and length for each.
Important example 1: # of terminations=103->3, Interval=300, heartbeatUntil=590, timeout=590. Last interruption event entering the processInterruptionEvent took 223 seconds since the first event was stored to the eventStore. The first heartbeat was sent out 90 seconds after the last interruption event entering.
Important example 2: # of terminations=103->3, Interval=90, heartbeatUntil=260, timeout=150. The timeout was not enough for the last heartbeat to be sent out.
Ungraceful termination 5->3, 10->3, 20->3, 103->3 (smaller number of workers than terminating instances).

Potential Improvements:
Cached API call for getting heartbeat timeout from ASG (describeLifecycleHook).

Details: 1 hour cache for retrieving the heartbeat timeout value and warning user if timeout < interval.
Pros: Reduced number of API calls.
Cons: Complexity of the system. Cache expires or not used often because termination events happen sparsely.

Single heartbeat manager (single gorountine for issuing heartbeats)

Pros: Reduced number of goroutines
Cons: Potential burst in API calls. goroutine is light and the number of goroutines does not grow more than the number of workers (default 10).

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Lu-David · 2025-01-10T19:18:30Z

Regarding potential improvements: I think caching describeLifecycleHook is unnecessary because like you said termination events are not that frequent and would add unnecessary complexity. Regarding single heartbeat manager, I think it also adds unnecessary complexity

Thanks for doing all the thorough manual testing! Can we also test with windows nodes?

pkg/monitor/sqsevent/asg-lifecycle-event.go

Lu-David · 2025-01-10T19:33:41Z

README.md

+
+### Important Notes
+
+- A lifecycle hook for instance termination is required for this feature. Longer grace periods are achieved by renewing the heartbeat timeout of the ASG's lifecycle hook. Instances terminate instantly without a hook.


I think this section is important enough that it could be described in the How to use section? The How to use section currently seems sparse.

Agree with David on this comment.

Updated this part with detailed and extensive explanation

Lu-David · 2025-01-10T19:48:53Z

test/e2e/asg-lifecycle-sqs-heartbeat-test

+ASG_TERMINATE_EVENT_ONE_LINE=$(echo "${ASG_TERMINATE_EVENT}" | tr -d '\n' |sed 's/\"/\\"/g')
+SEND_SQS_CMD="awslocal sqs send-message --queue-url ${queue_url} --message-body \"${ASG_TERMINATE_EVENT_ONE_LINE}\" --region ${AWS_REGION}"
+kubectl exec -i "${localstack_pod}" -- bash -c "$SEND_SQS_CMD"
+echo "✅ Sent Spot Interruption Event to SQS queue: ${queue_url}"


nit: ASG termination event

Lu-David · 2025-01-10T19:53:04Z

test/e2e/asg-lifecycle-sqs-heartbeat-test

+    if [[ $FOUND_HEARTBEAT_END_LOG -eq 0 ]] && kubectl logs -n kube-system "${NTH_POD}" | grep -q "Heartbeat deadline exceeded, stopping heartbeat"; then
+        FOUND_HEARTBEAT_END_LOG=1
+    fi
+    if [[ $HEARTBEAT_COUNT -eq 3 && $FOUND_HEARTBEAT_END_LOG -eq 1 ]]; then


nit: Instead of hardcoding these values, can we abstract the values of HEARTBEAT_INTERVAL and HEARTBEAT_UNTIL to figure out expected number of Heartbeats at the top of the file? Just to make it easier to update this test file if need be

What is your opinion about auto-calculating the HEARTBEAT_CHECK_CYCLES and HEARTBEAT_CHECK_SLEEP as well? They also derived from HEARTBEAT_INTERVAL and HEARTBEAT_UNTIL, it might be more consistent to either auto-calculate all related values or none of them. This would make the code easier to understand?

yeah I think that would be helpful thanks

Lu-David · 2025-01-10T19:56:23Z

test/e2e/asg-lifecycle-sqs-heartbeat-test

+
+    if [[ $cordoned -eq 1 && $(kubectl get deployments regular-pod-test -o=jsonpath='{.status.unavailableReplicas}') -eq 1 ]]; then
+        echo "✅ Verified the regular-pod-test pod was evicted!"
+        echo "✅ ASG Lifecycle SQS Test Passed $CLUSTER_NAME! ✅"


nit: "ASG Lifecycle SQS Test Passed with Heartbeat"

Just to avoid ambiguity with other tests that we run?

Lu-David · 2025-01-10T20:03:57Z

pkg/monitor/sqsevent/asg-lifecycle-event.go

+	heartbeatTimeout := int(*lifecyclehook.LifecycleHooks[0].HeartbeatTimeout)
+
+	if heartbeatInterval >= heartbeatTimeout {
+		log.Warn().Msgf("Heartbeat interval (%d seconds) is equal to or greater than the heartbeat timeout (%d seconds) for the lifecycle hook %s. The node would likely be terminated before the heartbeat is sent", heartbeatInterval, heartbeatTimeout, *lifecyclehook.LifecycleHooks[0].LifecycleHookName)


can we also add the ASG name (lifecycleDetail.AutoScalingGroupName) in this log warn? Just to help with debugging?

+1 to David's comment

LikithaVemulapalli

Left some inline comments, good work @hyeong01 !

README.md

LikithaVemulapalli · 2025-01-15T21:30:14Z

README.md

+
+### Important Notes
+
+- A lifecycle hook for instance termination is required for this feature. Longer grace periods are achieved by renewing the heartbeat timeout of the ASG's lifecycle hook. Instances terminate instantly without a hook.


Agree with David on this comment.

pkg/config/config.go

pkg/monitor/sqsevent/asg-lifecycle-event.go

LikithaVemulapalli · 2025-01-15T21:49:09Z

pkg/monitor/sqsevent/asg-lifecycle-event.go

+	heartbeatTimeout := int(*lifecyclehook.LifecycleHooks[0].HeartbeatTimeout)
+
+	if heartbeatInterval >= heartbeatTimeout {
+		log.Warn().Msgf("Heartbeat interval (%d seconds) is equal to or greater than the heartbeat timeout (%d seconds) for the lifecycle hook %s. The node would likely be terminated before the heartbeat is sent", heartbeatInterval, heartbeatTimeout, *lifecyclehook.LifecycleHooks[0].LifecycleHookName)


+1 to David's comment

pkg/monitor/sqsevent/sqs-monitor_test.go

…ation

hyeong01 · 2025-01-22T22:42:00Z

On top of the changes suggested from the comment, I modified the logging to reduce duplicate logs and added a check that prevents hearbeatUntil < heartbeatInterval.

LikithaVemulapalli

left some comments

LikithaVemulapalli · 2025-01-23T05:24:20Z

README.md

+- When NTH receives an ASG lifecycle termination event, it starts sending heartbeats to ASG to renew the heartbeat timeout associated with the ASG's termination lifecycle hook.
+- The heartbeat timeout acts as a timer that starts when the termination event begins.
+- Before the timeout reaches zero, the termination process is halted at the `Terminating:Wait` stage.
+- Previously, NTH couldn't issue heartbeats, limiting the maximum time for preventing termination to the maximum heartbeat timeout (7200 seconds).


This wasn't expected behavior of NTH when it was implemented, we shouldn't mention terms like previously in README, we should only concentrate on how the current functionality works...

LikithaVemulapalli · 2025-01-23T05:29:33Z

README.md

+##### How to use
+
+- Configure a termination lifecycle hook on ASG (required). Set the heartbeat timeout value to be longer than the `Heartbeat Interval`. Each heartbeat signal resets this timeout, extending the duration that an instance remains in the `Terminating:Wait` state. Without this lifecycle hook, the instance will terminate immediately when termination event occurs.
+- Configure `Heartbeat Interval` (required) and `Heartbeat Until` (optional). NTH operates normally without heartbeats if neither value is set. If only the interval is specified, `Heartbeat Until` defaults to 172800 seconds (48 hours) and heartbeats will be sent. Providing both values enables NTH to run with heartbeats. `Heartbeat Until` must be provided with a valid `Heartbeat Interval`, otherwise NTH will fail to start. Any invalid values (wrong type or out of range) will also prevent NTH from starting.


Providing both values enables NTH to run with heartbeats. This statement is confusing, in the previous line you mentioned that if Interval is set then we will send heart beats, again mentioning this immediately makes me think that it is mandatory to set both values, could you please remove this?

Plus 1 to this point

README.md

LikithaVemulapalli · 2025-01-23T17:46:22Z

pkg/config/config.go

@@ -166,6 +169,8 @@ type Config struct {
 	CompleteLifecycleActionDelaySeconds int
 	DeleteSqsMsgIfNodeNotFound          bool
 	UseAPIServerCacheToListPods         bool
+	HeartbeatInterval                   int


We need to have some coverage around these newly added configs in config-test.go file...

LikithaVemulapalli · 2025-01-23T18:25:12Z

test/e2e/asg-lifecycle-sqs-heartbeat-test

+    fail_and_exit 2
+fi
+
+ASG_TERMINATE_EVENT=$(cat <<EOF


nit: ASG_TERMINATION_EVENT

LikithaVemulapalli · 2025-01-23T18:25:32Z

test/e2e/asg-lifecycle-sqs-heartbeat-test

+EOF
+)
+
+ASG_TERMINATE_EVENT_ONE_LINE=$(echo "${ASG_TERMINATE_EVENT}" | tr -d '\n' |sed 's/\"/\\"/g')


same as above TERMINATION

Lu-David · 2025-01-23T18:27:51Z

README.md

+#### Issuing Lifecycle Heartbeats
+
+You can set NTH to send heartbeats to ASG in Queue Processor mode. This allows for a much longer grace period (up to 48 hours) for termination than the maximum heartbeat timeout of two hours.
+


I would add a line that explains when this feature would be useful: e.g. When a customer has pods that have long-running drain tasks.

hyeong01 · 2025-01-23T21:23:42Z

Changes made, thank you for your valuable insight!

LikithaVemulapalli

/lgtm

Heeyoung Jung and others added 17 commits December 30, 2024 18:44

add lifecycle heartbeat

d991814

Lifecycle heartbeat unit test

0e4b686

Refactor heartbeat logging statements

1fbd7cb

Heartbeat e2e test

d0f1ef4

Merge branch 'aws:main' into add/lifecycle-heartbeat

df0696f

Remove error handling for using heartbeat and imds together

d7f8e07

add e2e test for lifecycle heartbeat

a6cfd89

Add check heartbeat timeout and compare to heartbeat interval

64e9cff

Add error handling for using heartbeat and imds together

d3047a0

fix config error message

559adc3

update error message for heartbeat config

7012bab

Fix heartbeat flag explanation

bc79eb7

Update readme for new heartbeat feature

75400a9

Fix readme for heartbeat section

bbddcfa

Update readme on the concurrency of heartbeat

029fdf7

fix: stop heartbeat when target is invalid

56b3f55

Added heartbeat test for handling invalid lifecycle action

7221ed2

hyeong01 requested a review from a team as a code owner January 10, 2025 03:47

incorporated unsupoorted error types for unit testing

4bcb916

Lu-David reviewed Jan 10, 2025

View reviewed changes

pkg/monitor/sqsevent/asg-lifecycle-event.go Show resolved Hide resolved

Lu-David reviewed Jan 10, 2025

View reviewed changes

fix unit-test: reset heartbeatCallCount each test

4ff40d9

Lu-David reviewed Jan 10, 2025

View reviewed changes

Merge branch 'aws:main' into add/lifecycle-heartbeat

fe7fcc1

LikithaVemulapalli requested changes Jan 16, 2025

View reviewed changes

use helper function to reduce repetitive code in heartbeat unit test

265828d

Heeyoung Jung added 5 commits January 22, 2025 14:25

Update readme. Moved heartbeat under Queue Processor

044fc3a

Fix config.go for better readability and check until < interval

2732775

Update heartbeat to have better logging

0492976

Update unit test to cover whole process of heartbeat start and closure

1631bb6

Update heartbeat e2e test. Auto-value calculations for future modific…

b41751d

…ation

Heeyoung Jung added 2 commits January 22, 2025 14:49

Add inline comment for heartbeatUntil default behavior

9e3fe77

Fixed e2e variables to have double quotes

dbdeec1

LikithaVemulapalli requested changes Jan 23, 2025

View reviewed changes

Lu-David reviewed Jan 23, 2025

View reviewed changes

Heeyoung Jung added 3 commits January 23, 2025 13:09

fix readme for heartbeat

80b88a4

Added new flags in config test

9c54964

Fixed typo in heartbeat e2e test

56ea41d

LikithaVemulapalli approved these changes Jan 28, 2025

View reviewed changes

Lu-David approved these changes Jan 29, 2025

View reviewed changes

LikithaVemulapalli merged commit d483d25 into aws:main Jan 29, 2025
11 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add/lifecycle heartbeat #1116

Add/lifecycle heartbeat #1116

hyeong01 commented Jan 10, 2025 •

edited

Loading

Lu-David commented Jan 10, 2025 •

edited

Loading

Lu-David Jan 10, 2025

LikithaVemulapalli Jan 15, 2025

hyeong01 Jan 22, 2025

Lu-David Jan 10, 2025

Lu-David Jan 10, 2025

hyeong01 Jan 16, 2025 •

edited

Loading

Lu-David Jan 16, 2025

Lu-David Jan 10, 2025

Lu-David Jan 10, 2025

LikithaVemulapalli Jan 15, 2025

LikithaVemulapalli left a comment

LikithaVemulapalli Jan 15, 2025

LikithaVemulapalli Jan 15, 2025

hyeong01 commented Jan 22, 2025

LikithaVemulapalli left a comment

LikithaVemulapalli Jan 23, 2025

LikithaVemulapalli Jan 23, 2025

Lu-David Jan 23, 2025

LikithaVemulapalli Jan 23, 2025

LikithaVemulapalli Jan 23, 2025

LikithaVemulapalli Jan 23, 2025

Lu-David Jan 23, 2025

hyeong01 commented Jan 23, 2025

LikithaVemulapalli left a comment


		### Important Notes

		- A lifecycle hook for instance termination is required for this feature. Longer grace periods are achieved by renewing the heartbeat timeout of the ASG's lifecycle hook. Instances terminate instantly without a hook.

		#### Issuing Lifecycle Heartbeats

		You can set NTH to send heartbeats to ASG in Queue Processor mode. This allows for a much longer grace period (up to 48 hours) for termination than the maximum heartbeat timeout of two hours.

Add/lifecycle heartbeat #1116

Add/lifecycle heartbeat #1116

Conversation

hyeong01 commented Jan 10, 2025 • edited Loading

Lu-David commented Jan 10, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyeong01 Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

LikithaVemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyeong01 commented Jan 22, 2025

LikithaVemulapalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hyeong01 commented Jan 23, 2025

LikithaVemulapalli left a comment

Choose a reason for hiding this comment

hyeong01 commented Jan 10, 2025 •

edited

Loading

Lu-David commented Jan 10, 2025 •

edited

Loading

hyeong01 Jan 16, 2025 •

edited

Loading