Collect volume metrics and volume clean up for EBS-backed tasks

[mye956]

Summary

This PR will introduce the functionality to collect volume metrics for EBS-backed tasks via the CSI driver as well as task clean up for EBS-backed task.

Implementation details

Changes made to collect Volume metrics in EBS-backed Tasks

Implemented getEBSVolumeMetrics() and fetchEBSVolumeMetrics() where it'll collect the volume metrics via NodeGetVolumeStats from the CSI driver for each EBS-backed tasks. The getEBSVolumeMetrics() will be invoked when getting all of the other task and instance metrics in GetInstanceMetrics().

IsEBSTaskAttachEnabled() has also been implemented and will scan for any EBS volume configuration within the list of volumes of a task. Although it's being called in AddTask() within a check, it will temporarily always return false for all tasks (i.e. consider it not EBS-backed task) in order to avoid invoking the CSI driver functionality. This will be fixed and addressed in another PR.

List of files changed:

• agent/api/task/task.go: Added functionality to IsEBSTaskAttachEnabled()
• agent/engine/docker_task_engine.go: Temporarily assume that all tasks are not EBS-backed, Will be fixed in a future PR
• agent/stats/engine.go
• agent/stats/engine_test.go
• agent/stats/engine_unix.go: Essentially calls upon the CSI driver to run NodeGetVolumeStats on all EBS volumes attached to EBS-backed tasks
• agent/stats/engine_unix_test.go
• agent/stats/engine_windows.go
• ecs-agent/csiclient/csi_client.go

Changes made to clean up EBS-backed tasks

Implemented UnstageVolumes to unmount the host mountpoint for all EBS volumes on a EBS-backed task. This will happen on task clean up.

List of files changed:

• agent/engine/task_manager.go

Testing

Added new unit test.

TestFetchEBSVolumeMetrics: Test that the new fetchEBSVolumeMetrics function is working as intended. As a TODO, we'll need to add a unhappy test case as well.

TODO: Add unit test for NodeUnstage in a follow up PR.

Manual testing:
We've manually mounted/staged an EBS volume onto a testing EC2 instance with agent running and tried calling the getEBSVolumeMetrics functionality.
Agent logs to get volume metrics

level=debug time=2023-09-26T04:14:54Z msg="Fetching EBS volume metrics..."
level=debug time=2023-09-26T04:14:54Z msg="Is an ebs volume configuration"
level=debug time=2023-09-26T04:14:54Z msg="Found volume usage" UsedBytes=108707840 TotalBytes=10726932480
level=debug time=2023-09-26T04:14:54Z msg="Ignore inodes key"
level=debug time=2023-09-26T04:14:54Z msg="EBS TACS Metrics collected! UsedBytes: 1.0870784e+08 TotalBytes: 1.072693248e+10"
level=debug time=2023-09-26T04:14:54Z msg="sent telemetry message" module=engine.go

CSI driver container NodeGetVolumeStats logs

I1009 18:49:17.146637       1 node.go:303] "NodeGetVolumeStats: called" args={"volume_id":"vol-0f00d7ebcc1c0993f","volume_path":"/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"}
I1009 18:49:37.145296       1 node.go:303] "NodeGetVolumeStats: called" args={"volume_id":"vol-0f00d7ebcc1c0993f","volume_path":"/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"}

Agent logs to unstage volume

level=debug time=2023-10-09T18:17:16Z msg="No more tasks could be started at this moment, waiting"
level=debug time=2023-10-09T18:17:16Z msg="Successfully unstaged volume" Task="web-app:2 arn:aws:ecs:us-west-2:113424923516:task/evm-test-gamma/20b60545e5214c43818e73aa8f95550a, TaskStatus: (STOPPED->STOPPED) N Containers: 1, N ENIs 0"

CSI driver container NodeUnstageVolume logs

I1009 18:49:39.828644       1 node.go:253] "NodeUnstageVolume: called" args={"volume_id":"vol-0f00d7ebcc1c0993f","staging_target_path":"/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"}
I1009 18:49:39.828876       1 node.go:293] "NodeUnstageVolume: unmounting" target="/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"
I1009 18:49:39.828894       1 mount_helper_common.go:93] unmounting "/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I1009 18:49:39.828908       1 mount_linux.go:360] Unmounting /mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f
I1009 18:49:39.855136       1 mount_helper_common.go:150] Warning: deleting path "/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"
I1009 18:49:39.855207       1 node.go:298] "NodeUnStageVolume: successfully unstaged volume" volumeID="vol-0f00d7ebcc1c0993f" target="/mnt/ecs/ebs/mocktaskID_vol-0f00d7ebcc1c0993f"
I1009 18:49:39.855226       1 node.go:268] "NodeUnStageVolume: volume operation finished" volumeID="vol-0f00d7ebcc1c0993f"

New tests cover the changes: Yes

Description for the changelog

• Collect volume metrics for EBS-backed tasks and clean up EBS-backed tasks

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[amogh09]

Can we please fill out the "Testing" section in the PR description.