Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ratelimiter for fault injection handlers #4340

Merged
merged 7 commits into from
Sep 19, 2024

Conversation

harishxr
Copy link
Contributor

@harishxr harishxr commented Sep 16, 2024

Summary

This PR adds a tollbooth ratelimiter to the Fault Injection handlers, enforcing a request limit to control how frequently fault injection operations can be triggered.

Implementation details

  • createRateLimiter is used to create a new tollbooth rate limiter with rate limit of 0.2 ( 1 request per 5 seconds).
  • The rate limiter is then applied to network fault injection endpoints such as blackhole port faults, latency faults and packet loss faults.

Testing

  • Agent with these changes was built and a standard task was launched using the agent with rate limiter.
  • Accessed the task container and called the Fault Injection APIs back to back
# curl -X GET $ECS_AGENT_URI/fault/v1/network-blackhole-port -d '{"Port":123}' -v
*   Trying 169.254.170.2:80...
* Connected to 169.254.170.2 (169.254.170.2) port 80 (#0)
> GET /api/4e37f57c-c1c8-48ca-927f-0662dfa78094/fault/v1/network-blackhole-port HTTP/1.1
> Host: 169.254.170.2
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 12
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 400 Bad Request
< Content-Type: application/json
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Limit: 40.00
< X-Rate-Limit-Limit: 0.20
< X-Rate-Limit-Request-Forwarded-For: 
< X-Rate-Limit-Request-Forwarded-For: 
< X-Rate-Limit-Request-Remote-Addr: 172.17.0.2:35562
< X-Rate-Limit-Request-Remote-Addr: 172.17.0.2:35562
< Date: Mon, 16 Sep 2024 18:20:41 GMT
< Content-Length: 50
< 
* Connection #0 to host 169.254.170.2 left intact
{"Error":"required parameter Protocol is missing"}
# curl -X GET $ECS_AGENT_URI/fault/v1/network-blackhole-port -d '{"Port":123}' -v
*   Trying 169.254.170.2:80...
* Connected to 169.254.170.2 (169.254.170.2) port 80 (#0)
> GET /api/4e37f57c-c1c8-48ca-927f-0662dfa78094/fault/v1/network-blackhole-port HTTP/1.1
> Host: 169.254.170.2
> User-Agent: curl/7.88.1
> Accept: */*
> Content-Length: 12
> Content-Type: application/x-www-form-urlencoded
> 
< HTTP/1.1 429 Too Many Requests
< Content-Type: text/plain; charset=utf-8
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Duration: 1
< X-Rate-Limit-Limit: 40.00
< X-Rate-Limit-Limit: 0.20
< X-Rate-Limit-Request-Forwarded-For: 
< X-Rate-Limit-Request-Forwarded-For: 
< X-Rate-Limit-Request-Remote-Addr: 172.17.0.2:35568
< X-Rate-Limit-Request-Remote-Addr: 172.17.0.2:35568
< Date: Mon, 16 Sep 2024 18:20:43 GMT
< Content-Length: 38
< 
* Connection #0 to host 169.254.170.2 left intact
You have reached maximum request limit
  • As seen from the test results, the second request done in succession get blocked with a 429 error.
  • New TestRateLimiterIntegration integration test has been added as well to ensure the rate limiter works as expected.

New tests cover the changes: yes

Description for the changelog

Add ratelimiter for fault injection handlers

Additional Information

Does this PR include breaking model changes? If so, Have you added transformation functions?

Does this PR include the addition of new environment variables in the README?

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@harishxr harishxr requested a review from a team as a code owner September 16, 2024 23:42
agent/handlers/task_server_setup.go Outdated Show resolved Hide resolved
agent/handlers/task_server_setup_integ_test.go Outdated Show resolved Hide resolved
@harishxr harishxr force-pushed the fis-ratelimiter branch 4 times, most recently from 4b62271 to 7431dc5 Compare September 18, 2024 20:43
agent/handlers/task_server_setup_integ_test.go Outdated Show resolved Hide resolved
agent/handlers/task_server_setup_integ_test.go Outdated Show resolved Hide resolved
agent/handlers/task_server_setup_integ_test.go Outdated Show resolved Hide resolved
mye956
mye956 previously approved these changes Sep 18, 2024
xxx0624
xxx0624 previously approved these changes Sep 18, 2024
@harishxr harishxr dismissed stale reviews from xxx0624 and mye956 via 56a80f9 September 19, 2024 00:27
@harishxr harishxr merged commit 6cb5505 into aws:dev Sep 19, 2024
40 checks passed
mye956 pushed a commit to mye956/amazon-ecs-agent that referenced this pull request Oct 3, 2024
* Add ratelimiter for fault injection handler

* Add integ tests and refactor code

* Refactor integ test & ratelimiter setup

* Refactor tests to table driven tests

* Add execWrapper to integ tests

* Add dynamic port allocation

---------

Co-authored-by: Harish Senthilkumar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants