From 553c8ada7fe8754a75f2079842e828241ce355a8 Mon Sep 17 00:00:00 2001 From: Tim Collins <45351296+tico24@users.noreply.github.com> Date: Thu, 13 Feb 2025 16:01:02 +0000 Subject: [PATCH] docs: clarify qps/burst on controller (#14190) Signed-off-by: Tim Collins --- .spelling | 1 + docs/scaling.md | 39 ++++++++++++++++++++++++++++++--------- 2 files changed, 31 insertions(+), 9 deletions(-) diff --git a/.spelling b/.spelling index 661c7e52d415..c730cbedea96 100644 --- a/.spelling +++ b/.spelling @@ -204,6 +204,7 @@ parameterizing params pprof pre-commit +qps rc2 repo roadmap diff --git a/docs/scaling.md b/docs/scaling.md index 0b3d4130c089..822e859898c6 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -5,10 +5,7 @@ For running large workflows, you'll typically need to scale the controller to ma ## Horizontally Scaling You cannot horizontally scale the controller. - -> v3.0 and after - -As of v3.0, the controller supports having a hot-standby for [High Availability](high-availability.md#workflow-controller). +The controller supports having a hot-standby for [High Availability](high-availability.md#workflow-controller). ## Vertically Scaling @@ -32,7 +29,8 @@ If you have sufficient CPU cores, you can take advantage of them with more gorou ### K8S API Client Side Rate Limiting -The K8S client library rate limits the messages that can go out. +The Kubernetes client library used by the Workflow Controller rate limits the number of API requests that can be sent to the Kubernetes API server. +This rate limiting helps prevent overwhelming the API server with too many requests at once. If you frequently see messages similar to this in the Controller log (issued by the library): @@ -46,9 +44,34 @@ Or, in >= v3.5, if you see warnings similar to this (could be any CR, not just ` Waited for 7.090296384s, request:GET:https://10.100.0.1:443/apis/argoproj.io/v1alpha1/namespaces/argo/workflowtemplates/s2t ``` -Then, if your K8S API Server can handle more requests: +These messages indicate that the Controller is being throttled by the client-side rate limiting. + +#### Adjusting Rate Limiting + +By using cluster-wide observability tooling, you can determine whether or not your Kubernetes API server can handle more requests. +You can increase the rate limits by adjusting the `--qps` and `--burst` arguments for the Controller: + +- `--qps`: This argument sets the average number of queries per second allowed by the Kubernetes client. +The default value is 20. +- `--burst`: This argument sets the number of queries per second the client can send before it starts enforcing the qps limit. +The default value is 30. Typically, burst should be greater than qps. + +By increasing these values, you can allow the Controller to send more requests to the API server, reducing the likelihood of throttling. + +##### Example Configuration -- Increase both `--qps` and `--burst` arguments for the Controller. The `qps` value indicates the average number of queries per second allowed by the K8S Client. The `burst` value is the number of queries/sec the Client receives before it starts enforcing `qps`, so typically `burst` > `qps`. If not set, the default values are `qps=20` and `burst=30` (as of v3.5 (refer to `cmd/workflow-controller/main.go` in case the values change)). +To increase the rate limits, you might set the arguments as follows: + +```yaml +args: + - --qps=50 + - --burst=75 +``` + +This configuration allows the Controller to send an average of 50 queries per second, with a burst capacity of 75 queries per second before throttling is enforced. + +It is important to note that increasing these values can increase the load on the Kubernetes API server and that you must observe your Kubernetes API under load in order to determine whether or not the values you have chosen are correct for your needs. +It is not possible to provide a one-size-fits-all recommendation for these values. ## Sharding @@ -73,8 +96,6 @@ data: instanceID: i1 ``` -> v2.9 and after - You may need to pass the instance ID to the CLI: ```bash