Support gradual ramp up of clients

[rishabh6788]

Is your feature request related to a problem? Please describe

Currently, after a warmup period, all query tasks are executed together at the same time. After the first execution, they respect their configured scheduler for subsequent executions.
In cases with a high number of tasks (e.g. executing using a parallel block) this absence of a ramp up, can produce an initial extremely high load and result in excessive utilization.
Due to this it becomes very difficult to do red-line testing of OpenSearch cluster and reliably tell what is breaking point of a particular cluster.

Describe the solution you'd like

The objective of this feature is to support gradual ramp of clients to allow users to ramp-up load gradually by specifying the task property ramp-up-time-period. If a non-zero value is specified, OSB will gradually add clients during that time period until the target client count as specified by clients is reached. This reduces the potential for initial load spikes when running with many concurrent clients.

Describe alternatives you've considered

No response

Additional context

No response

[bowenlan-amzn]

Document a discussion with @rishabh6788 on this feature:

Assume the goal of users is to find the max qps, it would be more straightforward to provide a max_qps input param. The output user get is the max qps number this ramp feature find out in the end. So we don't require user to set client number and target throughput but OSB can default or even dynamically update these behind the scene at runtime.

Also another similar idea to expose qps concept and hide client param is to combine these 2 parameters — client number and target throughput into one — qps_step.

OSB steps up the qps until the cluster starts to reject, then it may be able to step down half step, step up 1/4 step, until it find an accurate max_qps.