Add new concurrent querying challenge

[cbuescher]

This change adds a new challenge definition to the elastic/logs track that focusses on higher parallel execution of the querying part. The goal is to provide a better basis for detecting changes in our concurrent execution model in the query phases on average and typical workloads that this track defines.

Main changes over the "querying-logging" track this is based one:

• diabling the query cache using the "disable_query_cache" operation (setting index.requests.cache.enable to false) before querying in order to hit the concurrent search path more often. This is also what the esql parts of this track are doing
• introducing a "worker_threads_enabled" track parameter (true by default) that controls the "search.worker_threads_enabled" node parameter to be able to switch off concurrency for experiments and comparisons

[javanna]

introducing a "worker_threads_enabled" track parameter (true by default) that controls the "search.worker_threads_enabled" node parameter to be able to switch off concurrency for experiments and comparisons

I would rather disable parallelism for the query phase, otherwise you disable offloading to search workers thread entirely, for all phases, and that is a thing that we would do only in case something really really bad happens, to restore the behaviour to before ES even supported parallelism.

[cbuescher]

I'm out for some time but handed this over to @carlosdelest who should be able to take the next steps.

[carlosdelest]

@javanna here is a recent execution of this challenge with the work Christoph has put on this PR.

You can check there are no rejections, and that active search workers are maximized.

Is there anything else we should add to this challenge for checking that inter-segment parallelization behaves correctly?

[javanna]

I am good with the latest results.

[gareth-ellis]

LGTM