[RFC]: Benchmarking support for new client transports

[finnegancarroll]

What are you proposing?

Add support in OSB to benchmark alternative client/server transport protocols. This feature should make it easy to plug in a new alternative client implementation and evaluate it's performance in contrast to the traditional REST OpenSearch python client.

What users have asked for this feature?

The traditional REST API stands out as an opportunity for optimization within OpenSearch and several issues within core propose replacements which aim to cut down on serialization or provide data in more SIMD friendly formats.

Flight/Arrow RFCs:

• [RFC] Streaming Aggregation - A Memory-Efficient Approach OpenSearch#16774
• [META] Search streams using Apache Arrow and Flight  OpenSearch#16679

gRPC/Protobuf RFCs:

• [META] GRPC Ecosystem Integration OpenSearch#16786
• [META] gRPC Server OpenSearch#16556

What problems are you trying to solve?

It is difficult to evaluate the performance of new client/server transports, often requiring custom scripts, tools, or hacky OSB feature branches. To benchmark the new protocol and API.

What will the developer experience be like?

The complexity of completely abstracting all OSB client operations to provide support for new client/server transports is large and would include a substantial amount of wasted work considering new client/server transports may not support more than a few specialized operations for which they are optimized.

For this feature I would propose new OpenSearch client implementations extend the existing OSClient, lightly wrapping the default REST client and cherry picking key operations of the OSClient interface to intercept and send with the new transport protocol.

class RpcClient(OsClient): 
    def __init__(self, restClient, altClient):
        super().__init__(restClient)
        self._altClient = altClient

    def search(self, index, body):
        return self.rpc_guarded(self._altClient.search, index=index, body=body)

    def rpc_guarded(self, target, *args, **kwargs):
        # handle rpc errors/retries

Adding new client transports to OSB:
Any new OpenSearch python client should mirror opensearchpy and be made available to OSB as a package.

Enabling and selecting additional transport clients:
Configuration options for new clients may be provided alongside all other client options in the configuration file. The OsClientFactory will be responsible for validating and constructing "hybrid" clients as configured by the user and enabled by feature flag.

Are there any security considerations?

Security related configuration options, such as TLS, should be configured on a per-client basis through flags/configuration file as they are for the REST client.

What will the user experience be like?

Executing OSB using an alternative client should be possible by adding some additional flags (or same options set in config file):

opensearch-benchmark execute-test \
    --pipeline=benchmark-only \
    --workload=big5 \
    --target-host=http://localhost:9200 \
    --alt-client=experimental-grpc-transport \
    --alt-host=http://localhost:9400 \
   ...

Are there breaking changes to user experience?

These changes should only be additive and have no impact on the existing OSB configuration options.

Why should it be built? Any reason not to?

It may not be appropriate to require OSB directly manage alternative client implementations. As this issue describes OSB may be further abstracted to support generalized APIs which would likely involve a more general client abstraction to support non OpenSearch APIs. It's possible adding this feature now could be wasted work in light of these proposals.

What will it take to execute?

While the changes to OSB will likely be limited, at time of opening this issue there is no mature client with which to test this change. Framework components for adding/managing new clients can be added but the usefulness of this feature is predicated on a mature client utilizing a new transport protocol.

Any remaining open questions?

No response