--target-host flag doesn't seem to correctly benchmark multiple host: timout error

[TitoReinhart-Ampere]

Please provide details on how and where the documentation can be improved

Target host flag would be helpful to show example of comma delimitted list. Thus far passing multiple hosts on different ports gets timeout error despite cluster heath being green for N-nodes.

Documentation:
--target-hosts TARGET_HOSTS, -t TARGET_HOSTS
Define a comma-separated list of host:port pairs which
only' (default: localhost:9200).

Based on the error log it looks like the python dictionary is being parsed correctly but the python3.9 site-packages can't execute GET. This is weird cause when I do it manually it works fine, and when this test is run with the same configuration on each node individually, test runs fine.

What am I missing?

Command: --target-hosts=<ip_addr>:9200,<ip_addr>:9201

Error from benchmark.log:

2025-01-15 18:18:03,215 -not-actor-/PID:20238 osbenchmark.benchmark INFO Python [namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=18, releaselevel='final', serial=0), hexversion=50926320, _multiarch='x86_64-linux-gnu')]
2025-01-15 18:18:03,215 -not-actor-/PID:20238 osbenchmark.benchmark INFO Benchmark version [1.11.0]
2025-01-15 18:18:03,215 -not-actor-/PID:20238 osbenchmark.utils.net INFO Connecting directly to the Internet (no proxy support).
2025-01-15 18:18:03,396 -not-actor-/PID:20238 osbenchmark.benchmark INFO Detected a working Internet connection.
2025-01-15 18:18:03,440 -not-actor-/PID:20238 osbenchmark.benchmark INFO Actor system already running locally? [False]
2025-01-15 18:18:03,440 -not-actor-/PID:20238 osbenchmark.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2025-01-15 18:18:03,773 -not-actor-/PID:20277 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2025-01-15 18:18:03,783 -not-actor-/PID:20238 osbenchmark.test_execution_orchestrator INFO Test Execution id [b252dc86-c459-4b8f-90b1-1900a12045a7]
2025-01-15 18:18:03,783 -not-actor-/PID:20238 osbenchmark.test_execution_orchestrator INFO User specified pipeline [benchmark-only].
2025-01-15 18:18:03,783 -not-actor-/PID:20238 osbenchmark.test_execution_orchestrator INFO Using configured hosts [{'host': '<ip_addr_replaced>', 'port': 9200}, {'host': '<ip_addr_replaced>', 'port': 9201}]
2025-01-15 18:18:03,784 -not-actor-/PID:20238 osbenchmark.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2025-01-15 18:18:03,785 ActorAddr-(T|:1900)/PID:20277 osbenchmark.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 9, 18, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1736965083766'}] match requirements [{'coordinator': True}].
2025-01-15 18:18:13,828 -not-actor-/PID:20279 opensearch WARNING GET http://<ip_addr_replaced>:9200/_cluster/health?wait_for_nodes=%3E%3D2 [status:N/A request:10.011s]
Traceback (most recent call last):
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 468, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 463, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.9/http/client.py", line 1377, in getresponse
response.begin()
File "/usr/lib64/python3.9/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.9/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/opensearch/lib64/python3.9/site-packages/opensearchpy/connection/http_urllib3.py", line 280, in perform_request
response = self.pool.urlopen(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 802, in urlopen
retries = retries.increment(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/util/retry.py", line 527, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 470, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='ip_addr_replaced>', port=9200): Read timed out. (read timeout=10)

Traceback (most recent call last):
File "/root/opensearch/lib64/python3.9/site-packages/osbenchmark/benchmark.py", line 1024, in dispatch_sub_command
execute_test(cfg, args.kill_running_processes)
File "/root/opensearch/lib64/python3.9/site-packages/osbenchmark/benchmark.py", line 746, in execute_test
with_actor_system(test_execution_orchestrator.run, cfg)
File "/root/opensearch/lib64/python3.9/site-packages/osbenchmark/benchmark.py", line 780, in with_actor_system
time.sleep(3)

Traceback (most recent call last):
File "/root/opensearch/lib64/python3.9/site-packages/opensearchpy/connection/http_urllib3.py", line 280, in perform_request
response = self.pool.urlopen(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 802, in urlopen
retries = retries.increment(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/util/retry.py", line 527, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/packages/six.py", line 770, in reraise
raise value
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 716, in urlopen
httplib_response = self._make_request(
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 470, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/root/opensearch/lib64/python3.9/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='<IP_ADDR_REPLACED>', port=9201): Read timed out. (read timeout=10)

Provide references

No response

Additional context

No response

[IanHoang]

Thanks for bring this up @TitoReinhart-Ampere. Most users seem to only be using a single endpoint but we will aim to reproduce this.