RFC: Comparison of Bybit HTTP vs WebSocket order placement speed

[sunlei]

Background

Based on the discussion in #2129, I conducted tests comparing HTTP and WebSocket order placement performance on Bybit.
Under identical code and parameter configurations, I collected two key metrics on both Server A and Server B:

• elapsed_ns_order_init_to_submitted: OrderInitialized -> OrderSubmitted
• elapsed_ns_order_init_to_accepted: OrderInitialized -> OrderAccepted

Instrument ID

ETHUSDT-LINEAR.BYBIT

Configuration

HTTP-ONLY

• use_ws_trade_api=False
• use_http_batch_api=True

WS-ONLY

• use_ws_trade_api=True
• use_http_batch_api=False

Time Range (UTC)

2024-12-26 11:00:00 ~ 2024-12-26 13:00:00

During this period, there's been some slight volatility in the market.

Test Results

elapsed_ns_order_init_to_submitted

elapsed_ns_order_init_to_accepted

At a certain point, HTTP order placement experienced a sudden and prolonged latency increase.
After restarting the strategy, the latency returned to normal, though the exact cause is still unclear.
Excluding that time window, the comparison results between the two methods are as follows:

elapsed_ns_order_init_to_submitted

elapsed_ns_order_init_to_accepted

Conclusions

• WebSocket offers a huge speed advantage over HTTP, and more stable.
• Data from node_exporter shows that when using HTTP-only, TCP_inuse fluctuates more significantly compared to WS-only, likely due to connection recycling mechanisms.
• In the current code, event.ts_event uses the timestamp from the WebSocket message under the order topic, specifically the updatedTime field.
  • Additionally, In my other test, it shows Bybit’s WebSocket messages createdTime and updatedTime differ by about 3–4 ms.
• Even when both servers use WS-only, Server A and Server B still show some discrepancies, likely due to factors such as “noisy neighbors,” differences in matching engines behind different accounts, and physical distance from the exchange servers. However, these variations are less pronounced than the discrepancy observed with HTTP.

Question

• It is currently impossible to measure the exact duration from “order initialization” to the moment the request is actually sent, because generate_order_submitted is the final step executed just before sending the request.
• In my implementation, there is almost no additional logic between self.order_factory and self.submit_order, so elapsed_ns_order_init_to_submitted effectively reflects the internal processing overhead of nautilus_trader from order creation to just before sending the request.
  • Is there any way to optimize the elapsed time here?

Code used for measurement

def on_order_event(self, event: OrderEvent):
    client_order_id = event.client_order_id

    order: Order = self.cache.order(client_order_id)
    if not order:
        return

    elapsed_ns: int = event.ts_event - order.ts_init

    metric_name = None
    if isinstance(event, OrderSubmitted):
        self.count_order_submitted_total += 1
        self.count_order_submitted_buy += 1 if order.is_buy else 0
        self.count_order_submitted_sell += 1 if order.is_sell else 0
        metric_name = "elapsed_ns_order_init_to_submitted"
    elif isinstance(event, OrderAccepted):
        self.count_order_accepted_total += 1
        self.count_order_accepted_buy += 1 if order.is_buy else 0
        self.count_order_accepted_sell += 1 if order.is_sell else 0
        metric_name = "elapsed_ns_order_init_to_accepted"
    elif isinstance(event, OrderFilled):
        self.count_order_filled_total += 1
        self.count_order_filled_buy += 1 if order.is_buy else 0
        self.count_order_filled_sell += 1 if order.is_sell else 0
        metric_name = "elapsed_ns_order_init_to_filled"
    elif isinstance(event, OrderCanceled):
        self.count_order_canceled_total += 1
        self.count_order_canceled_buy += 1 if order.is_buy else 0
        self.count_order_canceled_sell += 1 if order.is_sell else 0
        metric_name = "elapsed_ns_order_init_to_canceled"
    elif isinstance(event, OrderRejected):
        self.count_order_rejected_total += 1
        self.count_order_rejected_buy += 1 if order.is_buy else 0
        self.count_order_rejected_sell += 1 if order.is_sell else 0
        metric_name = "elapsed_ns_order_init_to_rejected"

    if metric_name:
        self.log.info(
            f"log_type=metrics client_order_id={client_order_id} "
            f"{metric_name}={elapsed_ns} "
            # submitted
            f"count_order_submitted_total={self.count_order_submitted_total} "
            f"count_order_submitted_buy={self.count_order_submitted_buy} "
            f"count_order_submitted_sell={self.count_order_submitted_sell} "
            # accepted
            f"count_order_accepted_total={self.count_order_accepted_total} "
            f"count_order_accepted_buy={self.count_order_accepted_buy} "
            f"count_order_accepted_sell={self.count_order_accepted_sell} "
            # filled
            f"count_order_filled_total={self.count_order_filled_total} "
            f"count_order_filled_buy={self.count_order_filled_buy} "
            f"count_order_filled_sell={self.count_order_filled_sell} "
            # canceled
            f"count_order_canceled_total={self.count_order_canceled_total} "
            f"count_order_canceled_buy={self.count_order_canceled_buy} "
            f"count_order_canceled_sell={self.count_order_canceled_sell} "
            # rejected
            f"count_order_rejected_total={self.count_order_rejected_total} "
            f"count_order_rejected_buy={self.count_order_rejected_buy} "
            f"count_order_rejected_sell={self.count_order_rejected_sell} "
        )

@cjdsellers @filipmacek