Skip to content

Home

Jump to bottom
Baoying Wang edited this page Jan 23, 2018 · 55 revisions

Introduction

It is an OrderBook(Matching Engine). It supports

  • LIMIT and MARKET order.
  • generate execution reports and market data
  • Web interface, FIX interface and TCP interface(for better performance)
  • the highlight is performance v1.1 performance - 5000 order/second (bg50000.100_lt1.1 via TCP) - see https://github.com/baoyingwang/OrderBook/wiki/Track-Performance-Tuning-for-v1.1 a) match:4.4us 99% (2.2us 95%) b) e2e: 1.5ms 99% (1.1 ms 95%) v1 performance - 5000 order/second(bg50000.10_lt1.1 via FIX - persist OFF) a) match:2.5us 95% b) e2e: 20ms 95%

Architecture - v2.0 - all bytes (on plan)

  • receive order bytes from sockets (use NIO - netty with poll/select/etc, or maybe vertx since it depends on netty)
  • direct memory(off-heap) is used for orders(fixed size on each field + version for easy upgrade)
  • order book will link to the direct memory with index
  • update direct memory for order status change, e.g. filled, etc
  • clean the order after order completed(filled/rejected/...)

Pro

  1. almost no GC, since off-heap memory operation
  2. save the time for thread context switch - single thread(or 2 threads) to process the orders.
  3. minimize message marshal(to bytes)/unmarshal(to object).

Cos

  1. hard code since all bytes operation. It can be mitigated to wrap the byte operation in a single class
  2. NIO code is more difficult. It can be mitigated to use netty library
  3. Hard to support FIX. We have to introduce our own protocol to define.

Architecture - v1.1 - add an interface with pure TCP without FIX store/IO overhead(ongoing)

The route of an order to matching engine TCP(vertx): read bytes from socket -> Q -> vertx event loop thread for matching FIX : read bytes from socket -> Q -> QFJ FIX message dispatcher -> Q (disruptor ring buffer-v1.0 or blocking Q of vertx event loop v1.1) -> disruptor thread-v1.0 or vertx event loop-v1.1 for matching

see : https://github.com/baoyingwang/OrderBook/wiki/Track-Performance-Tuning-for-v1.1 It is a wrong direction to reduce the Q inside. Because 1.the goal should be reduce e2e. The e2e is about 14 ms. But the Q will only cause a few us(2050). We should focus on the main part of the time cost.

ME_1.1_architecture_20180118.png

Architecture - v1.0 (released)

ME_1.0_architecture_20180107.png

Why not also use Disruptor on the output Q?

  • is it required so fast for the output Q, since order has already executed?
  • guava async event bus is rather easy to use. The code is clean.

Current Performance - v1.0 - Jan 7, 2018

  • note: rate per second during test 5000 orders per second
  • note: java ArrayBlockingQueue is used as input Q. Another option is LMAX Disruptor, which is also supported.
  • note: it is tested on my ubuntu(a VM of win7 64bit). 2 cpu are assigned to this vm. Disruptor_BusySpinWaitStrategy_bg5000perSec_lt60perMin_duration600sec_20180107_102049_latency_overall.png

For more images and test data see https://github.com/baoyingwang/OrderBook/wiki/Track-Performance-Tuning - section: Jan 7, 2018 - add e2e measurement (release v1.0 is defined since today)

Critical Aspects

performance

  1. v1.1 achieve much more better performance with TCP (rather than FIX).

But we have more things to do

  1. explain the output of v1.1 test output - useful_result_BaoyingOrderBookFat-v1.1_2018-01-20_085716.686.zip. How does the java process / sockets number impact the performance?
  2. Any possible to be better? any idea?
  3. keep in mind, to achieve guarantee delivery, the performance maybe downgrade.

non-functional requirements - Guarantee Delivery, HA(single point failure), Error Handling,

  1. Guarantee Delivery - how to send those skipped Execution Reports to the clients after they re-login?
  • a straight forward solution is to mark the delivery status in a in memory table. For any not-marked the ER in table(if any) will be sent to the related clients, with PossibleDup=Y. We have to consider 1) will it downgrade performance much? 2) how to work with HA.
  • note: if a messages is marked with FIX seq number, the message will be ensured delivery by FIX session(only when no-session-reset). But maybe the ERs do not trigger FIX seq change(can we trigger that somehow?).
  • note: the current v1.1 TCP does not support the seq number(FIX resend), yet.
  1. HA(single point failure) - how to quickly restore?
  • how?
  • e.g. share inbound messages with file: all inbound messages are written to a shared file(e.g. by ChronicleQ + SAN/NAS). The standby instance will read the file and re-build orderbook(and other state if any)
  • e.g. share the orderbook with file
  • ?
  1. Error Handling
  • how to identify and throttling the input, if the input rate is higher than the system load
  • how to failover(HA)? identify those manual operations if any, e.g. run some scripts to sync with clients(via email?).

functional requirements - OMS(order management service), MDS(market data service), and Credit Check Service, etc

How to use

after clone, run below to build a package. Find the package at home/build/distributions, e.g. BaoyingOrderBookFat-v1.1_2018-01-20_204648.673.zip

gradle buildPackage

unzip the package, and start the it

cd scripts
bash bash startMatchingEngine.sh

e.g.
$ bash startMatchingEngine.sh
Matching Engine APP_HOME:/c/baoying.wang/ws/gitnas/OrderBook/build/distributions/BaoyingOrderBookFat-v1.1_2018-01-21_215934.902
starting matching engine for symbols:USDJPY
jar file /c/baoying.wang/ws/gitnas/OrderBook/build/distributions/BaoyingOrderBookFat-v1.1_2018-01-21_215934.902/jars/BaoyingOrderBookFat.jar

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::  (vv1.1_2018-01-21_215934.902)

Populate OrderBook

jarfile=BaoyingOrderBookFat-2017-12-24_101139.396-all.jar

java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Bid -qty 50000000 -ordType Limit -px 112 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Bid -qty 50000000 -ordType Limit -px 113 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Bid -qty 50000000 -ordType Limit -px 114 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Bid -qty 50000000 -ordType Limit -px 115 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Bid -qty 50000000 -ordType Limit -px 116 -d 5
 

java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Offer -qty 50000000 -ordType Limit -px 122 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Offer -qty 50000000 -ordType Limit -px 123 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Offer -qty 50000000 -ordType Limit -px 124 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Offer -qty 50000000 -ordType Limit -px 125 -d 5
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 10 -client_prefix BACKGROUD_FIX_prepare -symbol USDJPY -side Offer -qty 50000000 -ordType Limit -px 126 -d 5

Send massive orders as background

e.g. -d 600 seconds , 10 clients, 1000 orders per minute to Buy
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 10 -ratePerMinute 1000 -client_prefix BACKGROUD_FIX_bid  -symbol USDJPY -side Bid   -qty 5 -ordType Market -d 600

e.g. -d 600 seconds , 10 clients, 1000 orders per minute to Sell
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 10 -ratePerMinute 1000 -client_prefix BACKGROUD_FIX_offer -symbol USDJPY -side Offer -qty 5 -ordType Market -d 600

Send those deals for latency test

java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 1 -client_prefix 'LTC$$_FIX'  -symbol USDJPY -side Bid   -qty 5 -ordType Market -d 600
java -cp ${jarfile} baoying.orderbook.testtool.FirstQFJClientBatch -clientNum 1 -ratePerMinute 1 -client_prefix 'LTC$$_FIX'  -symbol USDJPY -side Offer -qty 5 -ordType Market -d 600

Check the order book(empty if you don't place order), and place order(not yet supported) http://localhost:8080/main.html

Check latency test summary (empty if you don't place order) http://localhost:8080/test_summary.html

Load JMeter to send test deal, and check above links see the JMeter file - OrderBook\src\test\resources\jmeter\OrderBook.jmx

Clone this wiki locally