Socket protocol compatibility testing

[eswdd]

Been thinking about this for a couple of years now, but only really started thinking about how to solve the last week or so.

Currently we test protocol compatibility via unit tests that test every combination of protocol version against each other. The problem is that this is done using the latest code, so it's possible that changes elsewhere that the code protocol could affect protocol compatibility - we're not testing the code from the old version of Cougar.

What I propose is we build an additional all in one jar which contains a small commandline app which starts a socket transport, wires in a fake EV with a v simple Executable that provides echo capability for an RPC call that works and an RPC call that doesn't (and maybe later some v simple push capability). The jar also includes metadata on what versions of the protocol it supports. When started, it searches for a port, starts the binary protocol server on that port, echos the port to the console and waits until killed, processing any requests as they come in.

The jar would also have a client mode which expects certain data to come back from those calls based on the protocol version (i.e. all the data supported by that version).

We then build a test tool which can search a maven repo (local and central ideally) for all the test apps and then runs the full matrix of all of them. This ensures we properly test full compatibility. We should run at least as part of every release.

We could then simplify the unit tests by testing backwards compatibility with the previous version only, which would stop the already spiralling EV client test time getting out of control.

[bfmike]

Not everyone upgrades to every new version, so it would be good to run the tests with the current release and all of the released versions within the current major version rather than just the previous version. Major versions should probably pick and document which previous versions are still supported and test against all of those.

[eswdd]

Socket protocol versioning != cougar versioning. So for now at least and
the forseeable future, every version of the socket protocol should be able
to talk to every other version, across all cougar versions that supported
that version as it's highest capability. This is a large matrix, but agree
it should be tested every release.. It just is too big for unit testing.

[atropineal]

i had a poke around on the web and didn't really find any doco on how other
people approach this sort of testing. this sounds pretty decent

so you'd have a set of socket-protocol-compatibility-tests-x.y.z jars, and

for server in jars do
for client in jars do
java -jar $server -mode server -port 8080 &
echo $! > server.pid
# client quits when tests are complete
java -jar $client -mode client -server localhost -port 8080
-client_version $(version_of $client) -server_version $(version_of $server)
2>> error_detail.txt > report.txt
kill $(cat server.pid)
done
done

report.txt:

| client | server | result (failures) |
| x.y.z | a.b.c | PASS |
| a.b.c | x.y.z | FAIL (echo) |

errors.txt

+++ TEST client=a.b.c, server=x.y.z, operation=echo
com.betfair.cougar.client.WhateverException: "wibble"
at .....

you could run the script from a compatibility testing maven module, and
that module could depend on all the versions of clients and servers, then
the script could enumerate them all from the local repo. running something
more portable than bash might be preferable

anyway, liked it enough to think about it and comment :-)

[eswdd]

Yep, pretty much what I was thinking. Java seems pretty portable, boring that it is..

Not sure whether it will happen now. Might just do it for the sake of it.