Establish interop with Circl and other implementations

[baentsch]

Circl implements several algorithms also provided by liboqs. An automated interop test set would be good to avoid issues like #909 in the future. Possibly an application for liboqs-go?

[thomwiggers]

Although more testing is generally better, both liboqs and circl already do KAT tests. They also do development at different paces....

[baentsch]

They also do development at different paces

And also seem to have different goals (integrations of more/different PQ algs into different apps, say). But 4 months difference (judging for example for Kyber from cloudflare/circl@6168cdb) is a bit steep, indeed. Good we now have more upstream-pull automation.

--> Would anyone be in favour of setting a date for releasing liboqs 0.5? Would also be good to finally upgrade the pretty dated https://test.openquantumsafe.org to r3 (servers)...

[thomwiggers]

I've yolo-opened a discussion on release schedules here #912

[claucece]

I'm currently updating to the latest changes in some pq algorithms at CIRLC. It could be good to test between the two projects, and I can provide an status of them.

[baentsch]

@claucece Thanks for that update and your interest to look into this issue. In case you didn't find them yet, here's pointers to some interop test scripts we already have distributed across the various projects and that may provide thoughts:

• openssl/boringssl interop CI test
• script driving curl docker image against nginx test server
• script driving openssl3 provider image against nginx test server

Upon glancing at them, the latter two are particularly badly "documented" (my fault), so please don't hesitate to ask questions if you'd want to build off this if you also want run tests against the server. If you'd want to do something that runs as part of CI, I'd recommend the first link.

[xvzcf]

Having looked into checking all KATs for each KEM (let alone signatures) I see two issues:

• Time (as suggested in the meeting): python3 tests/test_kat.py went from taking 43.70 s to taking 461.13 s
• KAT hashes: copy_from_upstream can update the KAT hashes, which are taken over just the first KAT value. Modifying this to update and cover all 100 KAT values will take some doing

One way to move forward would be to test interop on the one KAT set, and then run a few more tests on randomly sampled inputs

[dstebila]

One way to move forward would be to test interop on the one KAT set, and then run a few more tests on randomly sampled inputs

Sounds like a good compromise.

[dstebila]

@baentsch Can we close this due as having been satisfactorily resolved via open-quantum-safe/oqs-provider#278?

[baentsch]

OK for me if the only truly supported PQ feature CIRCL has is two specific TLS KEMs (which open-quantum-safe/oqs-provider#278 ascertains interop of). My original impression was that it is much broader in scope, both in terms of algorithms as well as protocols supported.

[dstebila]

OK for me if the only truly supported PQ feature CIRCL has is two specific TLS KEMs (which open-quantum-safe/oqs-provider#278 ascertains interop of). My original impression was that it is much broader in scope, both in terms of algorithms as well as protocols supported.

You're right, there's also Dilithium and FrodoKEM in Circle. Nevermind.

[SWilson4]

@open-quantum-safe/liboqs-committers Given the discussion around external interoperability testing in this week's meeting, perhaps it's worth adding this to an upcoming release milestone?

[dstebila]

@baentsch As discussed in today's OQS status call, how much of TLS hybrid interop (with Google servers, Cloudflare servers, ...?) is being done already in oqs-provider?

[baentsch]

@baentsch As discussed in today's OQS status call, how much of TLS hybrid interop (with Google servers, Cloudflare servers, ...?) is being done already in oqs-provider?

Not an awful lot, see for yourself: https://github.com/open-quantum-safe/oqs-provider/blob/main/scripts/oqsprovider-externalinterop.sh

Re-reading the above, this issue originally called for interop tests across all supported algorithms, not just specific hybrid TLS1.3 KEM(s). Unless we re-define the goal, this is clearly not in scope for 0.12.0 but would require a serious effort. Also, I'd add BouncyCastle as an interop target as it seems to rival if not eclipe OQS in PQC "completeness".

[dstebila]

@baentsch As discussed in today's OQS status call, how much of TLS hybrid interop (with Google servers, Cloudflare servers, ...?) is being done already in oqs-provider?

Not an awful lot, see for yourself: https://github.com/open-quantum-safe/oqs-provider/blob/main/scripts/oqsprovider-externalinterop.sh

Yes, fairly barebones. Thanks for looking.

Re-reading the above, this issue originally called for interop tests across all supported algorithms, not just specific hybrid TLS1.3 KEM(s). Unless we re-define the goal, this is clearly not in scope for 0.12.0 but would require a serious effort. Also, I'd add BouncyCastle as an interop target as it seems to rival if not eclipe OQS in PQC "completeness".

Agreed. I think it's fair to say that the oqs-provider TLS interop doesn't cover the full scope of this goal. And I also agree that it's bigger in scope than we can do in 0.12.0, so I've retagged it for 0.13.0.