OQS sub projects: Which ones to drop for good

[baentsch]

This is to initiate a discussion on OQS sub projects: There are many that do not receive any attention in terms of issue-creation, -resolution or other activity. We have declared some as "inactive" to no great(ly visible) disagreement.

Proposal:

• Designate 3 sub project tiers: Fully supported (at least 2 fully committed maintainers), best effort (maybe a part-time maintainer), inactive (none of us cares).
• Assign all projects to these tiers
• Only discuss/track fully supported projects going forward, e.g., in https://www.openquantumsafe.org/dashboard.html
• Integrate (cross-project) CI only for fully supported projects, incl. support of the same platforms & binaries/distro-packages
• Get input from PQCA members as to which projects they want (& fund)

Strawman proposal: Only liboqs, oqsprovider and possibly liboqs-python receive "fully supported" status.

Input/feedback/alternative suggestions welcome.

[baentsch]

Without feedback after a week and in line with our lazy consensus principle I consider there is no objection to change all sub projects' summary view (except the ones above, i.e., those with a dedicated maintainer) to read similar to libssh:

THIS PROJECT IS PRESENTLY NOT FULLY SUPPORTED. CONTRIBUTORS AND MAINTAINER(S) WANTED.

Such statement sets proper expectations by users hitting problems: They simply may get no response, let alone a fix. Most notably this will be applied to oqs-demos and profiling for which I personally currently have no propensity to look after any more.

This way we may be able to catch the attention of the community -- and make it clear that the takeover by Linux Foundation does not mean an automatic improvement in terms of actual people working on the project.

Personally, I'm afraid the contrary is true: With the establishment of PQCA "the world" may have been convinced that OQS is now fully supported and well-staffed by the corporate sponsors and thus cease to have an interest to actively contribute, possibly even expect reported issues to be resolved (more) quickly.

Explicitly tagging @dstebila as our "man in the GB" should this TSC "decision" (in the absence of other statements :-) not otherwise get flagged in his Inbox.

[christianpaquin]

I agree with the proposed 3 tiers, I think all 3 should get a different "message" (i.e., don't mark "best effort" and "inactive" with the same message, and we want to run some "best effort" cross-CI testing to "best effort" projects (could be prioritized based on available resources). Agreed that we need to urgently reflect the expected level of support in the project's documentation.

[dstebila]

Here's the PQCA's working document for their project lifecycle: https://docs.google.com/document/d/1NV-0vNgXWdc81oqT0jv0C-9Funb8dySS06u90ghF-X4/edit

[baentsch]

Here's the PQCA's working document for their project lifecycle: https://docs.google.com/document/d/1NV-0vNgXWdc81oqT0jv0C-9Funb8dySS06u90ghF-X4/edit

What concrete impact does this document have on this issue? Is there any timeline for this document to be completed (seems inactive)? Should completion of this document stop this issue from proceeding?

My concrete proposal:

• Suggest to @vsoftco to create GOVERNANCE.md files modeled after those of oqs-provider & liboqs for those projects he's maintaining.
• Drop all sub projects without GOVERNANCE.md files (and hence maintainers) from the list of actively maintained OQS sub projects, e.g., on the Web site (as already done for ssh&openssl-111) as well as CI, incl. dashboard.

[dstebila]

Here is a table with my first draft attempt at classifying all our subprojects/repositories against the classification Michael listed at the top of this issue (Fully supported / Best effort / Inactive) and the PQCA project lifecycle (Experimentation / Labs / Incubation / Growth / Impact / Emeritus). My classification here is really a first gut reaction based on where I think we have the most activity and momentum. I wasn't sure about some of the language wrappers, such as .NET/Go/Java, for example -- just a gut feeling that they get less attention than C++/Python/Rust, but that could be mistaken.

I don't think either system is perfect. In terms of helping us make operational decisions (e.g., how to prioritize CI) I think the simpler classification is useful.

Please contribute your own thoughts, both about the classification systems and on any particular subprojects.

Subproject	Comment	Classification as in tsc/issue/2	Classification per PQCA lifecycle
boringssl	Periodically updated	Best effort
ci-containers	Support repository, not sub-project	Fully supported	N/A
liboqs		Fully supported	Impact stage
liboqs-cpp		Best effort? Fully supported?	Growth stage?
liboqs-dotnet		Best effort?	Incubation stage?
liboqs-go		Best effort?	Incubation stage?
liboqs-java		Best effort? Inactive?	??
liboqs-python		Best effort? Fully supported?	Growth stage?
liboqs-rust		Best effort? Fully supported?	Growth stage?
libssh		Inactive	Emeritus
openssh	Interest in reviving this?	Inactive -> Best effort?	Emeritus -? Incubation stage?
openssl	1.1.1	Inactive	Emeritus
oqs-demos		Best effort	Experimentation stage?
oqs-provider		Fully supported	Impact stage
profiling	Intended to be revived	Inactive	Experimentation stage

[baentsch]

Thanks for taking a stab at this summary, @dstebila.

I completely agree with columns 1-3 with one request for change: The term "Fully supported" should be changed to "Fully github supported" for oqs-provider in the third column as the former may imply support on all channels PQCA has chosen to create (email lists, DM tooling, ...) but that I cannot follow and hence, cannot promise to provide support on. Tagging @thb-sb for commenting on this: If you'd follow and interact with the PQCA comms channels (and keep acting as oqs-provider maintainer) (?), it may be OK to leave the entry as-is.

Regarding the forth column, I have to absolutely and adamantly protest the labelling of oqs-provider as (LF/PQCA) "Impact Stage": That project is NOT receiving financial nor any other support from PQCA (rather the opposite if I compare how OQS operates before and after the LF take-over: Contributions and procedures have surely become more difficult and cumbersome). Most notably, oqs-provider (well, I guess myself as the 95% contributor/maintainer of it) is accordingly not willing to "to cross promote the foundation along with their activities" as per the demand placed on such project according to the PQCA project classification --> please change over to "Incubation stage" if you need to apply this classification -- that I personally consider pretty much completely unsuitable to OQS:

To justify the latter statement, see these two pretty much contradictory statements:

Document that it is being used successfully in production by at least two independent end users which, in the TAC’s judgment, are of adequate quality and scope.
(from PQCA project classification)

WE DO NOT CURRENTLY RECOMMEND RELYING ON THIS LIBRARY IN A PRODUCTION ENVIRONMENT OR TO PROTECT ANY SENSITIVE DATA
(from liboqs usage advice).

and one unresolved issue #1 to document that labelling liboqs "Impact Stage" as being completely unjustifiable; according to the above, liboqs isn't even "Growth Stage". Now, if we were to correct this, not a single OQS sub project would be anywhere else than "Incubation stage" -- pretty much leading the whole PQCA classification ad absurdum -- which is why I'd suggest dropping that column outright (also automatically resolving my initial comments regarding the PQCA classification of oqs-provider :-).

Here's btw my practical, non-PQCA-process driven suggestion as to how to handle oqs-demos now: open-quantum-safe/oqs-demos#275

[christianpaquin]

WRT OpenSSH, I discussed with my AWS peers with whom we do interop testing in the NCCoE project, but there doesn't seem to have bandwidth to take maintenance ownership (@brian-jarvis-aws can confirm)

[brian-jarvis-aws]

WRT OpenSSH, I discussed with my AWS peers with whom we do interop testing in the NCCoE project, but there doesn't seem to have bandwidth to take maintenance ownership (@brian-jarvis-aws can confirm)

That’s correct regarding maintenance, however I do still plan to allocate time for one of my team members to bring the project up to date with the latest upstream OpenSSH version for continued interop testing.

[baentsch]

@dstebila Here's another voice arguing that liboqs should be rated in "an early lifecycle stage": @hartm writes in open-quantum-safe/liboqs#1832 (comment):

API changes are better the earlier they are in a project's lifecycle. As the project grows, it becomes harder and harder to make these kinds of changes. So if it's something that's needed, it's better to do now rather than later.

I fully support this view -- as long as we then also agree that liboqs cannot be rated at LinuxFoundation rating level "Impact" or even "Growth" (regardless of whether or not the project is 8 years old :-)

My rationale for that "low" rating was that OQS does not have a "healthy" contributor or maintainer base (using the term from the LF/PQCA lifecycle document they want to apply to OQS) -- but API stability obviously is another matter in that consideration.

[baentsch]

One more criterion helping to define to a sub project's maturity level: Security incident handling.

[baentsch]

To re-activate this issue, here's a straw man proposal for the oqsprovider sub project: We don't have to drop it, but everyone shall see (and communicate) it exactly (and "only") as

A test & experimentation vehicle for the widest possible range of PQC algorithms, algorithm combinations and use cases. It shall never cater for productive use cases.

Most notably, no-one in PQCA and beyond shall assume or represent this sub-project to be anything else than "Research code" or "Incubation stage" code. Top three reasons:

1. It has so many bells and whistles catering for all kinds of configurations that it's very hard to understand and maintain by now and thus absolutely not suitable for "real use". The good thing is, it does not have to be given the much cleaner code landing now in openssl.

2. For a whole year within PQCA, no-one tried to address issue 1 above or took up the baton of even defining what it would need to provide "reliability", let alone contribute work towards that end --short of asking to simply drop the usage warning. The good thing is, addressing these issues is not really essential for "test-and-prototyping-only" code.

3. It has always been an algorithm-agnostic wrapper around liboqs functionality -- and there's calls to change that and adapt specific algorithms to specific use cases and draft specifications which would make the code ever more "weird" or "cumbersome" -- and those are terms that should arguably not be associated with software handling sensitive data "for real" (ie, and eg. running when your grandma is doing her e-banking :).

Anyone disagreeing with this goal and argumentation, please speak up.

[dstebila]

In interest of reviving some of this discussion ahead of next week's TSC meeting, here is an update to #2 (comment) to reflect what I think is the current status of our sub-projects, according to the criteria in the top post of this thread ("Fully supported (at least 2 fully committed maintainers), best effort (maybe a part-time maintainer), inactive (none of us cares).") plus two new tiers: deprecated/archived, and unclassified

• Fully supported: liboqs, oqs-provider
• Best effort: boringssl, liboqs-cpp, liboqs-go, liboqs-python, oqs-demos
• Inactive: liboqs-java, liboqs-rust, openssh, profiling
• Deprecated/archived: liboqs-dotnet, libssh, openssl-111, oqs-engine
• Unclassified (supporting repositories): .github, ci-containers, liboqs-cupqc, tsc, www

This time I haven't tried classify these against the PQCA tiers since that didn't go well last time and we haven't been asked to do it at this point.

Some questions:

• Are these tiers still meaningful for (a) us in our decision making, and (b) in terms of communicating to the outside world?
• Is the classification of our sub-projects against these tiers accurate?
• Do we want (and are there willing contributors) to move a sub-project from one tier to another?