Project Pipelines Organization

[theo-s]

Moving from our email thread

This is summery of current state:

• Correctness tests run in corresponding pipelines – R tests in R pipeline and Python tests in Python pipeline.
  • R pipeline is CI pipeline for now, it only tests changes, but once we add step to expose built R package as pipeline artifact, it will become full continuous delivery pipeline.
  • Python pipeline is CD pipeline as it exposes artifacts (wheels and sdist) after testing them.
• If developer changes anything in /src (C++ code), that will trigger both R and Python pipeline. (This triggers C++ pipeline too, but it just lints the code for now. C++ unit test should tun there).
• If developer changes R code (/R/*), that will trigger R pipeline.
• If developer changes Python code (/Python/*), that will trigger Python pipeline.

So, what we don’t have now is point 3 from Theo’s email – output comparing, and the question is where/when to do it.

• One of the options is to do it as part of R/Python pipelines immediately after correctness step in each pipeline, before exposing/uploading artifacts in pipelines, allowing pipeline failure as soon as possible.
• Another option is to have separate pipeline which can be triggered either manually, or by certain trigger, and which would do this test. This would make sense if test takes a long time (like 30+ minutes), and we don’t want to trigger it all the time.

[edwardwliu]

Good summary over email Boban. I also like the ideal testing approach that Theo outlined. Specifically, our tests should have clear goals in coverage.

• R testthat, Python pytest, and C++ unit tests[1] should be sufficient for covering algorithmic correctness and integration testing.
• Cross-language tests should verify consistent behavior across package wrappers.

One benefit to only checking that outputs match is that we don't have to manage canonical results, which quickly become painful to maintain. Let's move forward with the following decisions:

1. Kick off the cross-language tests automatically as part of the R/Python pipelines. For now, the tests should run quickly. We can revisit this as needed, but pipelines that are not triggered by automatically tend to quickly fall out of date.
2. Verify consistency in outputs, rather than comparing against a separate canonical output. Output can be temp files so there isn't additional data for these tests.
3. Let's use the bash script approach outlined over email.

I'll assume we're aligned, but please feel free to interject if not. @petrovicboban could you take the first pass at setting up the cross-language script? Regarding the actual test content, both the Python and R tests use the external Iris dataset. We can test the output of just a single predict call across Python and R (see links). You can DM directly if you run into any blockers.

Regarding priority, this would make testing the Python package easier, but does not explicitly block release. We should ensure that other direct blockers are resolved first.

[1] Boban suggested we separate Kanban boards into specific milestones. One future milestone I had in mind was landing the speed-up of training in C++. The first step of which could be writing unit tests for the training code then refactoring safely with the coverage from unit tests.

[petrovicboban]

Discussion topic on work organization and kanban boards is here #71