Fix tests, update workflows, support Python 3.12 tests

[schuenke]

• set rf.dead_time of refocussing pulse in write_epi_se (was causing an error after merge of Add warning to make_block_pulse() if rf.delay is changed to rf.dead_time #236). Sorry, my fault!
• cleaned push_pr workflow:
  • limit repository access to minimum
  • added Python 3.12 to matrix
  • included flag for sigpy tests in matrix (allows to run everything in parallel)
  • removed unnecessary use of conda env in tests
  • added dependency caching for speed up
  • added concurrency to save resources and avoid unnecessary duplication
  • added coverage report
  • ensure test job fails if Failures or Errors occur

All in all, this hopefully simplifies the testing process, removes unnecessary complexity, and is generally better structured.

[github-actions]

Tests	Skipped	Failures	Errors	Time
1185	18 💤	0 ❌	0 🔥	2m 45s ⏱️

[github-actions]

Tests	Skipped	Failures	Errors	Time
1165	18 💤	0 ❌	0 🔥	2m 7s ⏱️

[FrankZijlstra]

The "slow" tests cannot run on their own, at least how it's currently set up, because it needs to load the sequence into the TestSequence object first with test_sequence, which is not marked as a "slow" test. A solution would be to change the class such that each test can create the sequence if it does not exist yet, or maybe we can mark test_sequence in a way that it always runs? (i.e. without -m, with -m "slow" and with -m "not slow"?).

[schuenke]

The "slow" tests cannot run on their own, at least how it's currently set up, because it needs to load the sequence into the TestSequence object first with test_sequence, which is not marked as a "slow" test. A solution would be to change the class such that each test can create the sequence if it does not exist yet, or maybe we can mark test_sequence in a way that it always runs? (i.e. without -m, with -m "slow" and with -m "not slow"?).

Yeah, realized that as well. I introduced a "required_for_slow" marker and then use pytest -m "slow or required_for_slow" in the sequence tests. That seems to work. Will push this in a few minutes after some final tests.

[schuenke]

okay, in the 3rd try on my local PC the slow / plot tests failed as well because of some random tkinter error. 4th run was okay, again. However, in the GitHub workflow it seems buggy. Takes ages. IMO we should just remove the current plot tests (and the slow marker) completely. It doesn't really tests anything anyway, does it?

Instead I would introduce some dedicated tests for the plot() functionality at some point.

Would you be fine with that @FrankZijlstra ? If not, I will just leave it as it was before and simply don't run the tests.

[schuenke]

Would be great if we can merge this soon, as we need it for #238 and #240
@FrankZijlstra: I think you alread had a look at the code, no? Maybe you can review the latest changes and approve it if it's fine for you this way?!

[FrankZijlstra]

Removing the plot tests for now is fine with me. Before adding the new Sequence tests, that was one of the only tests that existed, because @wtclarke found some bug with the plotting at the time, I think. It essentially just checks that calling plot raises no errors. But I agree that with automated tests in a headless environment, plotting could be difficult to get right...

[wtclarke]

Removing the plot tests for now is fine with me. Before adding the new Sequence tests, that was one of the only tests that existed, because @wtclarke found some bug with the plotting at the time, I think. It essentially just checks that calling plot raises no errors. But I agree that with automated tests in a headless environment, plotting could be difficult to get right...

Why remove them? Can you not just patch the show method as I originally did? @patch("matplotlib.pyplot.show"). Should then be fine in the headless environment. You definitely want to test plotting code as it's something that users are likely to use and people are likely to fiddle with.

[FrankZijlstra]

Why remove them? Can you not just patch the show method as I originally did? @patch("matplotlib.pyplot.show"). Should then be fine in the headless environment. You definitely want to test plotting code as it's something that users are likely to use and people are likely to fiddle with.

The plot tests have show patched (

pypulseq/tests/test_sequence.py

Line 278 in 4a25069

with patch('matplotlib.pyplot.show'):

). Apparently this was not enough to avoid @schuenke's error in his local environment.

The issue with plotting larger sequences (i.e. the sequence examples) is that it is very slow, so not ideal for automated tests on github.

[schuenke]

Removing the plot tests for now is fine with me. Before adding the new Sequence tests, that was one of the only tests that existed, because @wtclarke found some bug with the plotting at the time, I think. It essentially just checks that calling plot raises no errors. But I agree that with automated tests in a headless environment, plotting could be difficult to get right...

Why remove them? Can you not just patch the show method as I originally did? @patch("matplotlib.pyplot.show"). Should then be fine in the headless environment. You definitely want to test plotting code as it's something that users are likely to use and people are likely to fiddle with.

I think we all agree that testing the plot function makes sense, but running an extremely slow seq.plot() for all example and test sequences without checking the output does not make sense IMO. Some decent tests for the plot function itself should be more useful.

[wtclarke]

Surely just test on a small sequence then? I'd suggest adding a single block and try plotting that sequence. It's cool to see the tests over multiple sequence types, but don't use that as an excuse to not test a frequently used feature. As above, this is code that people will feel the need to tweak and users will use.

[schuenke]

Surely just test on a small sequence then? I'd suggest adding a single block and try plotting that sequence. It's cool to see the tests over multiple sequence types, but don't use that as an excuse to not test a frequently used feature. As above, this is code that people will feel the need to tweak and users will use.

I re-introduced and extended the seq.plot() tests for the basic test sequences (seq1, seq2, seq3, seq4). IMO this is still not a meaningful test, because we dont check the outcome at all, but now we know that the method does not raise any errors. Fine for you @wtclarke & @FrankZijlstra ?

[wtclarke]

Thanks for doing this. Yeah, I understand your point, but I would say execution without error is still a meaningful test for something that is otherwise hard to test.

[FrankZijlstra]

Yes, looks good to me. Minor nitpick is to put the time_range a bit higher, plotting the first millisecond is probably only ever going to plot the first block of the sequence.