feat: Add testing contribution guideline #229
Conversation
Isakdl
force-pushed
the
testing-guideline
branch
from
274501c to
0a9d1a2
Compare
|
|
||
| By adhering to the following principles, we ensure that our test suite remains a valuable and reliable asset as our codebase grows. | ||
|
|
||
| This document outlines Serverpod's approach to testing code. It serves as a guide for writing effective, maintainable, and meaningful tests across all our projects. |
There was a problem hiding this comment.
Suggestion: Move this paragraph to before the ## Overview heading
| - The outcome of a test must never depend on any other test running before or after it. | ||
| - The order in which tests are executed **should not matter.** | ||
| - Running a single test in isolation must produce the same result as running it alongside others. | ||
| - **Exception to the rule:** e2e and integration tests. In scenarios where an external state (like a shared database) is involved, tests may require concurrency mode 1 to prevent interference. But each test should start and end in a clean state. |
There was a problem hiding this comment.
There are two kinds of independence that can be worth while distinguishing:
- technical independence (reentrancy), which may not be possible with external resources such as DB, in which case non-concurrent execution is required.
- semantic independence, which should only be broken when the tests specifically test a sequence of operations (scenario tests / test scripts). This test scripts need to be "atomic" - they can't be subdivided or change internal order of execution, but should still be independent of each other.
Suggestion: Move the last "exception" point to a separate elaboration, tests ought to be written to be independent even if there are technical dependencies (currently... might be improved!)
| ### 1. Clear and Descriptive Test Descriptions | ||
|
|
||
| - **Test descriptions should be understandable without reading the test code.** | ||
| - If a test fails, the description alone should make it clear what went wrong. |
There was a problem hiding this comment.
Suggestion: Add behavior: "make it clear what behavior went wrong"
|
|
||
| - **Unit tests should avoid mocking and side effects.** | ||
| - Production code should push side effects **up the call stack**, allowing tests to cover pure methods. | ||
| - **Test the logical feature/unit** rather than a single method or class. |
There was a problem hiding this comment.
Suggestion: Add "Prefer" to the last point:
- Prefer to test the logical feature/unit rather than each single method or class.
| ### 3. Pure Unit Testing | ||
|
|
||
| - **Unit tests should avoid mocking and side effects.** | ||
| - Production code should push side effects **up the call stack**, allowing tests to cover pure methods. |
There was a problem hiding this comment.
Suggestion: Add definition of "pure" to this section, since this is for a public audience and not everyone knows exactly what it is, e.g.
Pure functions
Pure functions are simply put functions that lack side effects, lack external I/O, and always have the same results for the same inputs. See https://en.wikipedia.org/wiki/Pure_function
There was a problem hiding this comment.
We might want to rephrase this completely, maybe focus more on the non-side effect part. Since Unit tests have different definitions as well. Or we should explain what we view as a Unit test.
There was a problem hiding this comment.
Maybe it would be good to break them apart even? One that outlines the test abstraction level and one that focuses on side effects? This might already be covered in point 4. Implementation-agnostic tests.
| ### 4. Implementation-Agnostic Tests | ||
|
|
||
| - **Do not couple tests to implementation details.** | ||
| - Tests should only break if the behavior changes, not when refactoring code. |
There was a problem hiding this comment.
Suggestion: Since this is sometimes referred to as black box testing, add a point to spell it out:
- This may be referred to as black box testing.
|
|
||
| Consistent application of these principles leads to a robust and maintainable codebase, fostering confidence in our product's reliability and scalability. | ||
|
|
||
| We acknowledge that there are exceptions to the rules. But when deviating from the guidelines mentioned above you have to argue for why. You must have a reasonable argument that aligns with the core goals. |
There was a problem hiding this comment.
Nit: Remove double space between "there are"
There was a problem hiding this comment.
Nice writeup! I agree with all of it and only have some minor comments and suggestions around structure and phrasing.
I'm not sure we should actually add this to our documentation thoe. Maybe it would make more sense to place this in the actual project and keep the simple Roadmap and Constributions here?
Then we could also have information on how to run tests there as well?
I'm not sure it makes sense to have information specific to "contributors" outside of the project? What do you think?
| @@ -5,6 +5,7 @@ Serverpod is built by the community for the community. Pull requests are very mu | |||
| <div style={{ position : 'relative', paddingBottom : '56.25%', height : '0' }}><iframe style={{ position : 'absolute', top : '0', left : '0', width : '100%', height : '100%' }} width="560" height="315" src="https://www.youtube-nocookie.com/embed/V3CqPx4jykE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div> | |||
|
|
|||
| ## Roadmap | |||
There was a problem hiding this comment.
Suggestion: A redirect needs to be implemented to catch any links to the old contribution guidelines.
Also - do we reference this from the Serverpod repo? If so, the link there needs to be updated as well.
There was a problem hiding this comment.
We also recently flattened out this structure (creating the Roadmap & contributions menu item). Are we sure we want to nest it again?
|
|
||
| ## Key Principles | ||
|
|
||
| ### 0. Test Independence |
There was a problem hiding this comment.
Question/Discussion: In written text, the first item is often 1. Any particular reason for having 0 as the first rule here?
There was a problem hiding this comment.
What do we do in the rest of the documentation?
| **Example:** | ||
|
|
||
| ```dart | ||
| // Good - Tests are split |
There was a problem hiding this comment.
Suggestion: Use the same Don't and Do pattern used when explaining best practises for our testing framework to ensure consistency in the documentation.
| ### 3. Pure Unit Testing | ||
|
|
||
| - **Unit tests should avoid mocking and side effects.** | ||
| - Production code should push side effects **up the call stack**, allowing tests to cover pure methods. |
There was a problem hiding this comment.
We might want to rephrase this completely, maybe focus more on the non-side effect part. Since Unit tests have different definitions as well. Or we should explain what we view as a Unit test.
| **Example:** | ||
|
|
||
| ```dart | ||
| // Avoid mocking HTTP requests directly, test the logic that processes the response |
There was a problem hiding this comment.
Suggestion: add a do and don't to this section as well.
|
|
||
| This document outlines Serverpod's approach to testing code. It serves as a guide for writing effective, maintainable, and meaningful tests across all our projects. | ||
|
|
||
| ## Key Principles |
There was a problem hiding this comment.
Suggestion: Use capital letter only on the first word.
This needs to be applied to all headers.
| }); | ||
| ``` | ||
|
|
||
| ### 7. Beneficial Abstractions - Test Builders |
There was a problem hiding this comment.
Suggestion/Discussion: Should we mention factories here as well? When the built objects are dependent on each other?
|
|
||
| Consistent application of these principles leads to a robust and maintainable codebase, fostering confidence in our product's reliability and scalability. | ||
|
|
||
| We acknowledge that there are exceptions to the rules. But when deviating from the guidelines mentioned above you have to argue for why. You must have a reasonable argument that aligns with the core goals. |
There was a problem hiding this comment.
Suggestion: Change to: We acknowledge that there are exceptions to the rules. However, any deviation from the guidelines outlined above must be supported by a well-reasoned argument that aligns with the core objectives.
|
|
||
| - **Test descriptions should be understandable without reading the test code.** | ||
| - If a test fails, the description alone should make it clear what went wrong. | ||
| - **Format:** Descriptions follow the "Given, When, Then" style. |
There was a problem hiding this comment.
Suggestion/Discussion: Should we outline what the different words mean?
| ### 3. Pure Unit Testing | ||
|
|
||
| - **Unit tests should avoid mocking and side effects.** | ||
| - Production code should push side effects **up the call stack**, allowing tests to cover pure methods. |
There was a problem hiding this comment.
Maybe it would be good to break them apart even? One that outlines the test abstraction level and one that focuses on side effects? This might already be covered in point 4. Implementation-agnostic tests.
Changes
Adds contribution guideline for how we write tests at Serverpod including examples and explanation of our best pracitices.