Testing conventions

[heitorlessa]

Summary

Testing definitions that we will use throughout the project. We can update this with examples for each test category later (to get the ball rolling quickly).

Proposal

I'd like to propose we group tests in 5 categories:

Type	Scope	Notes	Speed
Unit	Verify the smallest possible unit works locally without any dependency or network.	Self-contained and won't span multiple parts of the code.	Lightning fast (nsec to ms)
Functional (subset of integration)	Verify domain logic works as expected for a customer operation.	In order, prefer fake/test doubles, SDK Stubs, Local emulators.	Fast (ms to few seconds at worst)
Integration	Verify one or more external integrations.	Use local code against an external dependency e.g., read-after write into DynamoDB, fetch config from SSM parameter.	Moderate to slow (a few minutes)
End-to-end	Ensure a customer operation works from their point of view.	Test as if you had no prior knowledge of its inner workings (black box) by using API/UI exposed to customers.	Slow (minutes)
Performance	Ensure increased load within our capacity won't degrade customers experience.	CI arbitrary hardware can make it flaky.	Fast to slow (a few seconds to a few minutes)

Why

It's common for every company and project to have variations of their tests. This can create friction during review and can limit our ability to enforce certain design practices (e.g., fail Unit tests if a TCP/Unix socket is opened).

Potential challenges

We need some guidance on how much we test CDK custom constructs - do we add them under unit? or functional, or create something entirely new?

Out of scope

N/A

[having-fun-serverless]

@heitorlessa I feel like unit and Functional are under the same umbrella in real life examples, how many times do you have a pure functional code that does not require fake or test doubles. I just think it adds to the complexity to the summary.

In addition I think it's interesting to add to the summary a reference to where should you invest your time, for example integration tests have better ROI than unit testing or functional testing. Check this for reference -

• https://theburningmonk.com/2022/05/my-testing-strategy-for-serverless-applications/
• https://www.youtube.com/watch?v=p3M1Y2Oi_Yk&t=441s
• https://medium.com/hackernoon/serverless-testing-from-the-trenches-790e77301c74 - something I wrote on the subject a couple of years ago

[ran-isenberg]

You can add my testing blog series:post

[ran-isenberg]

Functional test.can be around specific handler classes (db handler, external service handler.class etc.) or logic adapter classes.
It really depends because in many cases itay easier to test them through the integration tests and call the handler in IDE with a generated event.

[ran-isenberg]

@heitorlessa I'd add that in integration tests, we test from the handler entry point with a sample event to simulate a business use case/input