Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add e2e tests #273

Merged
merged 31 commits into from
Mar 6, 2024
Merged

feat: Add e2e tests #273

merged 31 commits into from
Mar 6, 2024

Conversation

Milouu
Copy link
Contributor

@Milouu Milouu commented Oct 20, 2023 •
edited
Loading

Description

Adds a first batch of automated tests for the frontend to minimize the manual testing needed on it when releasing a new version of Substra.

Important: Needs a companion PR on substra-ci to change the default test to frontend[mnist] instead of sdk

Closes FL-520

@Milouu Milouu force-pushed the feat/e2e-tests branch 2 times, most recently from 90826ae to 432d10f Compare October 20, 2023 15:20
@linear
Copy link

linear bot commented Oct 20, 2023

FL-520 Write front-end tests v1

Context and user need:
Release test plan front-end tests are currently manual. They

Functional spec:
They should be added to the automated cypress tests of the frontend (which are part of the CI), this way they could be run in a single command or as part of the e2e tests or whatnot.

Start with a very clear and narrow scope, like the General or User test, that are part of the https://app.asana.com/0/project-templates/1202906784456224/list.

Acceptance criteria:
A single command is enough to run the frontend test plan, or it is run without human intervention.

@Milouu
Copy link
Contributor Author

Milouu commented Oct 23, 2023

/e2e -h

@Owlfred
Copy link

Owlfred commented Oct 23, 2023 

Usage: /e2e [options] [help]

/e2e may appear anywhere as long as it is on its own line

Options:
  --refs <value>                          Extra refs (branch or tag) with format REPO=GIT_REF,REPO=GIT_REF.
  Supported repositories: orchestrator, substra-backend, substra-frontend, substra-tools, substrafl, substra, substra-tests, substra-documentation, substra-ci.
  Example: /e2e --refs substra-backend=some_branch,orchestrator=some_tag (default: {})
  --tests-to-run, --tests <tests-to-run>  Comma-separated list of tests to run. Valid options: sdk, substrafl, doc, frontend, mnist, camelyon or NONE. (default: "sdk")
  -h, --help                              display help for command

@Milouu
Copy link
Contributor Author

Milouu commented Oct 30, 2023

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Oct 30, 2023 •
edited
Loading

End to end tests: ❌ FAILURE

What a surprise... 🙄

@Milouu
Copy link
Contributor Author

Milouu commented Oct 30, 2023

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Oct 30, 2023 •
edited
Loading

End to end tests: ✔️ SUCCESS

@Milouu Milouu marked this pull request as ready for review October 30, 2023 14:20
@Milouu Milouu requested review from HamdyD, jmorel and a team as code owners October 30, 2023 14:20
@thbcmlowk
Copy link
Contributor

thbcmlowk commented Oct 30, 2023 •
edited
Loading

Great job! 🥳

Please wait for the companion PR to be created on substra-ci 🙏
Merging as is will break the nightly for sure as the default test dependency for frontend is still SDK

oleobal
oleobal reviewed Oct 30, 2023
View reviewed changes
Copy link
Contributor

@oleobal oleobal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

niiiiiiiiice

@thbcmlowk
Copy link
Contributor

End to end tests: ✔️ SUCCESS

⚠️

E2E frontend tests appear to be successful but they are not. See build logs:

Traceback (most recent call last):
  File "/home/runner/work/substra-ci/substra-ci/ci/tests.py", line 236, in _run_or_pass
    duration = test_runner(cfg, *test_runner_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/substra-ci/substra-ci/ci/tests.py", line 98, in run_frontend
    raise FrontendTestsException("Frontend tests failed")
ci.tests.FrontendTestsException: Frontend tests failed

There should be an issue with the implementation suggested in substra-ci:refactor/frontend-tests that will be investigated, but there is still something wrong with the frontend tests.

@Milouu
Copy link
Contributor Author

Milouu commented Jan 4, 2024

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Jan 4, 2024 •
edited
Loading

End to end tests: ❌ FAILURE

Jobs status:

What a surprise... 🙄

@Milouu
Copy link
Contributor Author

Milouu commented Jan 5, 2024

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Jan 5, 2024 •
edited
Loading

End to end tests: ❌ FAILURE

Jobs status:

Better luck next time!

@Milouu
Copy link
Contributor Author

Milouu commented Jan 5, 2024

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Jan 5, 2024 •
edited
Loading

End to end tests: ⚙️ RUNNING

“Patience et longueur de temps. Font plus que force ni que rage.” ― Jean de La Fontaine (Le Lion et le Rat)

@Milouu
Copy link
Contributor Author

Milouu commented Jan 8, 2024

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Jan 8, 2024 •
edited
Loading

End to end tests: ✔️ SUCCESS

@Milouu
Copy link
Contributor Author

Milouu commented Jan 15, 2024

/e2e --tests frontend[mnist] --refs substra-ci=refactor/frontend-tests

@Owlfred
Copy link

Owlfred commented Jan 15, 2024 •
edited
Loading

End to end tests: ❌ FAILURE

Jobs status:

“It’s just a flesh wound.” ― The Black Knight, Monty Python and the Holy Grail

Milouu and others added 25 commits March 6, 2024 15:31
@Milouu
add cypress commands
a8797d2 
Signed-off-by: Milouu <[email protected]>
@Milouu
add functions tests
4fb3333 
Signed-off-by: Milouu <[email protected]>
@Milouu
add users page test to menu tests
467b39c 
Signed-off-by: Milouu <[email protected]>
@Milouu
add cps tests
0e0f7ad 
Signed-off-by: Milouu <[email protected]>
@Milouu
add cp details tests
935e0d0 
Signed-off-by: Milouu <[email protected]>
@Milouu
add datasets tests
e312837 
Signed-off-by: Milouu <[email protected]>
@Milouu
add tasks tests
21cc0ae 
Signed-off-by: Milouu <[email protected]>
@Milouu
add support function
7d8fc69 
Signed-off-by: Milouu <[email protected]>
@Milouu
update changelog
b5fa1bc 
Signed-off-by: Milouu <[email protected]>
@Milouu
Add comments to commands file
755e2ba 
Signed-off-by: Milouu <[email protected]>
@Milouu
Increase retries for CI testing
406123e 
Signed-off-by: Milouu <[email protected]>
@Milouu
Fix misc tests
103cef6 
Signed-off-by: Milouu <[email protected]>
@Milouu
Add permissions conditions to download function test
d10b6cb 
Signed-off-by: Milouu <[email protected]>
@Milouu
intercept test
3a017a8 
Signed-off-by: Milouu <[email protected]>
@Milouu
Clean PR
9df69c9 
Signed-off-by: Milouu <[email protected]>
@Milouu
remove intercept
c4073b2 
Signed-off-by: Milouu <[email protected]>
@Milouu
skip download function test
33966f7 
Signed-off-by: Milouu <[email protected]>
@Milouu
Remove skipped test
bd73757 
Signed-off-by: Milouu <[email protected]>
@Milouu
add favorite test in cp details
b153a87 
Signed-off-by: Milouu <[email protected]>
@Milouu
Add performance test in cp details
8d2996d 
Signed-off-by: Milouu <[email protected]>
@Milouu
Add searchbykey test for functions & datasets
cdfb41b 
Signed-off-by: Milouu <[email protected]>
@Milouu
Add CRUD tests for users page
4e4312a 
Signed-off-by: Milouu <[email protected]>
@thbcmlowk @Milouu
style(src): run prettier
fa6655b 
Signed-off-by: Thibault Camalon <[email protected]>
@Milouu
fix: Wait before checking change on user role
d649948 
Signed-off-by: Milouu <[email protected]>
@Milouu
fix: replace wait by built-in retry process
88ad9f2 
Signed-off-by: Milouu <[email protected]>
@Milouu
Copy link
Contributor Author

Milouu commented Mar 6, 2024

/e2e --tests mnist,camelyon,substrafl,frontend --refs substra-ci=refactor/frontend-tests-on-all-tests

@Owlfred
Copy link

Owlfred commented Mar 6, 2024 •
edited
Loading

End to end tests: ✔️ SUCCESS

@Milouu Milouu merged commit 4c8834e into main Mar 6, 2024
3 checks passed
@Milouu Milouu deleted the feat/e2e-tests branch March 6, 2024 15:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@oleobal oleobal oleobal left review comments

@HamdyD HamdyD HamdyD approved these changes

@jmorel jmorel Awaiting requested review from jmorel jmorel is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Milouu @Owlfred @thbcmlowk @guilhem-barthes @ThibaultFy @oleobal @HamdyD