Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: BeforeFeature / AfterFeature hooks #219

Closed
vitalets opened this issue Sep 12, 2024 · 4 comments
Closed

Feature: BeforeFeature / AfterFeature hooks #219

vitalets opened this issue Sep 12, 2024 · 4 comments
Labels
enhancement New feature or request

Comments

@vitalets
Copy link
Owner

vitalets commented Sep 12, 2024

The problem
Sometimes we need to run code before / after particular feature file. Currently it can be achieved in playwright-bdd only with some hacks.
In CucumberJS it's not possible - there were several related requests:

CucumberJS FAQ states that these hooks are not possible, because runner does not consider features when running scenarios.

A solution
Playwright provides a direct way to run code before particular file - test.beforeAll / test.afterAll. We can just utilize that.
SpecFlow also provides BeforeFeature / AfterFeature hooks.

Binding to features can be done by tags - the same as for scenario hooks. Example code:

const { BeforeFeature, AfterFeature } = createBdd();

BeforeFeature({ tags: '@auth' }, async () => { 
  // run setup before features with @auth tag 
});

This also matches the proposed tags from path approach.
You can put feature files and BeforeFeature hooks into @-prefixed directory, and they will be bound automatically:

features
└── @auth
    ├── signin.feature
    └── hooks.ts

Hooks naming
Hooks naming between Cucumber and Playwright is a bit confusing:

  • Playwright's BeforeEach / AfterEach = Cucumber's Before / After - run before/after each scenario
  • Playwright's BeforeAll / AfterAll = (no alternative in CucumberJS) - run before/after each feature
  • Playwright's worker-scoped auto-fixtures = Cucumber's BeforeAll / AfterAll - run before/after each worker

I like the explicit naming in SpecFlow:

  • BeforeScenario / AfterScenario
  • BeforeFeature / AfterFeature
  • BeforeWorker / AfterWorker

I think, we can use these aliases in playwright-bdd.

Running once
There is a request from community to run beforeFeature hooks only once, even in fullyParallel mode / in case when some tests are failing and new worker is created.
Keep in mind during implementation, see microsoft/playwright#22520 (comment)

I you have any thoughts / concerns here, feel free to share in comments.
Or just add your 👍

@vitalets vitalets added the enhancement New feature or request label Sep 12, 2024
@vitalets vitalets pinned this issue Sep 12, 2024
@vitalets vitalets changed the title Feature: Introduce BeforeFeature / AfterFeature hooks Feature: BeforeFeature / AfterFeature hooks Sep 12, 2024
@vitalets
Copy link
Owner Author

vitalets commented Oct 9, 2024

What should be a default behavior of BeforeFeature / AfterFeature hooks?

  1. Run once in each worker: this is how Playwright's test.beforeAll / test.afterAll currently behave. If some test fails, new worker is created and hooks re-run again.
  2. Run once in a whole test run: this is not how Playwright's test.beforeAll / test.afterAll behave, but according to playwright#22520 it is often requested (to setup some database stuff, etc).

What do you think?

@alescinskis
Copy link

I vote for 1st option

@vitalets
Copy link
Owner Author

vitalets commented Oct 16, 2024

What should be a default behavior of BeforeFeature / AfterFeature hooks?

I thought more about that question. If BeforeFeature runs in each worker, it's actually not BeforeFeature but BeforeWorker. A confusion also goes from naming: while feature is related to BDD level, worker is related to runner. Mixing these entities in hooks naming does not make it clear.

As of now, I'm thinking about the following changes:

  1. Alias existing Before / After hooks as BeforeScenario / AfterScenario and recommend using the latter, as they are more explicit
  2. Alias existing BeforeAll / AfterAll as BeforeWorker / AfterWorker and recommend using the latter, as they are more explicit
  3. Allow to pass tags option to BeforeWorker / AfterWorker. If tags are passed, the hook runs only if matched features are being executed (so actually behaves like BeforeFeature / AfterFeature). If a new worker is created for matched feature, the hook runs again. If tags are not passed, the hook runs before / after each worker (current behavior).
  4. Allow to pass once option to BeforeWorker / AfterWorker. If once: true is passed, the hook runs only once during the whole test run session (so actually behave like Playwright's global setup / teardown). If additionally filtered by tags, the hook runs once and only if matched features are being executed - this is the main benefit for setting up some data once for the particular feature.
  5. No BeforeFeature / AfterFeature hooks

Example code:

const { BeforeWorker } = createBdd();

BeforeWorker({ tags: '@auth', once: true }, async () => { 
  // heavy setup of test user in database for checking all auth stuff
});

Feature file can contain @auth tag on the top level:

@auth
Feature: check auth

  Scenario: check login

Or on the scenario level:

Feature: check auth

  @auth
  Scenario: check login

If at least one scenario of feature has the matching tag, BeforeWorker hook will run before that feature.

@vitalets
Copy link
Owner Author

vitalets commented Dec 16, 2024

Playwright-BDD v8 shipped tagging support for BeforeAll / AfterAll. So the following requirement is fulfilled:

Allow to pass tags option to BeforeWorker / AfterWorker. If tags are passed, the hook runs only if matched features are being executed (so actually behaves like BeforeFeature / AfterFeature). If a new worker is created for matched feature, the hook runs again. If tags are not passed, the hook runs before / after each worker (current behavior).

So, BeforeFeature / AfterFeature can be now implemented with tags:

BeforeAll({ tags: '@my-feature' }, async () => { ... });
AfterAll({ tags: '@my-feature' }, async () => { ... });

One note here, that BeforeFeature / AfterFeature can be interpreted in different ways.
If there are two feature files with @my-feature tag, should BeforeAll run before each feature or only once?
The same for parallelization: if a feature is split across several workers, should BeforeAll in each worker or not?

In current implementation, tagged worker hooks will run in each worker - similar to Playwright beforeAll / afterAll hooks.

But in some cases this is not a desirable behavior. If I want to populate database for some feature, I'd like to do it once, no matter how many workers run my tests. This is a goal for mentioned once option, it's not implemented now. There are some ideas, how to achieve that, but it's not an easy task, will be investigated separately.

Closing for now. Feel free to open a new issue if needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants