e2e testing framework comparison in 2024

The aim of the project to compare the available free end to end testing frameworks in 2024. Currently Cypress, Playwright, Selenium and Nightwatch are planned, but I might add more later. We use a Laravel example CRUD API to test against.

Laravel

Laravel has PhpUnit based feature tests, we can use as baseline. Laravel supports only a single request per test, so we have to use the database directly to add fixtures and check the results. As of the e2e frameworks the tests depend on each other, even the test fiiles depend on each other, which is not the best practice. I use helper functions to get the id of certain users, their token and the id of certain posts to reuse some code.

public function test_deleting_post():void {
	$container = new Container();
	$data = new Data();
	$container->createUser($data->user1);
	$post = $container->createPost($data->post1);
	$this->assertEquals($container->countPosts(), 1);
	$this->withToken($container->createToken())
		->deleteJson('/api/posts/'.$post->id)
		->assertStatus(200);
	$this->assertEquals($container->countPosts(), 0);
}

Cypress

I think the main problem with Cypress that we have to fight against callback hell. With aliases we can try to workaround it, but still it is not the best experience. Another issue that I did not find matchers for JSON comparison for the scenario where the first JSON contains more properties than the second and we need to ignore those properties. These two problems make the test files extremely long though I still managed to write something readable.

it('can read and delete posts', () => {
	cy.getUserWithToken('user1b');
	cy.getPost('post1b');

	cy.get('@post1b')
		.then((post) => cy.request({
			method: 'GET', 
			url: `posts/${post.id}`
		}))
		.then((response) => {
			expect(response.status).to.eq(200);
			expect(response.body.title).to.eq(data.post1b.title);
			expect(response.body.body).to.eq(data.post1b.body);
		});

	cy.aliases(['user1b', 'post1b'])
		.then(([user, post]) => cy.request({
			method: 'DELETE',
			url: `posts/${post.id}`,
			headers: {
				authorization: 'Bearer '+user.token.plainText
			}
		}))
		.then((response) => {
			expect(response.status).to.eq(200);
		});

	cy.get('@post1b')
		.then((post) => cy.request({
			method: 'GET', 
			url: `posts/${post.id}`,
			failOnStatusCode: false
		}))
		.then((response) => {
			expect(response.status).to.eq(404);
		});
});

Playwright

Playwright is like the fresh air after Cypress. It supports async/await extensively. My only problem that I always have to call await response.json() instead of just getting response.body. Another little issue that the body is called data in the requests, which is a weird naming if you ask me, but it is acceptable. It supports partial JSON comparison as well, so I can solve it with a one-liner instead of two or more lines.

test('can read and delete posts', async ({request}) => {
	const user = await getUserWithToken(request, 'user1');
	const post = await getPost(request, 'post1b');

	let response = await request.get(`posts/${post.id}`);
	expect(response.status()).toBe(200);
	let body = await response.json();
	expect(body).toEqual(expect.objectContaining(postsFixture.post1b));

	response = await request.delete(`posts/${post.id}`, {
		headers: {
			authorization: 'Bearer '+user.token.plainText
		}
	});
	expect(response.status()).toBe(200);

	response = await request.get(`posts/${post.id}`);
	expect(response.status()).toBe(404);
});

Selenium

Selenium node does not seem to support API testing, so I skipped it after a few hours of googling. In theory I could add support by a simple form which gathers request data, sends it with fetch API and returns the response. In practice I rather play with other more complete solutions and keep Selenium for webpage testing.

Nightwatch

Nightwatch does not support import - from syntax and does not support partial JSON comparison as Playwright does. We can compare JSON partially in a loop if we want to, but in the current test it would not make sense. It supports response._body without another await for JSON parsing which is a plus. Having separate methods for setting the request body and headers is another plus, because it makes understanding the code somewhat easier. We have the same amount of code lines as with Playwright, so it is a real competitor. Personally I cannot decide I like both.

it('can read and delete posts', async ({supertest}) => {
	const user = await getUserWithToken(supertest, 'user1');
	const post = await getPost(supertest, 'post1b');

	let response = await supertest.request(api)
		.get(`posts/${post.id}`);
	expect(response.statusCode).to.eq(200);
	expect(response._body.title).to.eq(postsFixture.post1b.title);
	expect(response._body.body).to.eq(postsFixture.post1b.body);

	response = await supertest.request(api)
		.delete(`posts/${post.id}`)
		.set('authorization', 'Bearer '+user.token.plainText);
	expect(response.statusCode).to.eq(200);

	response = await supertest.request(api)
		.get(`posts/${post.id}`);
	expect(response.statusCode).to.eq(404);
});