Worker tests: use unique pointers when possible #4725

Job	Run time
ci (ubuntu-20.04, 18, Release)	6m 1s
ci (ubuntu-20.04, 18, Debug)	5m 20s
ci (ubuntu-22.04, 20, Release)	5m 40s
ci (ubuntu-22.04, 20, Debug)	5m 5s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	8m 41s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	5m 53s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	9m 7s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	6m 12s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	7m 12s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	6m 1s
ci (macos-12, 18, Release)	9m 55s
ci (macos-12, 18, Debug)	8m 3s
ci (macos-14, 20, Release)	4m 29s
ci (macos-14, 20, Debug)	3m 52s
ci (windows-2022, 20, Release)	9m 38s
ci (windows-2022, 20, Debug)	7m 9s
	1h 48m 18s

Provide feedback