Worker tests: use unique pointers when possible #4716

Job	Run time
ci (ubuntu-20.04, 18, Release)	6m 1s
ci (ubuntu-20.04, 18, Debug)	5m 36s
ci (ubuntu-22.04, 20, Release)	5m 56s
ci (ubuntu-22.04, 20, Debug)	5m 5s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	9m 9s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	6m 8s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	9m 14s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	6m 15s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	7m 1s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	6m 4s
ci (macos-12, 18, Release)	11m 42s
ci (macos-12, 18, Debug)	8m 7s
ci (macos-14, 20, Release)	4m 22s
ci (macos-14, 20, Debug)	5m 33s
ci (windows-2022, 20, Release)	8m 59s
ci (windows-2022, 20, Debug)	8m 27s
	1h 53m 39s

Provide feedback