Worker tests: use unique pointers when possible #4728

Job	Run time
ci (ubuntu-20.04, 18, Release)	6m 28s
ci (ubuntu-20.04, 18, Debug)	5m 58s
ci (ubuntu-22.04, 20, Release)	5m 48s
ci (ubuntu-22.04, 20, Debug)	5m 0s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	9m 5s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	6m 1s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	9m 20s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	6m 21s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	7m 1s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	6m 7s
ci (macos-12, 18, Release)	8m 17s
ci (macos-12, 18, Debug)	6m 35s
ci (macos-14, 20, Release)	3m 57s
ci (macos-14, 20, Debug)	3m 58s
ci (windows-2022, 20, Release)	7m 48s
ci (windows-2022, 20, Debug)	8m 5s
	1h 45m 49s

Provide feedback