Worker tests: use unique pointers when possible #4715

Job	Run time
ci (ubuntu-20.04, 18, Release)	6m 3s
ci (ubuntu-20.04, 18, Debug)	5m 31s
ci (ubuntu-22.04, 20, Release)	5m 55s
ci (ubuntu-22.04, 20, Debug)	5m 17s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	8m 38s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	5m 56s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	9m 21s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	5m 54s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	7m 1s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	5m 54s
ci (macos-12, 18, Release)	6m 42s
ci (macos-12, 18, Debug)	9m 33s
ci (macos-14, 20, Release)	4m 42s
ci (macos-14, 20, Debug)	4m 35s
ci (windows-2022, 20, Release)	9m 21s
ci (windows-2022, 20, Debug)	6m 59s
	1h 47m 22s

Provide feedback