Worker tests: use unique pointers when possible #4731

Job	Run time
ci (ubuntu-20.04, 18, Release)	5m 56s
ci (ubuntu-20.04, 18, Debug)	5m 34s
ci (ubuntu-22.04, 20, Release)	5m 34s
ci (ubuntu-22.04, 20, Debug)	5m 9s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Release)	8m 49s
ci (ubuntu-24.04, 20, -Db_sanitize=address, Debug)	6m 0s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Release)	9m 18s
ci (ubuntu-24.04, 20, -Db_sanitize=undefined, Debug)	6m 21s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Release)	7m 2s
ci (ubuntu-24.04, 20, -Db_sanitize=thread, Debug)	5m 54s
ci (macos-12, 18, Release)	7m 22s
ci (macos-12, 18, Debug)	5m 57s
ci (macos-14, 20, Release)	5m 54s
ci (macos-14, 20, Debug)	3m 44s
ci (windows-2022, 20, Release)	8m 21s
ci (windows-2022, 20, Debug)	7m 45s
	1h 44m 40s

Provide feedback