Fix performance regression due to extra arrow round-tripping #16081

Job	Run time
paths-filter	5s
Checks / Rerun lints	36s
Checks / Check if running codegen would produce any changes	1m 0s
Checks / Lint markdown	19s
Checks / Spell Check	7s
Checks / Misc formatting	21s
Checks / markdown-paths-filter	7s
Deploy Landing Preview	0s
C++ tests	0s
Python Checks	0s
Rust Checks / Test on ${{ matrix.name }}	0s
Build web viewer / Build web viewer	3m 25s
Build rerun_js / Build rerun_js	3m 33s
Minimum CLI Build / Set Config (linux-x64)	2s
Rust Checks / Rust lints (fmt, check, clippy, tests, doc)	13m 0s
Checks / Check links	0s
Minimum CLI Build / Build rerun-cli (ubuntu-latest-16-cores)	5m 53s
Upload Web / Upload web build to google cloud (wasm32 + wasm-bindgen)	10s
Minimum Wheel Build / Set Config (linux-x64)	2s
Minimum Wheel Build / Build Wheels (ubuntu-latest-16-cores)	2m 20s
Minimum Wheel Test / Set Config (linux-x64)	6s
Minimum Wheel Test / Test Wheels (ubuntu-latest-16-cores)	3m 57s
	35m 3s

Provide feedback