Question: How to render my custom output via ns-render dataset

[stanathong]

Hello there,

I have modified instant_ngp.py to include my custom output. My custom output can be seen in nerfstudio viewer. However, when I use ns-render dataset to save the rendered images to file, the app hangs and throws the error NotImplementedError for apply_colormap.

I am able to render all (e.g. depth, rgb) but my custom output. Is there anything I have missed?

Any pointer to this would be greatly appreciated. Thank you very much.

For the details, below is what I have done:

1. File: models\instant_ngp.py, implement my custom output and add into the function get_outputs():

outputs = {
    "rgb": rgb,
    "accumulation": accumulation,
    "depth": depth,
    "num_samples_per_ray": packed_info[:, 1],
    "custom": custom,
}

2. File: models\instant_ngp.py, apply colour map to my custom output in the function get_image_metrics_and_images():

#custom = colormaps.apply_float_colormap(outputs["custom"])
custom = colormaps.apply_colormap(outputs["custom"])

...
combined_custom = torch.cat([custom], dim=1)

...
images_dict = {
    "img": combined_rgb,
    "accumulation": combined_acc,
    "depth": combined_depth,
    "custom": combined_custom,
}

[WojGaj24]

Have similar issue when trying to use ns-render dataset on the instant-ngp or instant-ngp-bound model (nerfacto and splatfacto worked fine with the same custom dataset).

When running the following command:

ns-render dataset --load-config D:\outputs\car\car\instant-ngp-bounded\2025-01-09_120024\config.yml --output-path D:\renders\car\instant-ngp-bound --split train+test

I get the same error message:

Traceback (most recent call last):
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\Scripts\ns-render.exe_main.py", line 7, in
sys.exit(entrypoint())
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\render.py", line 917, in entrypoint
tyro.cli(Commands).main()
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\lib\site-packages\nerfstudio\scripts\render.py", line 872, in main
colormaps.apply_colormap(
File "C:\Users\D3LUX\anaconda3\envs\nerfstudio\lib\site-packages\nerfstudio\utils\colormaps.py", line 90, in apply_colormap
raise NotImplementedError
NotImplementedError

Haven't found any solutions for this on any forums yet.

[stanathong]

Thank you for confirming this.

Do you mean custom dataset or custom output?

I think the reason could be that ns-render dataset doesn't support rendering custom output.

Anyway, I did a workaround that could render the outputs including my custom output.
In my case, I solved this by using ns-render camera-path instead of ns-render dataset.
What I have to do is to convert transforms.json, the one you obtained after running colmap via ns-process-data, to camera-path.json. It's simply just a json file with different formatting.
Then I just use ns-render camera-path with the camera-path.json from the conversion to render the results. This way, I got everything except gt-rgb. I'm not sure if the results are the same as that of ns-render dataset but it looks reasonable for me. Hope this helps!

[Factral]

Hi! have you tried using the --rendered-output-names flag? You can specify all the outputs you want, including custom ones like "rgb", "depth", "custom1", "custom", etc. This might help render your custom outputs without issues.

the flag is defined here

nerfstudio/nerfstudio/scripts/render.py

Lines 426 to 427 in 66cd681

	rendered_output_names: List[str] = field(default_factory=lambda: ["rgb"])
	"""Name of the renderer outputs to use. rgb, depth, etc. concatenates them along y axis"""

[stanathong]

Thanks for your comments. I did but it throw out error when I tried to render my custom output while other non-customs are fine.

I don't know if I need to do something extra, apart from adding new custom output to the returned output dict, to be able to render using ns-render dataset, as it works for ns-render camera-path, but of course they're different classes.

I believe it's around here to trace back which I haven't done yet after having found a workaround.

nerfstudio/nerfstudio/scripts/render.py

Lines 871 to 879 in 66cd681

	output_image = (
	colormaps.apply_colormap(
	image=output_image,
	colormap_options=self.colormap_options,
	)
	.cpu()
	.numpy()
	)

[WojGaj24]

In my case im just using a custom dataset not a custom output, I tried the --rendered-output-names flag for rgb but it doesn't solve the problem entirely , I no longer get an error and the images do render but here are the results :

instant-ngp:

same frame generated with nerfacto:

I restarted with a fresh install of the environment and dependencies and still have the same issues, not sure if the I'm missing some crucial step for making ns-render dataset work on instant-ngp model or if this functionality is currently not available.

[stanathong]

@WojGaj24 Thank for your update.
I do observe that the quality of rendered results from instant-ngp is noticeably poorer than that of nerfacto.
If you do not concern about training speed, I would recommend you stick with nerfacto.