not_zero_canonical

[smoreira00]

What does exactly this 'not_zero_canonical' parameter do? In which situations do I have to set it to True/False?

[yuehaowang]

When not_zero_canonical is set to False, the canonical frame is set to the first frame (t=0), i.e., all the displacements of the first frame are set to zeros. When set to True, the canonical frame is not explicitly specified.

Please refer to the related code:

EndoNeRF/run_endonerf_helpers.py

Lines 121 to 122 in 2d4546f

	if cur_time == 0. and self.zero_canonical:
	dx = torch.zeros_like(input_pts[:, :3])

In general, this hyperparameter only affects the rendering/reconstruction of the first frame.

[smoreira00]

So when we set not_zero_canonical to True, does the EndoNeRF randomly chooses a frame to set as the canonical frame? Or it doesn't set any frame as the canonical one?

[yuehaowang]

When not_zero_canonical is set to True, the model does not take any frame as the canonical space but just predicts displacement fields (generally non-zero) for each frame.

[smoreira00]

And when not_zero_canonical=False, why does the rendering/reconstruction of the first frame get so affected?

[yuehaowang]

This is because the canonical field is supposed to contain all the appeared areas in the whole video. Since there is no strong regularization of the predicted displacements, those new areas in the live frame might be remapped to random locations in the canonical field, resulting in some artifacts in the first frame.