The key is working in projective space, estimating only fundamental matrices and distortion parameters. These can then be used to initialize full SfM, leading to an overall more robust pipeline.

Check out the jupyter notebook for a typical example with python bindings github.com/DaniilSinits...

in practice the angle between the observation ray and principal axis will always be limited by the camera fov, so not sure how much difference the fix would do tbh

but yeah, eg if translation noise is the main source of error I could see midpoint being optimal!

consider me nerd-sniped 😅 one benefit I can see for the reprojection error is that it gives a better tradeoff when cameras are at different distances.

Here's a 3-view example:
blue=GT point
red=optimal projection error
green=optimal point-to-ray distance

all views have the same observation noise