Now I see it. Counting inliers van work if there are only doppelgangers but can break down eg if there are doppelgangers + seasonal/illumination changes, which will influence the inlier count.

I see that assuming a solid object can help a lot in this case. But I don't think it is necessary, if the surroundings around the object and its doppelganger are different, since the correct pairs should have more inliers than the wrong ones, which could be used in the optimization.

OK. I would still not think that the reliance of pairwise matches should be an issue per se. If the optimizer is robust enough, it should be able to remove the wrong image pairs and keep only the good ones, I guess. And thanks for sharing your experience.

OK, thanks! Btw, how does GLOMAP handle doppelgangers? Because I would guess that the fact that it avoids incremental camera registration could help in this case.

Have you tried the new classical SfM pipelines like MP-SfM and GLOMAP?

Automatic generators for minimal solvers. And the homotopy continuation.