Less Decoder is More Encoder: Geometric Representation Learning from Novel View Synthesis
Keerthi Kaashyap, Dennis Anthony, Akshay Krishnan, Nhi Ngoc Nguyen, Jeremy Collins, James Hays, et al.
arXiv:2610.03717v1Novel view synthesis sounds like a rendering problem, but this paper asks a deeper question: can it also teach a model about 3D geometry? The challenge is that many self-supervised methods use a powerful decoder that can do too much of the work, letting the encoder learn weak, transferable features. They also train on pixel-level reconstruction, which encourages low-level details instead of geometric structure. The authors’ solution, called SNAP, changes both parts of the recipe. It uses a pose-conditioned local decoder, so the decoder is less able to hide mistakes, and it reconstructs in latent space rather than directly in pixels. That pushes the encoder to capture scene structure that survives viewpoint changes. The result is a task-agnostic representation that performs competitively on localization, pose estimation, correspondence, depth, and even robot manipulation. What makes this exciting is the broader lesson: sometimes better representations come from limiting model flexibility, not increasing it.
Also spotted that day
Previous daily papers