We found that this image-computable model performs as well as the previous state-based model, and also better predicts participants’ behavioral patterns than alternative models.

Here’s an example of how the model (right) tracks the object in the video (left) as it moves and continues simulating after the video ends.

In this work, we treat vision as inverse graphics and develop a model that infers a distribution over object states from raw visual input. This uncertainty is then passed to a probabilistic physics simulator to generate predictions about what will happen next.

Some theories suggest we run mental simulations to predict physical events. But it's unclear how people figure out what's in a scene just by looking — and how that affects their physical predictions. Most current models assume the 3D states (poses, velocities, etc.) of objects are already known.

We asked participants to predict whether an object would touch another after viewing a short video clip. The videos feature realistic 3D scenarios generated using a game engine (see my CogSci paper from last year for more details! tinyurl.com/43fh62yt).

Can you tell if a tower will fall or if two objects will collide — just by looking? 🧠👀 Come check out my #CogSci2025‪ poster (P1-W-207) on July 31, 13:00–14:15 PT to learn how people do general-purpose physical reasoning from visual input!