A big thank you to my co-authors: @nishadg.bsky.social, @joshtenenbaum.bsky.social and @thiskevinsmith.bsky.social, see us at #CogSci2025 @cogscisociety.bsky.social

In ongoing work, we are exploring ways to incorporate more physics knowledge into the inverse-graphics framework to model people’s inference about physical properties, object’s motion under occlusion, etc.

We found that this image-computable model performs as well as the previous state-based model, and also better predicts participants’ behavioral patterns than alternative models.

Here’s an example of how the model (right) tracks the object in the video (left) as it moves and continues simulating after the video ends.

In this work, we treat vision as inverse graphics and develop a model that infers a distribution over object states from raw visual input. This uncertainty is then passed to a probabilistic physics simulator to generate predictions about what will happen next.

Some theories suggest we run mental simulations to predict physical events. But it's unclear how people figure out what's in a scene just by looking — and how that affects their physical predictions. Most current models assume the 3D states (poses, velocities, etc.) of objects are already known.

We found that people did a good job at this task (80% accuracy)! And more importantly, they were highly consistent with each other (95% inter-participant correlation), meaning they not only made good predictions but also similar errors. What explains this interesting behavior?

We asked participants to predict whether an object would touch another after viewing a short video clip. The videos feature realistic 3D scenarios generated using a game engine (see my CogSci paper from last year for more details! tinyurl.com/43fh62yt).