A massive thank you to all those involved in this work: Lyle Muller, Roberto Budzinski, and Demba Ba!

For more details, check out our paper recently accepted in workshop form to the 2025 ICLR Re-Align workshop, as well as the full preprint!

Paper: arxiv.org/abs/2502.06034
Code: github.com/KempnerInsti...

13/13

Check out our @kempnerinstitute.bsky.social blog post for audio on what different shapes sound like (to our models), as well as for more details and visualizations.

kempnerinstitute.harvard.edu/research/dee...

12/13

Here are some examples of the wave dynamics used to segment Multi-MNIST images:

11/13

We also compared our model to U-Nets, which have global receptive fields via skip connections and bottlenecks.

Incredibly, on Multi-MNIST, wave-based models outperformed similarly sized U-Nets, despite having fewer parameters and only local connectivity.

10/13

Notably, CNNs with small receptive fields (small # of layers) are unable to segment these images, while deeper models - with large receptive fields - are sometimes able to solve the task, but are generally more unstable yielding lower average performance and significantly higher variance.

9/13

We see that more complex linear transformations of the hidden state timeseries are the best for extracting the global information, with a learned linear transformation performing the best (even better than the Fourier transform or the common technique of using the last RNN hidden state).

8/13

We then studied both our wave-biased model and a standard ConvLSTM (with no wave-based inductive bias). Incredibly, we found that both models learned to generate waves. The ConvLSTM’s emergent waves (shown below on a Tetrominoes image) suggest a degree of optimality for a wave-based solution.

7/13

To test this, we built a trainable RNN (the Neural Wave Machine/NWM) that generates traveling waves in its hidden states. We began by testing it on segmenting simple polygons.

We find that wave-based models produce unique dynamics for each shape, resulting in distinct Fourier spectra.

6/13

We found that we could actually predict the area of the drums analytically by looking at the frequency of oscillations of each neuron (see below).

This finding led us to wonder: can we actually learn (via trainable parameters) dynamics for more complex shapes?

5/13

The problem "Can One Hear the Shape of a Drum", posed by Mark Mac, is a classical example of spatial integration. Strike a drumhead, and its vibrations encode the boundary shape.

We can see (with fixed RNNs that simulate drums) that different sized drumheads have different dynamics:

4/13

Spatial integration means that a neuron at one location can access signals from distant points. This could mean linking information together across an image to classify objects or linking words together in a sentence to derive meaning.

3/13

The act of vision is a coordinated activity involving millions of neurons in the visual cortex. How is information shared over these large distances?

Evidence suggests traveling waves could carry this information across space, allowing neurons to “know” what’s happening far away.

2/13