Herbie(Zizhan) He
@herbiehe.bsky.social
Incoming CS master's student @mcgillu.bsky.social
[12/12]
We believe ESWM points to a new generation of brain-inspired models—ones that reason over fragments, generalize across structure, and adapt on the fly.
📄 arxiv.org/abs/2505.13696
👥 @maximemdaigle.bsky.social , @bashivan.bsky.social
We believe ESWM points to a new generation of brain-inspired models—ones that reason over fragments, generalize across structure, and adapt on the fly.
📄 arxiv.org/abs/2505.13696
👥 @maximemdaigle.bsky.social , @bashivan.bsky.social
Building spatial world models from sparse transitional episodic memories
Many animals possess a remarkable capacity to rapidly construct flexible mental models of their environments. These world models are crucial for ethologically relevant behaviors such as navigation, ex...
arxiv.org
June 28, 2025 at 1:09 AM
[12/12]
We believe ESWM points to a new generation of brain-inspired models—ones that reason over fragments, generalize across structure, and adapt on the fly.
📄 arxiv.org/abs/2505.13696
👥 @maximemdaigle.bsky.social , @bashivan.bsky.social
We believe ESWM points to a new generation of brain-inspired models—ones that reason over fragments, generalize across structure, and adapt on the fly.
📄 arxiv.org/abs/2505.13696
👥 @maximemdaigle.bsky.social , @bashivan.bsky.social
[11/12]
🔧 When environments change—say a new wall appears—ESWM adapts instantly. No retraining is needed. Just update the memory bank and the model replans.
This separation of memory and reasoning makes ESWM highly flexible.
🔧 When environments change—say a new wall appears—ESWM adapts instantly. No retraining is needed. Just update the memory bank and the model replans.
This separation of memory and reasoning makes ESWM highly flexible.
June 28, 2025 at 1:09 AM
[11/12]
🔧 When environments change—say a new wall appears—ESWM adapts instantly. No retraining is needed. Just update the memory bank and the model replans.
This separation of memory and reasoning makes ESWM highly flexible.
🔧 When environments change—say a new wall appears—ESWM adapts instantly. No retraining is needed. Just update the memory bank and the model replans.
This separation of memory and reasoning makes ESWM highly flexible.
[10/12]
🧭 It gets even better!
ESWM can navigate between arbitrary points using only its memory bank—planning efficiently in latent space with near-optimal paths.
No access to global maps or coordinates required.
🧭 It gets even better!
ESWM can navigate between arbitrary points using only its memory bank—planning efficiently in latent space with near-optimal paths.
No access to global maps or coordinates required.
June 28, 2025 at 1:09 AM
[10/12]
🧭 It gets even better!
ESWM can navigate between arbitrary points using only its memory bank—planning efficiently in latent space with near-optimal paths.
No access to global maps or coordinates required.
🧭 It gets even better!
ESWM can navigate between arbitrary points using only its memory bank—planning efficiently in latent space with near-optimal paths.
No access to global maps or coordinates required.
[9/12]
🚶 With no additional training, ESWM can explore novel environments efficiently by acting on uncertainty.
🚶 With no additional training, ESWM can explore novel environments efficiently by acting on uncertainty.
June 28, 2025 at 1:09 AM
[9/12]
🚶 With no additional training, ESWM can explore novel environments efficiently by acting on uncertainty.
🚶 With no additional training, ESWM can explore novel environments efficiently by acting on uncertainty.
[8/12]
⚙️ How are these maps built?
We find that ESWM stitches together memories via overlapping states—merging local transitions into global structure.
Obstacles and boundaries serve as spatial anchors, guiding how memories are organized in latent space.
⚙️ How are these maps built?
We find that ESWM stitches together memories via overlapping states—merging local transitions into global structure.
Obstacles and boundaries serve as spatial anchors, guiding how memories are organized in latent space.
June 28, 2025 at 1:09 AM
[8/12]
⚙️ How are these maps built?
We find that ESWM stitches together memories via overlapping states—merging local transitions into global structure.
Obstacles and boundaries serve as spatial anchors, guiding how memories are organized in latent space.
⚙️ How are these maps built?
We find that ESWM stitches together memories via overlapping states—merging local transitions into global structure.
Obstacles and boundaries serve as spatial anchors, guiding how memories are organized in latent space.
[7/12]
🏞️ How does ESWM solve the task?
Using ISOMAP, we visualize its latent representations—beautifully organized spatial layouts emerge from its internal states, even when the model sees only a small part or out-of-distribution environments.
🏞️ How does ESWM solve the task?
Using ISOMAP, we visualize its latent representations—beautifully organized spatial layouts emerge from its internal states, even when the model sees only a small part or out-of-distribution environments.
June 28, 2025 at 1:09 AM
[7/12]
🏞️ How does ESWM solve the task?
Using ISOMAP, we visualize its latent representations—beautifully organized spatial layouts emerge from its internal states, even when the model sees only a small part or out-of-distribution environments.
🏞️ How does ESWM solve the task?
Using ISOMAP, we visualize its latent representations—beautifully organized spatial layouts emerge from its internal states, even when the model sees only a small part or out-of-distribution environments.
[6/12]
⚡️ Transformer-based ESWM models outperform LSTMs and Mamba, especially in settings where observations are compositional. Attention allows the model to flexibly bind relevant memories and generalize across structures.
⚡️ Transformer-based ESWM models outperform LSTMs and Mamba, especially in settings where observations are compositional. Attention allows the model to flexibly bind relevant memories and generalize across structures.
June 28, 2025 at 1:09 AM
[6/12]
⚡️ Transformer-based ESWM models outperform LSTMs and Mamba, especially in settings where observations are compositional. Attention allows the model to flexibly bind relevant memories and generalize across structures.
⚡️ Transformer-based ESWM models outperform LSTMs and Mamba, especially in settings where observations are compositional. Attention allows the model to flexibly bind relevant memories and generalize across structures.
[5/12]
To train ESWM, we use meta-learning across diverse environments. At test time, the model gets a minimal set of disjoint episodic memories (single transitions) and must predict a missing element in a new transition—without ever seeing the full map.
To train ESWM, we use meta-learning across diverse environments. At test time, the model gets a minimal set of disjoint episodic memories (single transitions) and must predict a missing element in a new transition—without ever seeing the full map.
June 28, 2025 at 1:09 AM
[5/12]
To train ESWM, we use meta-learning across diverse environments. At test time, the model gets a minimal set of disjoint episodic memories (single transitions) and must predict a missing element in a new transition—without ever seeing the full map.
To train ESWM, we use meta-learning across diverse environments. At test time, the model gets a minimal set of disjoint episodic memories (single transitions) and must predict a missing element in a new transition—without ever seeing the full map.
[4/12]
🧠 Inspired by the MTL’s architecture and function, we built ESWM: a neural network that infers the structure of its environment from isolated, one-step transitions—just like the brain integrates episodes into a cognitive map.
🧠 Inspired by the MTL’s architecture and function, we built ESWM: a neural network that infers the structure of its environment from isolated, one-step transitions—just like the brain integrates episodes into a cognitive map.
June 28, 2025 at 1:09 AM
[4/12]
🧠 Inspired by the MTL’s architecture and function, we built ESWM: a neural network that infers the structure of its environment from isolated, one-step transitions—just like the brain integrates episodes into a cognitive map.
🧠 Inspired by the MTL’s architecture and function, we built ESWM: a neural network that infers the structure of its environment from isolated, one-step transitions—just like the brain integrates episodes into a cognitive map.
[3/12]
Such shortcut-seeking behavior is supported by internal world models in the brain’s medial temporal lobe (MTL), an area involved in both episodic memory and spatial navigation.
Such shortcut-seeking behavior is supported by internal world models in the brain’s medial temporal lobe (MTL), an area involved in both episodic memory and spatial navigation.
June 28, 2025 at 1:09 AM
[3/12]
Such shortcut-seeking behavior is supported by internal world models in the brain’s medial temporal lobe (MTL), an area involved in both episodic memory and spatial navigation.
Such shortcut-seeking behavior is supported by internal world models in the brain’s medial temporal lobe (MTL), an area involved in both episodic memory and spatial navigation.
[2/12]
Many animals can infer unexperienced paths by integrating disjoint memories. Mice, e.g., take shortcuts they’ve never physically traversed: pubmed.ncbi.nlm.nih.gov/17469957/
Many animals can infer unexperienced paths by integrating disjoint memories. Mice, e.g., take shortcuts they’ve never physically traversed: pubmed.ncbi.nlm.nih.gov/17469957/
Rats take correct novel routes and shortcuts in an enclosed maze - PubMed
In 3 experiments, rats were allowed to travel selected routes along the internal alleys of a cross-maze that led from one distinctive end box to another. The maze and procedures used were designed to ...
pubmed.ncbi.nlm.nih.gov
June 28, 2025 at 1:09 AM
[2/12]
Many animals can infer unexperienced paths by integrating disjoint memories. Mice, e.g., take shortcuts they’ve never physically traversed: pubmed.ncbi.nlm.nih.gov/17469957/
Many animals can infer unexperienced paths by integrating disjoint memories. Mice, e.g., take shortcuts they’ve never physically traversed: pubmed.ncbi.nlm.nih.gov/17469957/