I am deeply grateful to @lorenzlinhardt.bsky.social, Marco Morik, Jonas Dippel, Simon Kornblith, and @lukasmut.bsky.social for their great work and support in this project! We also thank our collaborators, @bifold.berlin and HFA 7/7
📄Paper: arxiv.org/abs/2411.05561
💻Code: github.com/lciernik/sim...

2nd key insight: The link between model similarity & behavior varies by dataset. Single-domain sets show strong correlations, while some multi-domain ones have high-performing, dissimilar models. Thus, the Platonic Representation Hypothesis may depend on the dataset's nature. 🧵 6/7

Key finding: Training objective is a crucial factor for similarity consistency! SSL models show remarkably consistent representations across stimulus sets compared to image-text and supervised models, which show high variance in their consistency due to dataset dependence. 🧵 5/7

Thus, we suggest a framework to systematically study if relative representational similarities between models remain consistent. We measure similarities between sets of models with different traits and their correlation across dataset pairs to assess stability across stimuli. 🧵4/7

First finding: Representational similarities do not transfer directly across datasets, showing high variability across datasets, such as different ranges and patterns. 🧵 3/7

The Platonic Rep. Hypothesis @phillipisola.bsky.social et al. suggests foundation models converge to a shared representation space. Yet, most studies consider single datasets when measuring representational similarity. Thus, we were wondering: Does this convergence hold more broadly? 🧵 2/7