3⃣ We demonstrate what linear properties are shared by all or none LLMs.

🔥 Under mild assumptions, relational linear properties are shared!

⚠️ Parallel vectors may not be shared (they are under diversity)!

7/9

We also describe other linear properties: linear subspaces, probing, steering, based on relational strings (Paccanaro and Hinton, 2001).

💡They arise when the LLM can predict next-tokens for textual queries like: "What is the written language?" for many context strings!

6/9

2⃣ We reformulate linear properties of LLMs based on textual strings, depending on how LLMs predict next tokens

💡Parallel vectors arise from same log-ratios of next-token probs

E.g. same ratio for "easy"/"easiest" and "strong"/"strongest" in all contexts => parallel vecs

5/9

💡The extended linear equivalence underlies that two models' representations are linearly related, but in a subspace

‼️Outside that subspace, representations can differ a lot!

4/9

1⃣We extend the results by Khemakem et al. (2020), Roeder et al. (2021), removing a diversity assumption.

For the first time, we relate models with different repr. dimensions & find that repr.s of LLMs with same distribution are related by an “extended linear equivalence”!

3/9

🧵Why are linear properties so ubiquitous in LLM representations?

We explore this question through the lens of 𝗶𝗱𝗲𝗻𝘁𝗶𝗳𝗶𝗮𝗯𝗶𝗹𝗶𝘁𝘆:

“All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling”

Published at #AISTATS2025🌴

1/9

Hey hey! We have an accepted paper at #AISTATS2025!!
Time to prepare for Thailand 🪷🏖️🌴🐒

Huge thanks to my coauthors
Luigi Gresele, Sebastian Weichwald, and @seblachap.bsky.social for all the joint effort!

More details soon 👇

arxiv.org/abs/2410.235...

I know @looselycorrect.bsky.social well enough eheh