I am also surprised, maybe an effort to privatise AI research?

➕ Bonus: Theory can explain the “Platonic Representation Hypothesis”—the striking observation that different models often learn the same representations.

arxiv.org/abs/2405.07987

With the right assumptions (hopefully not too 🧚‍♀️), you get a rigorous mathematical explanation for why this happens 🤓

So—does this mean theory is doomed, and AI engineering is just a random walk? Not at all!

💡 @rpatrik96.bsky.social @wielandbrendel.bsky.social Randall Balestriero did an amazing job clarifying where theory can help practice—and where practice should inspire theory.

🤝

In Theory fairyland:
Infinite data, models converge globally, latents on the right manifold, with the right statistics... 🧚

In Practice:
Noise, Batch size, learning rate, data augmentations, inductive biases, did I mention noise? & all the gritty stuff that actually matters 🛠️

(We still 💙 theory.)

What about any old system identification paper that shows how predictive performance goes up as a function of data?Not sure it fits the bill of scaling law and foundation model, but it shows pretty much the same thing 🤓

have you tried something like chromewebstore.google.com/detail/tree-...