Compute is increasing much faster than data. How can we improve classical supervised learning long term (the underlying tech of most of GenAI)?

Our ICML position paper's answer: simply train on a bunch of artificial data (noise) and only do inference on real-world data! 1/n

I am so proud to co-organize the workshop on foundation models for structured data at ICML. At this workshop, we will discuss on how to further extend the GenAI revolution to tabular data, time series forecasting etc. Consider submitting your work by May 19!
icml-structured-fm-workshop.github.io

Could it be that @fchollet.bsky.social is not Francois Chollet?? They have a lot of ML followers 😅

I believe lmarena.ai scores are not to be trusted, as the people voting are likely to come from the AI labs in the leaderboard and push their own models unintentionally. A thread 🧵

How wrong do you think are the lmarena scores? Grok must be very easy to distinguish from other models in a blind evaluation

TabPFN for Chemistry, check it out

MiniMax-01 takeaways

- 7 of 8 layers are linear att
- implemented a flash-variant of linear attention + ring-att
- post-norm is back in large models! (using deepnorm)
- prob. wrong scaling laws, as lr schedule is not adapted (see Chinchilla)

Let's see how it fares in the arena!

Los modelos preentrenados para datos tabulares (TabPFN) podrían ser el nuevo state of the art para regresión y clasificación. 🙄 Habrá que probarlo. El GitHub al final del hilo. Éste en concreto es enorme y si se comporta como dicen es un gran salto adelante en el state of the art del campo.

Groundbreaking work, congrats to the team!! 🎉 When I started my PhD 3 years ago, our tabular benchmark showed tree-based models miles ahead of neural networks. On the same benchmark, TabPFN v2 now reaches in 10s what CatBoost achieves in 4h of tuning 🤯

The tabular foundation model TabPFN v2 is finally public 🎉🥳
This is excellent news for (small) tabular ML! Checkout our Nature article (nature.com/articles/s41...) and code (github.com/PriorLabs/Ta...)

This might be the first time after 10 years that boosted trees are not the best default choice when working with data in tables.
Instead a pre-trained neural network is, the new TabPFN, as we just published in Nature 🎉

While ICLR forced everyone to review, it also had the worst reviewer-paper matching of any conference for me. 2 of the 3 papers were on topics that I never worked on before... :(