I connected my laptop to my piano and typed into the terminal “connect to my piano and play a few notes with midi” and it worked first try. This is some Star Trek shit. If you told me 5 years ago this would be possible today I would not have believed you.
“I don’t want to hear from Mitchell because I don’t think I would enjoy her content” - sure whatever (you’re misrepresenting her work but that’s your choice). “I don’t want to hear from Mitchell because she doesn’t know how NNs work” makes you sound like an uninformed asshole.
The paper as a whole holds up! It’s about the risks/limitations of scaling language models - all very relevant today! How many NLP papers from 2020-2021 can you say that about?
So to recap, you don’t want to ever hear from Mitchell because of one sentence in a paper that summarizes her co-authors position re: a linguistic theory about form vs meaning, which disqualifies her from ever knowing how these things work “in a relevant sense” ?
The premise of the paper is “there are risks/downsides to larger models.” Nowhere in the paper does it claim anything like “language models can’t generalize to unseen prompts.” You’re just straw manning some thesis onto the paper based on the phrase “Stochastic Parrots.”
I don’t think this is bad faith. Margaret Mitchell has a long CV with plenty of papers that go beyond the scope of the Stochastic Parrots paper that clearly demonstrate she knows how NNs work?
My takeaway is deberta baseline is the winner here? Way easier to train/deploy. Also what if you scaled the encoder-classifier up to a comparable size?
Right but we have users who are like “I can’t find the [microsoft] copilot button” - getting them to install/figure out Claude code is just not practical.
Good stuff. Does this thinking extend to more general things like Microsoft copilot and ChatGPT? Or are you saying normies should start using coding agents
There was an interesting paper earlier this year about a “recurrent depth” technique that allowed the model to reuse layers … this what you mean? arxiv.org/abs/2502.05171
Yeah plenty of examples of code golf / people trying to put like 5 lines of code in a single line to show that They Can and it just makes unreadable garbage
Was thinking about this re: “wow I should really get better and writing clear and consistent documentation for my repos so my agents know how to use it”
I love your concept about building bespoke dev tools (like ways to search logs) for the agents - would love to hear about more of these and how you approach building them!
Build fully open models in the US in the next two years to enable a flourishing, global scientific AI ecosystem to balance China's surge in open-source and an alternative to building products ontop of leading closed models. buff.ly/kvJQE3I
