Will Held
@williamheld.com
Modeling Linguistic Variation to expand ownership of NLP tools
Views my own, but affiliations that might influence them:
ML PhD Student under Prof. Diyi Yang
2x RS Intern🦙 Pretraining
Alum NYU Abu Dhabi
Burqueño
he/him
Views my own, but affiliations that might influence them:
ML PhD Student under Prof. Diyi Yang
2x RS Intern🦙 Pretraining
Alum NYU Abu Dhabi
Burqueño
he/him
Pinned
Will Held
@williamheld.com
· Jan 22
Balancing data across domains is key to training the best generalist LLMs!
In my summer work on the Meta Llama team, we introduce UtiliMax and MEDU, new methods to estimate data utility and optimize data mixes efficiently.
HF Blog: huggingface.co/blog/WillHel...
ArXiv: arxiv.org/abs/2501.11747
In my summer work on the Meta Llama team, we introduce UtiliMax and MEDU, new methods to estimate data utility and optimize data mixes efficiently.
HF Blog: huggingface.co/blog/WillHel...
ArXiv: arxiv.org/abs/2501.11747
Super interested to what degree this interaction can be fine-tuned into models in a non-reversible fashion!
Voice cloning is unfortunately a capability which inherently shows up in pretrained audio models. It would be great to be able to largely limit the capability at the level of model weights!
Voice cloning is unfortunately a capability which inherently shows up in pretrained audio models. It would be great to be able to largely limit the capability at the level of model weights!
🤖 Did you know your voice might be cloned without your consent from just *one sentence* of audio?
That's not great. So with @frimelle.bsky.social, we brainstormed a new idea for developers who want to curb malicious use: ✨The Voice Consent Gate.✨
Details, code, here: huggingface.co/blog/voice-c...
That's not great. So with @frimelle.bsky.social, we brainstormed a new idea for developers who want to curb malicious use: ✨The Voice Consent Gate.✨
Details, code, here: huggingface.co/blog/voice-c...
October 29, 2025 at 3:01 PM
Super interested to what degree this interaction can be fine-tuned into models in a non-reversible fashion!
Voice cloning is unfortunately a capability which inherently shows up in pretrained audio models. It would be great to be able to largely limit the capability at the level of model weights!
Voice cloning is unfortunately a capability which inherently shows up in pretrained audio models. It would be great to be able to largely limit the capability at the level of model weights!
Reposted by Will Held
Now that school is starting for lots of folks, it's time for a new release of Speech and Language Processing! Jim and I added all sorts of material for the August 2025 release! With slides to match! Check it out here: web.stanford.edu/~jurafsky/sl...
Speech and Language Processing
Speech and Language Processing
web.stanford.edu
August 24, 2025 at 7:28 PM
Now that school is starting for lots of folks, it's time for a new release of Speech and Language Processing! Jim and I added all sorts of material for the August 2025 release! With slides to match! Check it out here: web.stanford.edu/~jurafsky/sl...
"GPT-5 shows scaling laws are coming to an end"
August 11, 2025 at 5:46 PM
"GPT-5 shows scaling laws are coming to an end"
Reposted by Will Held
We’ve discovered a literal miracle with almost unlimited potential and it’s being scrapped for *no reason whatsoever*. This isn’t even nihilism, it’s outright worship of death and human suffering.
"The U.S. Department of Health and Human Services (HHS) today announced the beginning of a coordinated wind-down of its mRNA vaccine development activities...."
cc: Sen. Bill Cassidy
cc: Sen. Bill Cassidy
August 5, 2025 at 11:09 PM
We’ve discovered a literal miracle with almost unlimited potential and it’s being scrapped for *no reason whatsoever*. This isn’t even nihilism, it’s outright worship of death and human suffering.
Really great pointer from Hao Zhang on the other site in relation to GPT OSS use of attention sinks.
If I were to guess, the attention sink is what allows them to omit QK-Norm which has become otherwise standard.
www.evanmiller.org/attention-is...
If I were to guess, the attention sink is what allows them to omit QK-Norm which has become otherwise standard.
www.evanmiller.org/attention-is...
Attention Is Off By One
Let’s fix these pesky Transformer outliers using Softmax One and QuietAttention.
www.evanmiller.org
August 6, 2025 at 12:48 PM
Really great pointer from Hao Zhang on the other site in relation to GPT OSS use of attention sinks.
If I were to guess, the attention sink is what allows them to omit QK-Norm which has become otherwise standard.
www.evanmiller.org/attention-is...
If I were to guess, the attention sink is what allows them to omit QK-Norm which has become otherwise standard.
www.evanmiller.org/attention-is...
The SALT Lab is at #ACL2025 with our genius leader @diyiyang.bsky.social.
Come see work from
@yanzhe.bsky.social,
@dorazhao.bsky.social @oshaikh.bsky.social,
@michaelryan207.bsky.social, and myself at any of the talks and posters below!
Come see work from
@yanzhe.bsky.social,
@dorazhao.bsky.social @oshaikh.bsky.social,
@michaelryan207.bsky.social, and myself at any of the talks and posters below!
July 28, 2025 at 7:45 AM
The SALT Lab is at #ACL2025 with our genius leader @diyiyang.bsky.social.
Come see work from
@yanzhe.bsky.social,
@dorazhao.bsky.social @oshaikh.bsky.social,
@michaelryan207.bsky.social, and myself at any of the talks and posters below!
Come see work from
@yanzhe.bsky.social,
@dorazhao.bsky.social @oshaikh.bsky.social,
@michaelryan207.bsky.social, and myself at any of the talks and posters below!
I'm in Vienna for #ACL2025!
My work is all presented tomorrow, but today you'll find me today at the poster session from 11-12:30 evangelizing
my labmate Yanzhe Zhang's work on his behalf.
If you're interested in the risks traditional pop-up attacks present for AI agents, come chat!
My work is all presented tomorrow, but today you'll find me today at the poster session from 11-12:30 evangelizing
my labmate Yanzhe Zhang's work on his behalf.
If you're interested in the risks traditional pop-up attacks present for AI agents, come chat!
July 28, 2025 at 4:24 AM
I'm in Vienna for #ACL2025!
My work is all presented tomorrow, but today you'll find me today at the poster session from 11-12:30 evangelizing
my labmate Yanzhe Zhang's work on his behalf.
If you're interested in the risks traditional pop-up attacks present for AI agents, come chat!
My work is all presented tomorrow, but today you'll find me today at the poster session from 11-12:30 evangelizing
my labmate Yanzhe Zhang's work on his behalf.
If you're interested in the risks traditional pop-up attacks present for AI agents, come chat!
A while ago I mentioned that for marin.community project, this gradient increase led to problematic loss ascent which we patched with Z-loss.
I was curious, does AdamC just work?
So over the weekend, I ran 4 experiments—130M to 1.4B params—all at ~compute-optimal token counts...🧵
I was curious, does AdamC just work?
So over the weekend, I ran 4 experiments—130M to 1.4B params—all at ~compute-optimal token counts...🧵
July 3, 2025 at 3:15 PM
A while ago I mentioned that for marin.community project, this gradient increase led to problematic loss ascent which we patched with Z-loss.
I was curious, does AdamC just work?
So over the weekend, I ran 4 experiments—130M to 1.4B params—all at ~compute-optimal token counts...🧵
I was curious, does AdamC just work?
So over the weekend, I ran 4 experiments—130M to 1.4B params—all at ~compute-optimal token counts...🧵
kyutai.org/next/unmute has built in turn-detection on the ASR and full I/O streaming for the TTS. Solves the latency issues that I think are 90% of why people use end-to-end speech models in the first place!
From the details, you can @kyutai-labs.bsky.social is focused on real-world utility.
From the details, you can @kyutai-labs.bsky.social is focused on real-world utility.
Unmute by Kyutai
Make LLMs listen and speak.
unmute.sh
July 3, 2025 at 3:05 PM
kyutai.org/next/unmute has built in turn-detection on the ASR and full I/O streaming for the TTS. Solves the latency issues that I think are 90% of why people use end-to-end speech models in the first place!
From the details, you can @kyutai-labs.bsky.social is focused on real-world utility.
From the details, you can @kyutai-labs.bsky.social is focused on real-world utility.
Reposted by Will Held
Flattered and shocked for our paper to receive the #facct2025 best paper award.
🏆 Announcing the #FAccT2025 best paper awards! 🏆
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...
Announcing Best Paper Awards
The Best Paper Award Committee was chaired this year by Alex Chouldechova and included six Area Chairs. The committee selected three papers for the Best Paper Award and recognized three additional pap...
facct-blog.github.io
June 21, 2025 at 1:16 AM
Flattered and shocked for our paper to receive the #facct2025 best paper award.
I've only seen Veo 3 (or any other video generation model) used to produce viral videos. The fake videos seem to successfully trick the majority of commenters and have no visible watermark or disclosure of AI use.
June 17, 2025 at 1:24 AM
I've only seen Veo 3 (or any other video generation model) used to produce viral videos. The fake videos seem to successfully trick the majority of commenters and have no visible watermark or disclosure of AI use.
Reposted by Will Held
What would you say if you saw it in another country? A senator from a coequal branch of government dragged away by security from asking a question of a Cabinet official
Kristi Noem: "We are not going away. We are staying here to liberate the city from the socialists and the burdensome leadership that this governor and that this mayor have placed on this country and what they have tried to insert into the city."
Sen. Alex Padilla is then forcibly removed!
Sen. Alex Padilla is then forcibly removed!
June 12, 2025 at 6:33 PM
What would you say if you saw it in another country? A senator from a coequal branch of government dragged away by security from asking a question of a Cabinet official
Reposted by Will Held
🚨 70 million US workers are about to face their biggest workplace transmission due to AI agents. But nobody’s asking them what they want.
While AI R&D races to automate everything, we took a different approach: auditing what workers want vs. what AI can deliver across the US workforce.🧵
While AI R&D races to automate everything, we took a different approach: auditing what workers want vs. what AI can deliver across the US workforce.🧵
June 12, 2025 at 4:34 PM
🚨 70 million US workers are about to face their biggest workplace transmission due to AI agents. But nobody’s asking them what they want.
While AI R&D races to automate everything, we took a different approach: auditing what workers want vs. what AI can deliver across the US workforce.🧵
While AI R&D races to automate everything, we took a different approach: auditing what workers want vs. what AI can deliver across the US workforce.🧵
Really cool to see theory connect to practice! We observed this phenomenon when trying to do deeper WSD cooldowns of our 8B model in the marin.community project!
We Z-Lossed our way through the pain, but cool to see some stronger theory: marin.readthedocs.io/en/latest/re...
We Z-Lossed our way through the pain, but cool to see some stronger theory: marin.readthedocs.io/en/latest/re...
June 6, 2025 at 1:27 AM
Really cool to see theory connect to practice! We observed this phenomenon when trying to do deeper WSD cooldowns of our 8B model in the marin.community project!
We Z-Lossed our way through the pain, but cool to see some stronger theory: marin.readthedocs.io/en/latest/re...
We Z-Lossed our way through the pain, but cool to see some stronger theory: marin.readthedocs.io/en/latest/re...
Reposted by Will Held
What foreign power could do as much damage to the United States as Trump is doing to it right now? www.whitehouse.gov/presidential...
Enhancing National Security by Addressing Risks at Harvard University
BY THE PRESIDENT OF THE UNITED STATES OF AMERICA A PROCLAMATION Admission into the United States to attend, conduct research, or teach at our
www.whitehouse.gov
June 5, 2025 at 1:07 AM
What foreign power could do as much damage to the United States as Trump is doing to it right now? www.whitehouse.gov/presidential...
Based on current administration policies, China is about to have an influx of returning talent and a accelerated advantage in research investments.
You need to be both sinophobic and irrational to expect the US to continue as the global scientific powerhouse with these policy own-goals.
You need to be both sinophobic and irrational to expect the US to continue as the global scientific powerhouse with these policy own-goals.
June 2, 2025 at 2:59 AM
Based on current administration policies, China is about to have an influx of returning talent and a accelerated advantage in research investments.
You need to be both sinophobic and irrational to expect the US to continue as the global scientific powerhouse with these policy own-goals.
You need to be both sinophobic and irrational to expect the US to continue as the global scientific powerhouse with these policy own-goals.
Reposted by Will Held
"“From time-to-time instances will arise in which the society, or segments of it, threaten the very mission of the university & its values... In such a crisis, it becomes the obligation of the university as an institution to oppose such measures & actively to defend its interests and its values.”
Bravo, to Stanford faculty, led by physics, to ask their administrators to stand up and fight Trump.
stanforddaily.com/2025/05/22/f...
stanforddaily.com/2025/05/22/f...
From the Community | Stanford professors respond to political interference in the governance of U.S. universities
Over 300 Stanford professors respond to Trump administration's interference in U.S. universities.
stanforddaily.com
May 25, 2025 at 3:07 PM
"“From time-to-time instances will arise in which the society, or segments of it, threaten the very mission of the university & its values... In such a crisis, it becomes the obligation of the university as an institution to oppose such measures & actively to defend its interests and its values.”
Reposted by Will Held
Super excited Marin is finally out! Come see what we've been building! Code/platform for training fully reproducible models end-to-end, from data to evals. Plus a new high quality 8B base model. Percy did a good job explaining it on the other place. marin.community
x.com/percyliang/s...
x.com/percyliang/s...
Percy Liang on X: "What would truly open-source AI look like? Not just open weights, open code/data, but *open development*, where the entire research and development process is public *and* anyone can contribute. We built Marin, an open lab, to fulfill this vision: https://t.co/racsvmhyA3" / X
What would truly open-source AI look like? Not just open weights, open code/data, but *open development*, where the entire research and development process is public *and* anyone can contribute. We built Marin, an open lab, to fulfill this vision: https://t.co/racsvmhyA3
x.com
May 19, 2025 at 7:35 PM
Super excited Marin is finally out! Come see what we've been building! Code/platform for training fully reproducible models end-to-end, from data to evals. Plus a new high quality 8B base model. Percy did a good job explaining it on the other place. marin.community
x.com/percyliang/s...
x.com/percyliang/s...
How much faster would the science of large-scale AI advance if we could open-source the *process* of building a frontier model?
Not just the final models/code/data, but also negative results, toy experiments, and even spontaneous discussions.
That's what we're trying @ marin.community
Not just the final models/code/data, but also negative results, toy experiments, and even spontaneous discussions.
That's what we're trying @ marin.community
May 19, 2025 at 7:05 PM
How much faster would the science of large-scale AI advance if we could open-source the *process* of building a frontier model?
Not just the final models/code/data, but also negative results, toy experiments, and even spontaneous discussions.
That's what we're trying @ marin.community
Not just the final models/code/data, but also negative results, toy experiments, and even spontaneous discussions.
That's what we're trying @ marin.community
It feels worth conference organizers running a study to see if this significantly impacts reviewer scores.
I hope things like this are placebos, but if not we need to seriously consider whether existing peer-review processes for big ML conferences are providing value.
I hope things like this are placebos, but if not we need to seriously consider whether existing peer-review processes for big ML conferences are providing value.
May 15, 2025 at 6:19 PM
It feels worth conference organizers running a study to see if this significantly impacts reviewer scores.
I hope things like this are placebos, but if not we need to seriously consider whether existing peer-review processes for big ML conferences are providing value.
I hope things like this are placebos, but if not we need to seriously consider whether existing peer-review processes for big ML conferences are providing value.
Introducing CAVA: The Comprehensive Assessment for Voice Assistants
A new benchmark for evaluating the capabilities required for speech-in-speech-out voice assistants!
- Latency
- Instruction following
- Function calling
- Tone awareness
- Turn taking
- Audio Safety
TalkArena.org/cava
A new benchmark for evaluating the capabilities required for speech-in-speech-out voice assistants!
- Latency
- Instruction following
- Function calling
- Tone awareness
- Turn taking
- Audio Safety
TalkArena.org/cava
Comprehensive Assessment for Voice Assistants
CAVA is a new benchmark for assessing how well Large Audio Models support voice assistant capabilities.
TalkArena.org
May 7, 2025 at 4:15 PM
Introducing CAVA: The Comprehensive Assessment for Voice Assistants
A new benchmark for evaluating the capabilities required for speech-in-speech-out voice assistants!
- Latency
- Instruction following
- Function calling
- Tone awareness
- Turn taking
- Audio Safety
TalkArena.org/cava
A new benchmark for evaluating the capabilities required for speech-in-speech-out voice assistants!
- Latency
- Instruction following
- Function calling
- Tone awareness
- Turn taking
- Audio Safety
TalkArena.org/cava
Reposted by Will Held
How does the public conceptualize AI? Rather than self-reported measures, we use metaphors to understand the nuance and complexity of people’s mental models. In our #FAccT2025 paper, we analyzed 12,000 metaphors collected over 12 months to track shifts in public perceptions.
May 2, 2025 at 1:19 AM
How does the public conceptualize AI? Rather than self-reported measures, we use metaphors to understand the nuance and complexity of people’s mental models. In our #FAccT2025 paper, we analyzed 12,000 metaphors collected over 12 months to track shifts in public perceptions.
Reposted by Will Held
I wrote something up for AI people who want to get into bluesky and either couldn't assemble an exciting feed or gave up doomscrolling when their Following feed switched to talking politics 24/7.
The AI Researcher's Guide to a Non-Boring Bluesky Feed | Naomi Saphra
How to migrate to bsky without a boring feed.
nsaphra.net
April 26, 2025 at 1:31 AM
I wrote something up for AI people who want to get into bluesky and either couldn't assemble an exciting feed or gave up doomscrolling when their Following feed switched to talking politics 24/7.
Reposted by Will Held
Worth noting that a number of universities have now sued over withheld and canceled grants, but no university has yet sued over the arrest, detention, and threatened deportation of its foreign students. www.nytimes.com/2025/04/19/o...
Opinion | Our Foreign Students Are Terrified, and They’re Right to Be
The immigration crackdown has come to America’s campuses.
www.nytimes.com
April 21, 2025 at 12:33 AM
Worth noting that a number of universities have now sued over withheld and canceled grants, but no university has yet sued over the arrest, detention, and threatened deportation of its foreign students. www.nytimes.com/2025/04/19/o...
Reposted by Will Held
Mahmoud Khalil writes movingly about what his detention by ICE means for America: www.washingtonpost.com/opinions/202...
Opinion | Mahmoud Khalil: What does my detention by ICE say about America?
A democracy for some is no democracy at all.
www.washingtonpost.com
April 20, 2025 at 1:06 PM
Mahmoud Khalil writes movingly about what his detention by ICE means for America: www.washingtonpost.com/opinions/202...