Hanna Wallach
banner
hannawallach.bsky.social
Hanna Wallach
@hannawallach.bsky.social
VP and Distinguished Scientist at Microsoft Research NYC. AI evaluation and measurement, responsible AI, computational social science, machine learning. She/her.

One photo a day since January 2018: https://www.instagram.com/logisticaggression/
And I'll be at @wimlworkshop.bsky.social and @neuripsconf.bsky.social Tue, Wed, and Thu next week if folks want to chat.
November 24, 2025 at 3:33 PM
For all three positions, we welcome applicants with backgrounds in technical fields (e.g., ML, AI, NLP, statistics) and sociotechnical fields (e.g., human--computer interaction, information science, law, media studies, philosophy, science and technology studies, sociology).
November 24, 2025 at 3:33 PM
And you can read more about STAC here: microsoft.com/en-us/resear...
November 24, 2025 at 3:33 PM
You can read more about FATE here: microsoft.com/en-us/resear...
November 24, 2025 at 3:33 PM
Reposted by Hanna Wallach
1) (Tomorrow!) Wed 7/16, 11am-1:30 pm PT poster for "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" (E. Exhibition Hall A-B, E-503)

Work led by @hannawallach.bsky.social + @azjacobs.bsky.social

arxiv.org/abs/2502.00561
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges com...
arxiv.org
July 16, 2025 at 12:46 AM
Oh whoops! You are indeed correct -- it starts at 11am PT!
July 15, 2025 at 8:34 PM
I also want to note that this paper has been in progress for many, many years, so we're super excited it's finally being published. It's also one of the most genuinely interdisciplinary projects I've ever worked on, which has made it particularly challenging and rewarding!!! ❤️
June 16, 2025 at 9:49 PM
Check out the camera-ready version of our ACL Findings paper ("Taxonomizing Representational Harms using Speech Act Theory") to learn more!!! arxiv.org/pdf/2504.00928
arxiv.org
June 16, 2025 at 9:49 PM
Why does this matter? You can't mitigate what you can't measure, and our framework and taxonomy help researchers and practitioners design better ways to measure and mitigate representational harms caused by generative language systems.
June 16, 2025 at 9:49 PM
Using this theoretical grounding, we provide new definitions for stereotyping, demeaning, and erasure, and break them down into a detailed taxonomy of system behaviors. By doing this, we unify many of the different ways representational harms have been previously defined.
June 16, 2025 at 9:49 PM
We bring some much-needed clarity by turning to speech act theory—a theory of meaning from linguistics that allows us to distinguish between a system output’s purpose and its real-world impacts.
June 16, 2025 at 9:49 PM
These are often called “representational harms,” and while they’re easy for people to recognize when they see them, definitions of these harms are commonly under-specified, leading to conceptual confusion. This makes them hard to measure and even harder to mitigate.
June 16, 2025 at 9:49 PM
Check out the camera-ready version of our ICML position paper ("Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge") to learn more!!! arxiv.org/abs/2502.00561

(6/6)
Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges com...
arxiv.org
June 15, 2025 at 12:20 AM
Real talk: GenAI systems aren't toys. Bad evaluations don't just waste people's time---they can cause real-world harms. It's time to level up, ditch the apples-to-oranges comparisons, and start doing measurement like we mean it.

(5/6)
June 15, 2025 at 12:20 AM
We propose a framework that cuts through the chaos: first, get crystal clear on what you're measuring and why (no more vague hand-waving); then, figure out how to measure it; and, throughout the process, interrogate validity like your reputation depends on it---because, honestly, it should.

(4/6)
June 15, 2025 at 12:20 AM
Here's our hot take: evaluating GenAI systems isn't just some techie puzzle---it's a social science measurement challenge.

(3/6)
June 15, 2025 at 12:20 AM
But there's a dirty little secret: the ways we evaluate GenAI systems are often sloppy, vague, and quite frankly... not up to the task.

(2/6)
June 15, 2025 at 12:20 AM
Please spread the word to anyone who you think might be interested! We will begin reviewing applications on June 2.
May 20, 2025 at 1:47 PM