Megan Brown
@m-dot-brown.bsky.social
PhD Student University of Michigan School of Information & Michigan Institute for Computational Discovery and Engineering | formerly engineering @csmapnyu.org | NYU ‘19
Grateful for the collaboration with excellent researchers for this paper @orangechair.org, Gabe Maldoff, @solmg.bsky.social, @zevesanderson.com, & @michaelzimmer.bsky.social
November 20, 2025 at 3:19 PM
Grateful for the collaboration with excellent researchers for this paper @orangechair.org, Gabe Maldoff, @solmg.bsky.social, @zevesanderson.com, & @michaelzimmer.bsky.social
7/ 📄 You can find our full paper here: arxiv.org/abs/2503.23243
w/ Shubham Atreja, @libbyh.bsky.social, & @patrickwu.bsky.social
w/ Shubham Atreja, @libbyh.bsky.social, & @patrickwu.bsky.social
Evaluating how LLM annotations represent diverse views on contentious topics
Researchers have proposed the use of generative large language models (LLMs) to label data for research and applied settings. This literature emphasizes the improved performance of these models relati...
arxiv.org
July 11, 2025 at 4:44 PM
7/ 📄 You can find our full paper here: arxiv.org/abs/2503.23243
w/ Shubham Atreja, @libbyh.bsky.social, & @patrickwu.bsky.social
w/ Shubham Atreja, @libbyh.bsky.social, & @patrickwu.bsky.social
6/ This has implications for both research and industry:
⚠️ Fairness evaluations should be context-specific
🤔 Model choice alone will not solve bias
🔍 Human disagreements are part of the complexity—not a flaw
⚠️ Fairness evaluations should be context-specific
🤔 Model choice alone will not solve bias
🔍 Human disagreements are part of the complexity—not a flaw
July 11, 2025 at 4:44 PM
6/ This has implications for both research and industry:
⚠️ Fairness evaluations should be context-specific
🤔 Model choice alone will not solve bias
🔍 Human disagreements are part of the complexity—not a flaw
⚠️ Fairness evaluations should be context-specific
🤔 Model choice alone will not solve bias
🔍 Human disagreements are part of the complexity—not a flaw
5/ We also find that the difficulty of the labeling task is most predictive of LLM agreement with human annotators.
July 11, 2025 at 4:44 PM
5/ We also find that the difficulty of the labeling task is most predictive of LLM agreement with human annotators.
4/ Key finding: While LLMs disagree with human annotators on the basis of demographics, it tends to be in the same directions on the same demographic categories within the same dataset. In other words, the direction of bias is not LLM-specific, but dataset-specific.
July 11, 2025 at 4:44 PM
4/ Key finding: While LLMs disagree with human annotators on the basis of demographics, it tends to be in the same directions on the same demographic categories within the same dataset. In other words, the direction of bias is not LLM-specific, but dataset-specific.
3/ This study evaluates LLM annotations across 4 datasets and tasks, analyzing whether these models disproportionately reflect majority group opinions.
July 11, 2025 at 4:44 PM
3/ This study evaluates LLM annotations across 4 datasets and tasks, analyzing whether these models disproportionately reflect majority group opinions.
2/ Prior research praises LLMs for their high accuracy, precision, recall, and F1 scores in labeling tasks—but also raises concerns about bias, especially around sensitive or polarizing content (e.g., toxicity).
July 11, 2025 at 4:44 PM
2/ Prior research praises LLMs for their high accuracy, precision, recall, and F1 scores in labeling tasks—but also raises concerns about bias, especially around sensitive or polarizing content (e.g., toxicity).
Reposted by Megan Brown
Special thanks to Mango Brown and Taylor Swift's 'Mr. Perfectly Fine' for their help in getting this paper over the finish line
March 17, 2025 at 3:30 PM
Special thanks to Mango Brown and Taylor Swift's 'Mr. Perfectly Fine' for their help in getting this paper over the finish line
Reposted by Megan Brown
I’ll keep working with academics, civil society researchers, and journalists—including via the Coalition for Independent Technology Research—to continue these important accountability efforts. 12/ (independenttechresearch.org/introducing-...)
Introducing the DSA Data Access Audit - Coalition for Independent Technology Research
In a year of more than 70 global elections, the need for independent researchers to have access to social media data has never been greater. That’s why the Coalition for Independent Technology Researc...
independenttechresearch.org
March 14, 2024 at 1:05 PM
I’ll keep working with academics, civil society researchers, and journalists—including via the Coalition for Independent Technology Research—to continue these important accountability efforts. 12/ (independenttechresearch.org/introducing-...)