Nari Johnson
@narijohnson.bsky.social
researching AI [evaluation, governance, accountability]
Reposted by Nari Johnson
Can public involvement in AI evaluation improve the science? Or does it compromise quality, speed, cost?
In @pnas.org, Megan Price & I summarize challenges of AI evaluation, review strengths/weaknesses, & suggest how participatory methods can improve the science of AI
www.pnas.org/doi/10.1073/...
In @pnas.org, Megan Price & I summarize challenges of AI evaluation, review strengths/weaknesses, & suggest how participatory methods can improve the science of AI
www.pnas.org/doi/10.1073/...
How public involvement can improve the science of AI | PNAS
As AI systems from decision-making algorithms to generative AI are deployed more widely,
computer scientists and social scientists alike are being ...
www.pnas.org
November 17, 2025 at 12:47 PM
Can public involvement in AI evaluation improve the science? Or does it compromise quality, speed, cost?
In @pnas.org, Megan Price & I summarize challenges of AI evaluation, review strengths/weaknesses, & suggest how participatory methods can improve the science of AI
www.pnas.org/doi/10.1073/...
In @pnas.org, Megan Price & I summarize challenges of AI evaluation, review strengths/weaknesses, & suggest how participatory methods can improve the science of AI
www.pnas.org/doi/10.1073/...
Reposted by Nari Johnson
Can LLMs accurately aggregate information over long, information-dense texts? Not yet…
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
November 7, 2025 at 5:07 PM
Can LLMs accurately aggregate information over long, information-dense texts? Not yet…
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
We introduce Oolong, a dataset of simple-to-verify information aggregation questions over long inputs. No model achieves >50% accuracy at 128K on Oolong!
Reposted by Nari Johnson
📣 Our method for conducting community-based algorithmic impact assessments is now available! We’ve just launched a new section on our website where you can find an extensive toolkit, documentation of our pilots, and a series of reflections on lessons learned. datasociety.net/research/alg...
October 29, 2025 at 7:10 PM
📣 Our method for conducting community-based algorithmic impact assessments is now available! We’ve just launched a new section on our website where you can find an extensive toolkit, documentation of our pilots, and a series of reflections on lessons learned. datasociety.net/research/alg...
Reposted by Nari Johnson
𝐒𝐨𝐜𝐢𝐞𝐭𝐚𝐥 𝐈𝐦𝐩𝐚𝐜𝐭 𝐀𝐬𝐬𝐞𝐬𝐬𝐦𝐞𝐧𝐭 𝐟𝐨𝐫 𝐈𝐧𝐝𝐮𝐬𝐭𝐫𝐲 𝐂𝐨𝐦𝐩𝐮𝐭𝐢𝐧𝐠 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬
🏅 Best Paper Honorable Mention (Top 3% Submissions)
🔗 dl.acm.org/doi/10.1145/...
📆 Wed, 22 Oct | 9:00 AM, CET: Toward More Ethical and Transparent Systems and Environments
🏅 Best Paper Honorable Mention (Top 3% Submissions)
🔗 dl.acm.org/doi/10.1145/...
📆 Wed, 22 Oct | 9:00 AM, CET: Toward More Ethical and Transparent Systems and Environments
Supporting Industry Computing Researchers in Assessing, Articulating, and Addressing the Potential Negative Societal Impact of Their Work | Proceedings of the ACM on Human-Computer Interaction
Recent years have witnessed increasing calls for computing researchers to grapple with the societal impacts of their work. Tools such as impact assessments have gained prominence as a method to uncover potential impacts, and a number of publication ...
dl.acm.org
October 19, 2025 at 1:49 PM
𝐒𝐨𝐜𝐢𝐞𝐭𝐚𝐥 𝐈𝐦𝐩𝐚𝐜𝐭 𝐀𝐬𝐬𝐞𝐬𝐬𝐦𝐞𝐧𝐭 𝐟𝐨𝐫 𝐈𝐧𝐝𝐮𝐬𝐭𝐫𝐲 𝐂𝐨𝐦𝐩𝐮𝐭𝐢𝐧𝐠 𝐑𝐞𝐬𝐞𝐚𝐫𝐜𝐡𝐞𝐫𝐬
🏅 Best Paper Honorable Mention (Top 3% Submissions)
🔗 dl.acm.org/doi/10.1145/...
📆 Wed, 22 Oct | 9:00 AM, CET: Toward More Ethical and Transparent Systems and Environments
🏅 Best Paper Honorable Mention (Top 3% Submissions)
🔗 dl.acm.org/doi/10.1145/...
📆 Wed, 22 Oct | 9:00 AM, CET: Toward More Ethical and Transparent Systems and Environments
Reposted by Nari Johnson
💡Can we trust synthetic data for statistical inference?
We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data
We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data
October 10, 2025 at 4:12 PM
💡Can we trust synthetic data for statistical inference?
We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data
We show that synthetic data (e.g., LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moment residuals of synthetic data and those of real data
Reposted by Nari Johnson
Our Responsible AI team at Apple is looking for spring/summer 2026 PhD research interns! Please apply at jobs.apple.com/en-us/detail... and email [email protected]. Do not send extra info (e.g., CV), just drop us a line so we can find your application in the central pool!
Machine Learning / AI Internships - Jobs - Careers at Apple
Apply for a Machine Learning / AI Internships job at Apple. Read about the role and find out if it’s right for you.
jobs.apple.com
October 10, 2025 at 2:28 AM
Our Responsible AI team at Apple is looking for spring/summer 2026 PhD research interns! Please apply at jobs.apple.com/en-us/detail... and email [email protected]. Do not send extra info (e.g., CV), just drop us a line so we can find your application in the central pool!
Reposted by Nari Johnson
✨I’m on the academic job market ✨
I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻💻💪🏽💥
I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action
More info: cella.io
I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻💻💪🏽💥
I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action
More info: cella.io
Cella M. Sum –
cella.io
October 9, 2025 at 2:39 PM
✨I’m on the academic job market ✨
I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻💻💪🏽💥
I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action
More info: cella.io
I’m a PhD candidate at @hcii.cmu.edu studying tech, labor, and resistance 👩🏻💻💪🏽💥
I research how workers and communities contest harmful sociotechnical systems and shape alternative futures through everyday resistance and collective action
More info: cella.io
Reposted by Nari Johnson
🌟 If you’re applying to CMU SCS PhD programs, and come from a background that would bring additional dimensions to the CMU community, our PhD students are here to help!
Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:
Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:
September 24, 2025 at 4:00 PM
🌟 If you’re applying to CMU SCS PhD programs, and come from a background that would bring additional dimensions to the CMU community, our PhD students are here to help!
Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:
Apply to the Graduate Applicant Support Program by Oct 13 to receive feedback on your application materials:
Reposted by Nari Johnson
📌📌📌
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com
Stephen Casper
Visit the post for more.
stephencasper.com
September 4, 2025 at 3:39 AM
📌📌📌
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com
I'm excited to be on the faculty job market this fall. I just updated my website with my CV.
stephencasper.com
Reposted by Nari Johnson
📢2026 Fellowship applications are OPEN!📢
If you are someone looking to inform technology policy through rigorous original reporting or policy analyses, we want to hear from you!
Apply here: airtable.com/appIrc1F9M5d...
If you are someone looking to inform technology policy through rigorous original reporting or policy analyses, we want to hear from you!
Apply here: airtable.com/appIrc1F9M5d...
September 4, 2025 at 11:47 AM
📢2026 Fellowship applications are OPEN!📢
If you are someone looking to inform technology policy through rigorous original reporting or policy analyses, we want to hear from you!
Apply here: airtable.com/appIrc1F9M5d...
If you are someone looking to inform technology policy through rigorous original reporting or policy analyses, we want to hear from you!
Apply here: airtable.com/appIrc1F9M5d...
Reposted by Nari Johnson
What can #CSCW learn from tech workers who have been involved in collective action and unionization about how to make transformative change within our field?
My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question.
Pre-print: arxiv.org/pdf/2508.12579
My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question.
Pre-print: arxiv.org/pdf/2508.12579
August 28, 2025 at 2:14 PM
What can #CSCW learn from tech workers who have been involved in collective action and unionization about how to make transformative change within our field?
My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question.
Pre-print: arxiv.org/pdf/2508.12579
My new #CSCW2025 paper with Mona Wang, Anna Konvicka, and Sarah Fox seeks to answer this question.
Pre-print: arxiv.org/pdf/2508.12579
Reposted by Nari Johnson
The exchanges between Adam and ChatGPT are devastating. This, in my mind, is the worst one.
One of his last messages was a photo of the noose hung in his bedroom closet, asking if it was "good." ChatGPT offered a technical analysis of the set up and told him it 'could potentially suspend a human."
One of his last messages was a photo of the noose hung in his bedroom closet, asking if it was "good." ChatGPT offered a technical analysis of the set up and told him it 'could potentially suspend a human."
August 26, 2025 at 1:37 PM
The exchanges between Adam and ChatGPT are devastating. This, in my mind, is the worst one.
One of his last messages was a photo of the noose hung in his bedroom closet, asking if it was "good." ChatGPT offered a technical analysis of the set up and told him it 'could potentially suspend a human."
One of his last messages was a photo of the noose hung in his bedroom closet, asking if it was "good." ChatGPT offered a technical analysis of the set up and told him it 'could potentially suspend a human."
Reposted by Nari Johnson
Adam Raine, 16, died from suicide in April after months on ChatGPT discussing plans to end his life. His parents have filed the first known case against OpenAI for wrongful death.
Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...
Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...
A Teen Was Suicidal. ChatGPT Was the Friend He Confided In.
www.nytimes.com
August 26, 2025 at 1:01 PM
Adam Raine, 16, died from suicide in April after months on ChatGPT discussing plans to end his life. His parents have filed the first known case against OpenAI for wrongful death.
Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...
Overwhelming at times to work on this story, but here it is. My latest on AI chatbots: www.nytimes.com/2025/08/26/t...
Reposted by Nari Johnson
A cognitively impaired New Jersey man grew infatuated with a Meta chatbot originally created in partnership with celebrity influencer Kendall Jenner. His fatal attraction shines a light on Meta's guidelines for its AI chatbots reut.rs/45DQIRj
@jeffhorwitz.bsky.social
@jeffhorwitz.bsky.social
August 14, 2025 at 11:16 AM
A cognitively impaired New Jersey man grew infatuated with a Meta chatbot originally created in partnership with celebrity influencer Kendall Jenner. His fatal attraction shines a light on Meta's guidelines for its AI chatbots reut.rs/45DQIRj
@jeffhorwitz.bsky.social
@jeffhorwitz.bsky.social
Reposted by Nari Johnson
🧵 New paper from UK AISI x @eleutherai.bsky.social rai.bsky.social that I led with @kyletokens.bsky.social y.social���:
Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.
Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.
August 12, 2025 at 11:45 AM
🧵 New paper from UK AISI x @eleutherai.bsky.social rai.bsky.social that I led with @kyletokens.bsky.social y.social���:
Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.
Open-weight LLM safety is both important & neglected. But filtering dual-use knowledge from pre-training data improves tamper resistance *>10x* over post-training baselines.
Reposted by Nari Johnson
* STS folks! * CMU is hiring up to 2 tenure track faculty focused on: the intersection of tech & social change, the environmental and social impacts of science, tech, and medicine. They will be housed in History, a department of both historians and anthropologists.
apply.interfolio.com/170040
apply.interfolio.com/170040
July 23, 2025 at 4:47 PM
* STS folks! * CMU is hiring up to 2 tenure track faculty focused on: the intersection of tech & social change, the environmental and social impacts of science, tech, and medicine. They will be housed in History, a department of both historians and anthropologists.
apply.interfolio.com/170040
apply.interfolio.com/170040
Reposted by Nari Johnson
One of the largest text-image datasets is full of PII, including credit card numbers and birth certificates. Excellent writeup by @eileenguo.bsky.social www.technologyreview.com/2025/07/18/1... Read our full audit and legal analysis arxiv.org/pdf/2506.17185
A major AI training data set contains millions of examples of personal data
Personally identifiable information has been found in DataComp CommonPool, one of the largest open-source data sets used to train image generation models.
www.technologyreview.com
July 19, 2025 at 12:02 AM
One of the largest text-image datasets is full of PII, including credit card numbers and birth certificates. Excellent writeup by @eileenguo.bsky.social www.technologyreview.com/2025/07/18/1... Read our full audit and legal analysis arxiv.org/pdf/2506.17185
Reposted by Nari Johnson
If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...
ICML Poster Position: Evaluating Generative AI Systems Is a Social Science Measurement ChallengeICML 2025
icml.cc
July 15, 2025 at 6:35 PM
If you're at @icmlconf.bsky.social this week, come check out our poster on "Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge" presented by the amazing @afedercooper.bsky.social from 11:30am--1:30pm PDT on Weds!!! icml.cc/virtual/2025...
Reposted by Nari Johnson
🏦 Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees’ Practices, Challenges, and Needs by @narijohnson.bsky.social et al. explores procurement in the context of recent calls for governments to use their "purchasing power" to incentivize responsible AI.
July 15, 2025 at 4:31 PM
🏦 Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees’ Practices, Challenges, and Needs by @narijohnson.bsky.social et al. explores procurement in the context of recent calls for governments to use their "purchasing power" to incentivize responsible AI.
Reposted by Nari Johnson
I did an interview w/ Pittsburgh's NPR station to share some of my views on the topic of the McCormick/Trump AI & Energy summit at CMU tomorrow. Despite being hosted at the university, there will not be opportunities for our university experts to contribute viewpoints at the event.
President Donald Trump travels to Carnegie Mellon University Tuesday for a summit on energy and artificial intelligence. Leaders say Western Pennsylvania's universities and natural-gas deposits could be vital to both industries. But researchers are concerned about AI's energy demands.
With Trump set to attend AI & energy summit, CMU professor worries climate issues will be lost
Carnegie Mellon University professor Emma Strubell says that while AI is promising, the threat of climate change "does keep me up at night a lot"
www.wesa.fm
July 14, 2025 at 3:49 PM
I did an interview w/ Pittsburgh's NPR station to share some of my views on the topic of the McCormick/Trump AI & Energy summit at CMU tomorrow. Despite being hosted at the university, there will not be opportunities for our university experts to contribute viewpoints at the event.
New article out today, covering our past two years of research asking US cities how they govern AI ✨
The future of AI governance in public services is being shaped right now, through public procurement
The future of AI governance in public services is being shaped right now, through public procurement
In the absence of federal regulation of AI vendors, procurement remains one of the few levers governments have to push for public values, such as safety, non-discrimination, privacy, and accountability, Nari Johnson, Elise Silva, and Hoda Heidari write.
Want Accountable AI in Government? Start with Procurement | TechPolicy.Press
Procurement plays a powerful role in shaping critical decisions about artificial intelligence, Nari Johnson, Elise Silva, and Hoda Heidari write.
www.techpolicy.press
July 15, 2025 at 3:50 PM
New article out today, covering our past two years of research asking US cities how they govern AI ✨
The future of AI governance in public services is being shaped right now, through public procurement
The future of AI governance in public services is being shaped right now, through public procurement
Reposted by Nari Johnson
In the absence of federal regulation of AI vendors, procurement remains one of the few levers governments have to push for public values, such as safety, non-discrimination, privacy, and accountability, Nari Johnson, Elise Silva, and Hoda Heidari write.
Want Accountable AI in Government? Start with Procurement | TechPolicy.Press
Procurement plays a powerful role in shaping critical decisions about artificial intelligence, Nari Johnson, Elise Silva, and Hoda Heidari write.
www.techpolicy.press
July 15, 2025 at 2:18 PM
In the absence of federal regulation of AI vendors, procurement remains one of the few levers governments have to push for public values, such as safety, non-discrimination, privacy, and accountability, Nari Johnson, Elise Silva, and Hoda Heidari write.
Reposted by Nari Johnson
FAccT Day 4 word of the day was impasse. As a field, AI ethics is at a critical moment in which our "big tent" and pluralistic research directions can either complement or cancel each other out. Molly Crockett gave an amazing final keynote calling out genAI-human performance research.
July 2, 2025 at 6:39 PM
FAccT Day 4 word of the day was impasse. As a field, AI ethics is at a critical moment in which our "big tent" and pluralistic research directions can either complement or cancel each other out. Molly Crockett gave an amazing final keynote calling out genAI-human performance research.
Reposted by Nari Johnson
I am so excited to be in 🇬🇷Athens🇬🇷 to present "A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms" by me, @kizilcec.bsky.social, and @allisonkoe.bsky.social, at #FAccT2025!!
🔗: arxiv.org/pdf/2506.04419
🔗: arxiv.org/pdf/2506.04419
June 23, 2025 at 2:45 PM
I am so excited to be in 🇬🇷Athens🇬🇷 to present "A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms" by me, @kizilcec.bsky.social, and @allisonkoe.bsky.social, at #FAccT2025!!
🔗: arxiv.org/pdf/2506.04419
🔗: arxiv.org/pdf/2506.04419
Reposted by Nari Johnson
🏆 Announcing the #FAccT2025 best paper awards! 🏆
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...
Announcing Best Paper Awards
The Best Paper Award Committee was chaired this year by Alex Chouldechova and included six Area Chairs. The committee selected three papers for the Best Paper Award and recognized three additional pap...
facct-blog.github.io
June 20, 2025 at 9:14 PM
🏆 Announcing the #FAccT2025 best paper awards! 🏆
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...
Congratulations to all the authors of the three best papers and three honorable mention papers.
Be sure to check out their presentations at the conference next week!
facct-blog.github.io/2025-06-20/b...