Lightnews — Scholar-powered news

Jamie Cummins

@jamiecummins.bsky.social

2.7K followers 660 following 870 posts

Currently a visiting researcher at Uni of Oxford. Normally at Uni of Bern. Meta-scientist building tools to help other scientists. NLP, simulation, & LLMs. Creator and developer of RegCheck (https://regcheck.app). 1/4 of @error.reviews. 🇮🇪

regcheck.app).

Posts Media Videos Starter Packs

Pinned

Jamie Cummins @jamiecummins.bsky.social · Jul 23

Introducing RegCheck: a tool which uses Large Language Models to automatically compare preregistered protocols with their corresponding published papers and highlights deviations.

@malte.the100.ci @ianhussey.bsky.social @ruben.the100.ci @bjoernhommel.bsky.social

regcheck.app

RegCheck.app

RegCheck is an AI tool to compare preregistrations with papers instantly.

regcheck.app

5 51 100

Reposted by Jamie Cummins

Lisa DeBruine @debruine.bsky.social · 2h

Please help us, #MetaScience community!

It's time to decide on a forever name for papercheck (scienceverse.github.io/papercheck/). We don't want it to be confused with papercheck.ai, and we plan to check other research artifacts like repo contents, data, code, and prereg. Any suggestions?

Check Scientific Papers for Best Practices

A modular, extendable system for automatically checking scientific papers for best practices using text search, R code, and/or (optional) LLM queries.

scienceverse.github.io

2 7 6

Jamie Cummins @jamiecummins.bsky.social · 23h

Answer coming soon...

Jamie Cummins @jamiecummins.bsky.social · 1d

glad they're taking it as seriously as we are

1 5

Reposted by Jamie Cummins

Ian Hussey @ianhussey.mmmdata.io · 1d

Make an effect size prediction!

@jamiecummins.bsky.social and I are replicating Balcetis & Dunning's (2010) "chocolate is more desirable than poop" (Cohen's d = 4.52)

Let us known in the replies what effect size you think we'll find. Details of the study in the thread below.

17 20 54

Jamie Cummins @jamiecummins.bsky.social · 1d

There are tears in my eyes!

Reposted by Jamie Cummins

Jack Wilkinson @jdwilko.bsky.social · 11d

Introductory online INSPECT-SR workshop. November 6th, 12-2pm UK-time. Free, places limited. BOOK: www.trybooking.com/uk/events/la...

Introduction to INSPECT-SR Training Workshop November

An introductory 2-hour online workshop will introduce participants to the INSPECT-SR tool for assessing trustworthiness of randomised controlled...

www.trybooking.com

1 6 5

Jamie Cummins @jamiecummins.bsky.social · 7d

thanks for the mention! 😊

Reposted by Jamie Cummins

Crystal Lewis @cghlewis.bsky.social · 7d

Issue 16 of RDM Weekly is out! 📬

It includes:
- Data is Not Available Upon Request @ianhussey.mmmdata.io
- AI Generated Participants in Social Science @jamiecummins.bsky.social @science.org
- Why’s it Hard to Teach Data Cleaning? @randyau.com
and more!

rdmweekly.substack.com/p/rdm-weekly...

RDM Weekly - Issue 016

A weekly roundup of Research Data Management resources.

rdmweekly.substack.com

2 8 19

Reposted by Jamie Cummins

Beth Popp Berman @epopppp.bsky.social · 10d

Interesting article/paper.

I'm much less anti-AI than a lot of people on my feed. But pretty skeptical it can simulate human behavior effectively for social scientific purposes -- at least in cases where variation among humans, rather than acting like an average human, is what's important.

AI-generated ‘participants’ can lead social science experiments astray, study finds

Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings

www.science.org

1 6 37

Jamie Cummins @jamiecummins.bsky.social · 9d

WCL winners, they’ll never sing that 😉

Reposted by Jamie Cummins

Ian Hussey @ianhussey.mmmdata.io · 10d

My article "Data is not available upon request" was published in Meta-Psychology. Very happy to see this out!
open.lnu.se/index.php/me...

LnuOpen | Meta-Psychology

open.lnu.se

5 44 110

Jamie Cummins @jamiecummins.bsky.social · 11d

I’ve seen @malte.the100.ci recently using one that looked very cool

1 1

Jamie Cummins @jamiecummins.bsky.social · 12d

OMG I can’t wait to listen!

1 3

Reposted by Jamie Cummins

Saloni @scientificdiscovery.dev · 12d

New episode of HARD DRUGS!

AlphaFold, ProteinMPNN & other AI tools are transforming biology and drug design.

But how do they work? What can’t they do? And can we use them to make a vaccine against Strep A for the very first time?

In this episode, Jacob and I talk about hacking proteins with AI.

Hacking proteins with AI

open.spotify.com

3 9 36

Reposted by Jamie Cummins

Benjamin Paaßen @bpaassen.bsky.social · 13d

Our work on simulated participants also makes a small appearance arxiv.org/abs/2508.06950

Large Language Models Do Not Simulate Human Psychology

Large Language Models (LLMs),such as ChatGPT, are increasingly used in research, ranging from simple writing assistance to complex data annotation tasks. Recently, some research has suggested that LLM...

arxiv.org

1 3

Reposted by Jamie Cummins

Benjamin Paaßen @bpaassen.bsky.social · 13d

@cathleenogrady.bsky.social has just published the story "AI-generated ‘participants’ can lead social science experiments astray, study finds" for Science. It is, once more, a reason to be careful when relying on LLM-generated data in empirical research. www.science.org/content/arti...

AI-generated ‘participants’ can lead social science experiments astray, study finds

Data produced by “silicon samples” depends on researchers’ exact choice of models, prompts, and settings

www.science.org

1 1 4

Jamie Cummins @jamiecummins.bsky.social · 12d

Looking forward to reading this, and I’m glad you’ve written it!

Reposted by Jamie Cummins

Chris Chapman @cchapman.bsky.social · 13d

Excellent 🧵 about LLM synthetic data (silicon samples etc) and why they don't solve any particular problem in human research.

FWIW, in addition to results and considerations like these, I've argued elsewhere that the entire question is ill-formed: quantuxblog.com/synthetic-su...

2 2 5

Jamie Cummins @jamiecummins.bsky.social · 13d

There isn't really a fixed term tbh, people use a few different ones depending on field/domain/preference. Silicon samples seems to be the most common but there are a bunch of others, like synthetic samples/synthetic participants/etc.

Jamie Cummins @jamiecummins.bsky.social · 13d

Couldn't agree more.

Jamie Cummins @jamiecummins.bsky.social · 13d

Clearly I missed my true career-calling as a diplomat lol

Jamie Cummins @jamiecummins.bsky.social · 13d

OMG. Did not catch this one during my lit review. Wow.

Jamie Cummins @jamiecummins.bsky.social · 13d

Starting to feel eerily like Severance....

1 1

Jamie Cummins @jamiecummins.bsky.social · 13d

that should have been my full abstract!

Reposted by Jamie Cummins

Lora Kolodny @lorak.bsky.social · 13d

👀 studying real humans better for understanding humans than not

Jamie Cummins @jamiecummins.bsky.social · 13d

@science.org just dropped a story covering this preprint! Check it out below, and thanks to @cathleenogrady.bsky.social for the great write-up! www.science.org/content/arti...

2 4 23