Lightnews — Scholar-powered news

Joe Bak-Coleman @jbakcoleman.bsky.social · 2h

Study funded by meta

Kevin M. Kruse @kevinmkruse.bsky.social · 4h

In honor of spooky month, share a 4 word horror story that only someone in your profession would understand:

Archives banned making copies.

Natasha Simonova @philistella.bsky.social · 4h

In honor of spooky month, share a 4 word horror story that only someone in your profession would understand:

New library catalogue website.

1 18

Joe Bak-Coleman @jbakcoleman.bsky.social · 9h

This is almost certainly just picking up on ai slop proliferating. Check figure 6.

1 2

Joe Bak-Coleman @jbakcoleman.bsky.social · 10h

My guess is if they restricted this to reputable journals their effect would evaporate. It’s kind of wild to see a finding clearly about the proliferation of slop recast as productivity.

1

Joe Bak-Coleman @jbakcoleman.bsky.social · 10h

So the effect seems to, almost certainly, be driven by metric benefits of paying for paper mills and I see nothing pointing to this being applicable to earnest science.

1 1

Joe Bak-Coleman @jbakcoleman.bsky.social · 10h

But even if they identify a causal effect, I don’t think they can attribute it to the story they’re telling. In figure 6 when they restrict to the most extreme users the effect size grows.

These are papers that are most likely to contain the most ai words, and we know paper mills push slop now.

1 1

Joe Bak-Coleman @jbakcoleman.bsky.social · 10h

Some discussion here

bsky.app/profile/jben...

jon ben-menachem @jbenmenachem.com · 23h

Given that there is probably variation in treatment uptake, I do wonder whether there's a case for using the Callaway estimator here. It's also just kind of odd to see a DiD paper posted without an obvious parallel trends diagnostic? They say it's Table 5 but that's... not the right diagnostic?

2 1

Joe Bak-Coleman @jbakcoleman.bsky.social · 10h

Yes, and I served as a consulting expert and contributor to the UNDP’s flagship human development report which focused on the benefits and risks of AI.

hdr.undp.org/content/huma...

Human Development Report 2025 | Human Development Reports

The 2025 Human Development Report explores the implications of artificial intelligence for human development and the choices we can make to ensure that it enhances human capabilities. Rather than atte...

hdr.undp.org

1 1

Reposted by Joe Bak-Coleman

Carl T. Bergstrom @carlbergstrom.com · 1d

Fustiliarian Friday: Trump/Kennedy/GOP use the shutdown as an excuse to destroy the CDC’s elite field epidemiology team and one of the most important epidemiology journals in the world.

Gift link.

Trump Administration Lays Off Dozens of C.D.C. Officials

www.nytimes.com

4 86 150

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

Science Taylorism strikes again.

1

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

Yeah, especially given figure 6 uptake is almost certainly driven by “oh shit I need to buy a paper”

2

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

I certainly believe folks could rip out more papers but to the extent that science (and any collective professional) thrives on diversity of experience and perspective it’s hard to see how driving us towards the most likely translates to scientific productivity broadly.

1

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

Yah every inch of the paper is a causal pitfall. Especially when you look at their model ran on unmatched data. Wild.

1 3

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

If it were helping with translation or whatnot we wouldn’t expect this because the language of the first draft is pinned down by the original text before assistance.

The quality indicator is equally explained by this as well, just a little less intuitively.

Joe Bak-Coleman @jbakcoleman.bsky.social · 23h

Figure 6, the effect sizes dwindle as you move away from the top single digit percentile of most ai heavy users. As we know some papers are *entirely* ai slop, these stricter criteria can be viewed as containing a higher abundance of slop.

In other words, these gains are happening for ai slop.

2 1 5

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

The gutting of CDC talent happening right now is just fucking scary.

9

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

I just got hit with a reviewer request for ai slop from a nature journal (and one of the exclusive ones..) so I’d wager a lot of this is just slop proliferating.

1 2

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

Bespoke: paying a paper mill to write your ChatGPT paper

1

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

Quality here is measured by impact factor, which…

My guess is the signal being picked up on is ai slop proliferation in the shoddy self-citing corner of scam journals.

1 7

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

Broke: ai slop infiltrating journals and citing itself
Woke: lots of high impact papers

1 1

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

I love the author didn’t even consider paper mills.

1 2

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

Betteridge’s law of headlines is strong with this one.

arxiv.org/abs/2510.02408

Can GenAI Improve Academic Performance? Evidence from the Social and Behavioral Sciences

This paper estimates the effect of Generative AI (GenAI) adoption on scientific productivity and quality in the social and behavioral sciences. Using matched author-level panel data and a difference-i...

arxiv.org

2 2 13

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

If you have theory there’s a lot less poking in the dark then it seems. You can intuit the functional forms from first principles and the distributions to be expected. A bit of algebra and statistical theory goes a long way towards a better model.

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

Something we don’t discuss enough are the trade offs between getting it wrong because the model/theory is shit and getting it wrong because of wishful modification to the model or data. The modal errors I see in papers are the former these days.

1 2

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

You go back and revise and the model fits like a glove to the bulk properties of the data across groups. Your decision isn’t conditioned on the difference in means, so you’re not coaxing your inferences but you’re working to avoid drawing the wrong inference from bad model fit.

1

Joe Bak-Coleman @jbakcoleman.bsky.social · 1d

As a simple example, say you’re interested in means and originally assume normal distributions for both groups. Your simulations come back fine but then you look at data simulated from the posterior and notice it’s extremely skewed by comparison. You adjust your model to allow for skew.

1