So how does that work with regular meta analyses where follow up queries to original authors are common. I wouldn't be allowed to use any information obtained via email then?

Shocking, to be honest! We acknowledge of course that researchers may be forced to comply with such policies, and hope that perhaps they may evolve over time.

No need to publish the contents of the correspondence or quotes from the emails

But you don't collect data on them! Or rather, the data you would publish is a Yes/No to the question "Are the data in fact available on request as promised?", which is not data about the authors.

With a spectacular team: @ianhussey.mmmdata.io @bethclarke.bsky.social @sabrinanorwood.bsky.social @grinschglsandra.bsky.social @ruben.the100.ci @jdwilko.bsky.social @mjpages.bsky.social @ioanaacristea.bsky.social @pimcuijpers.bsky.social @simine.com @ndevito1.bsky.social @jamiecummins.bsky.social

The alternative future is that meta-science builds community norms that embrace critical assessment. Reforms motivated by meta-science become easier to evaluate and easier to justify when the evidence is robust – but also easier to discard when reforms are counterproductive for the greater good

In closing, we think there are two possible futures for meta-science: One where meta-science risks eroding the very basis on which it claims authority by being unverifiable. Reforms will either proceed without meta-science’s guidance, or it will be treated as just another opinionated subfield.

From this follow simple recommendations: as a default, meta-scientific studies of published research artefacts need to include 1) a full, identifiable list of included studies, 2) the full coding instrument and decision rules, and 3) the individual ratings together with a codebook.

Personal priorities come at epistemic costs, there is confusion about what (rather than whom) is being assessed, it's incoherent with other forms of critique, sets a dangerous precedent, and there is undeniably hypocrisy in asking for greater transparency, but not returning it in-kind.

We acknowledge there are reasons for anonymising, such as fear of reputational harm (for the meta-researcher and others), fairness, peer review pressure, or policy compliance. We argue that the damaging long-term consequences of this practice outweigh these concerns, and offer counterarguments:

As a widespread practice, there are substantial consequences to consider: No verification, no robustness checks, no way to expand research, missed opportunities for learning and training, and building a culture where errors are taboo.

Meta-scientific studies, too, often involve quality ratings (e.g., reporting, compliance, trustworthiness) of public research artefacts (e.g. articles, preregs). Are these evaluations more sensitive? We don't think so, and yet, we increasingly observed that such data are anonymised before sharing.

But even in Psychological Bulletin, quality assessments were done in 59/100 meta-analyses, and of those 50 had made their data available (47 included study identifiers). As such, /where/ quality ratings are shared, these typically include identifiers.

1) All major guidelines on research synthesis (Cochrane, MARS) recommend that included studies are rated on quality, and that these should be transparent. We examine compliance in 100 reviews in Cochrane and Psychological Bulletin. Unsurprisingly, 100% of Cochrane reviews comply.