Regardless, this issue does pose problems for the usefulness of RV. I discuss this some in my PhD thesis discussion. See figure 6.3.C for a similar example: pure.tue.nl/ws/portalfil...

Test severity may at once improve the ability of a replication study to reduce uncertainty, but it can also leave less uncertainty to be reduced to begin with.

”paradox: test severity seems to imply both increased and reduced uncertainty reduction, depending on how you think about it”
I think this is not a paradox per se. Rather, I think test severity can have two causal effects that work against each other.

I think it is possible to reduce uncertainty about such claims through replication, even without a SESOI (e.g. if you’re willing to be a Bayesian).

”we need a SESOI and an equivalence-test framework”.
On the one hand I agree, but on the other hand, many scientific claims are not related to a SESOI. The claim is simply ”drug A works better than drug B”, ”there is a positive correlation between X and Y”, and so on.

Thanks for sharing your thoughts, this is definitely a worthwhile problem to consider. Let me share some unfinished thoughts of my own:

The post is a light and practical introduction to causal inference with variable control. It assumes som familiarity with regression analysis, and it helps to read my previous posts on causal inference, but that is it. Links to more in-depth literature on the topic are provided in references.

I am currently experimenting with AI-augmentation of my educational materials. This post includes a NotebookLM video summary for those out there who prefer video tutorials to written summaries: youtu.be/EYTNzfHmTvc. The video does not replace the post, but provides a decent high-level summary.

and why adjusting for the wrong ones can introduce new, serious biases. The post features a worked example in Jamovi using simulated data. All files available on OSF >

Thanks to all the commentary authors for your contributions, and thanks especially to the editors at Meta-psychology especially for allowing us to test out this format for our publication. It was a long road, but the end result is in my opinion terrific!

Quantifying replication value as a combination of citation count and sample size is our first stab at solving a very complex problem. Such early attempts benefit enormously from being critiqued right away. We were really lucky to receive several excellent commentaries from many experts in the field.

This paper forms one of Meta-Psychology’s Special Topics. Eight commentaries have been published alongside the paper which criticizes and extends the ideas presented within. We have written a response to these commentaries here: open.lnu.se/index.php/me...

Indeed, I'm looking forward!

True, and maybe that should be emphasized in the blog post. Will consider adding a section at the end of the post. Any sources you'd recommend I'd cite in a section like this?

To be clear, the point of this post is not to say only experiments support causal inference and correlational research never can. Quite the opposite in fact. If that was your takeaway, I may need to add some language to clarify.

The motivation for picking the example in this post was simply that I wanted a thought experiment that a 1st year undergraduate in most areas of social and health science could wrap their head around without any additional reading.

I actually use the history of smoking~cancer research as my running example when introducing our bachelor students to research design and experimental vs observational research. It's a terrific example, albeit on a tragic subject.