Test severity may at once improve the ability of a replication study to reduce uncertainty, but it can also leave less uncertainty to be reduced to begin with.

Thank you for the quick response, that helped for sure, but I will also dive into the PhD thesis discussion!

You can see I have not thought this through completely, but I would be very interested in your opinions.

@isager.bsky.social @lakens.bsky.social

This raises the question: Should we restrict potential replication targets to genuinely confirmatory studies on the grounds that claims arising from exploratory studies carry so many layers of uncertainty that a single replication study cannot meaningfully reduce them?

This leads to what appears to be a paradox: test severity can function both as an argument for replication (because only severe tests yield falsifiable claims) and as an argument against replication (because severe tests leave less uncertainty to be reduced). ->

studies lacking a SESOI might therefore seem more attractive replication targets—but then no clear criterion for replication success or failure exists. (Replicators could define a SESOI, but that would test the claim under different empirical assumptions than the original study.) ->

Uncertainty aspect.
At the same time, claims based on severe tests (with a clear SESOI and controlled Type II error) already involve less uncertainty than claims without such standards. From an uncertainty-reduction perspective ->

This implies that only original studies that reasonably specified a SESOI—i.e., studies that were “confirmatory” from an error-control perspective—should be considered feasible targets for replication. ->

Feasibility aspect
To generate falsifiable predictions in a replication study, we need a SESOI and an equivalence-test framework to define what would count as replication success or failure. In this sense, only falsifiable claims can be subjected to severe testing and thus meaningfully replicated ->

Assume we are at the third step of the replication-value selection procedure proposed by @isager.bsky.social , where a small set of potential replication targets is qualitatively evaluated (and assume they differ only on the two aspects described below). ->

These are exactly the right discussions to have IMO, especially when they help us better understand what knowledge we need (and might still be missing) for setting up meaningful, informative tests.

I think the thing that most people struggle with though is how to set that SESOI... @cspaeth.bsky.social and I been chatting about it in this thread (and I give examples of how we've gone about it... though our recent theory prediction + practical SESOI is best I think) bsky.app/profile/cspa...

Yes, in my specific example, we are investigating the "minimal perceived change" using a "global rating of change" as an anchor. So basically a similar approach.

But thank you also for the other recommendations!!! I will have a look!

So (beyond this specific example of anchor-based approaches) I would be happy to see many diverse applied examples on how SESOIs have been reasonably specified in sport and exercise science (if they exist...).

But the literature on the best estimation procedure to validly derive a SESOI based on these anchor-variables is very heterogenous: 
doi.org/10.1016/j.ym...

The most common approach is an anchor-based approach (i.e., using an external anchor/reference variable as a guidance to derive a SESOI), especially for psychological variables that have no natural metric and are measured with Likert-type scales.

doi.org/10.1016/j.je...

For example, I am currently working on determining a SESOI for improvements in affective responses during exercise (hope to submit the paper by the end of the year). ->