...and further research could explore whether different presentation options improve alignment of interpretations. For example, rather than presenting evaluative phrases in isolation, readers could view them in the context of the full underlying ordinal scale.

We also used heat maps to localise the specific phrases that were problematic. Darker colours and higher percentages indicate greater concordance between the implied rank and the intended rank of particular phrases.

Most participants implied rankings of the phrases were misaligned with the ranking intended by eLife. By contrast, most participant's implied rankings of the alternative vocabulary were aligned with the intended ranking. We used Kendall’s distance to quantify the degree of mismatch.

Some eLife phrases elicited relatively consistent interpretation (e.g. “exceptional”, “landmark”), but several had broad/overlapping interpretations (e.g. “fundamental”, “important”, “valuable”). The alt vocab elicited more consistent interpretation, but still had moderate overlap at scale poles

We proposed an alternative vocabulary (see pic). We then asked 301 participants to rate the importance and strength of support of hypothetical studies described using both vocabularies using a 0% to 100% scale to see if their intuitions aligned with each other and with the intended interpretations

Our initial impression was that the intended ordinal structure of the vocab is not intuitive e.g., an “important” study does not necessarily seem more significant than a “valuable” study (also see “compelling” vs “convincing”). The words also do not always map onto gradations of the same construct.

We found that transparency has improved moderately since ~2015 and some practices (open access, disclosure of funding and conflicts of interest) are quite common. However, the availability of core research artifacts (preregistration, materials, data, analysis scripts) remains low.

Of the 160 articles, 29 (18%) already had public data, 7 had restricted-access data, & 7 (4%) shared data in the Data Ark. Data for 117 (73%) articles were not available. Most authors did not respond to data requests. A minority shared reasons for not sharing, such as legal or ethical constraints.

Come and join us doing meta-research in Melbourne!

(very nearly) final programme for next week's Royal Society meeting on "The promise and pitfalls of preregistration": docs.google.com/document/d/1... Still time to register to attend online or in-person in London: www.eventbrite.co.uk/e/the-promis...

A preliminary agenda is available for our meeting on "The promises and pitfalls of preregistration". We've also extended the submission deadline for lightning talks and posters to 29th January. Registration and submission details on the meeting website: royalsociety.org/science-even...

I'm speaking at UBC tomorrow/today, free webinar registration here: www.ti.ubc.ca/2023/09/19/i...