Lightnews — Scholar-powered news

Magnus Johansson @pgmj.bsky.social · 1d

Re-updated the page and the R package, including the conditional reliability method by McNeish & Dumas as well.

McNeish & Dumas (2025). Reliability representativeness: How well does coefficient alpha summarize reliability across the score distribution? doi.org/10.3758/s134...

Reliability representativeness: How well does coefficient alpha summarize reliability across the score distribution? - Behavior Research Methods

Scale scores in psychology studies are commonly accompanied by a reliability coefficient like alpha. Coefficient alpha is an index that summarizes reliability across the entire score distribution, implying equal precision for all scores. However, an underappreciated fact is that reliability can be conditional such that scores in certain parts of the score distribution may be more reliable than others. This conditional perspective of reliability is common in item response theory (IRT), but psychologists are generally not well versed in IRT. Correspondingly, the representativeness of a single summary index like alpha across the entire score distribution can be unclear but is rarely considered. If conditional reliability is fairly homogeneous across the score distribution, coefficient alpha may be sufficiently representative and a useful summary. But, if conditional reliability is heterogeneous across the score distribution, alpha may be unrepresentative and may not align with the reliability of a typical score in the data or with a particularly important score like a cut point where decisions are made. This paper proposes a method, R package, and Shiny application to quantify the potential differences between coefficient alpha and conditional reliability across the score distribution. The goal is to facilitate comparisons between conditional reliability and reliability summary indices so that psychologists can contextualize the reliability of their scores more clearly and comprehensively.

doi.org

Magnus Johansson @pgmj.bsky.social · 1d

I agree. #openscience combined with #opensource software is great!

Dr Mircea Zloteanu ☀️ 🌊🌴 @mzloteanu.bsky.social · 1d

This is what I find awesome (in the truest sense) about scientific development in the modern age. One group posts a new technique and within a few days it is implemented by others. #rstats is magic (and efficient)

Magnus Johansson @pgmj.bsky.social · 2d

This is really neat. I have borrowed the reliability() function to my `easyRasch` package, and use plausible values instead of fully Bayesian estimation to produce similar estimates/CIs, see code example below. RMU point estimates are similar to EAP reliability.

pgmj.github.io/easyRasch/re...

4

Magnus Johansson @pgmj.bsky.social · 1d

The RelRep.R function "just works" with a dataframe with items as columns and produces pretty neat output for alpha or omega. Point estimate and CI for alpha is almost identical to RMU for my example with `eRm::raschdat1[,1:20]` data. github.com/melissagwolf...

1

Dr Mircea Zloteanu ☀️ 🌊🌴 @mzloteanu.bsky.social · 1d

This is what I find awesome (in the truest sense) about scientific development in the modern age. One group posts a new technique and within a few days it is implemented by others. #rstats is magic (and efficient)

Magnus Johansson @pgmj.bsky.social · 2d

This is really neat. I have borrowed the reliability() function to my `easyRasch` package, and use plausible values instead of fully Bayesian estimation to produce similar estimates/CIs, see code example below. RMU point estimates are similar to EAP reliability.

pgmj.github.io/easyRasch/re...

Giacomo Bignardi @bignardi.bsky.social · 11d

New preprint with @rogierk.bsky.social @paulbuerkner.com - we introduce "relative measurement uncertainty" - a reliability estimation method that's applicable across a broad class of Bayesian measurement models (e.g., generative-, computational- and item response theory-models osf.io/h54k8

1 7

Magnus Johansson @pgmj.bsky.social · 1d

Looking closer at McNeish & Dumas (2025). How well does coefficient alpha summarize reliability across the score distribution? doi.org/10.3758/s134...

The RelRep.R function seems like it could be combined with your approach as a conditional reliability metric. I'll probably have a go at it.

Reliability representativeness: How well does coefficient alpha summarize reliability across the score distribution? - Behavior Research Methods

Scale scores in psychology studies are commonly accompanied by a reliability coefficient like alpha. Coefficient alpha is an index that summarizes reliability across the entire score distribution, implying equal precision for all scores. However, an underappreciated fact is that reliability can be conditional such that scores in certain parts of the score distribution may be more reliable than others. This conditional perspective of reliability is common in item response theory (IRT), but psychologists are generally not well versed in IRT. Correspondingly, the representativeness of a single summary index like alpha across the entire score distribution can be unclear but is rarely considered. If conditional reliability is fairly homogeneous across the score distribution, coefficient alpha may be sufficiently representative and a useful summary. But, if conditional reliability is heterogeneous across the score distribution, alpha may be unrepresentative and may not align with the reliability of a typical score in the data or with a particularly important score like a cut point where decisions are made. This paper proposes a method, R package, and Shiny application to quantify the potential differences between coefficient alpha and conditional reliability across the score distribution. The goal is to facilitate comparisons between conditional reliability and reliability summary indices so that psychologists can contextualize the reliability of their scores more clearly and comprehensively.

doi.org

1

Magnus Johansson @pgmj.bsky.social · 2d

I've updated the reliability section of my `easyRasch` vignette as well: pgmj.github.io/raschrvignet...
#psychometrics #rasch #rstats

easyRasch vignette – R, Rasch, etc

R package for Rasch analysis

pgmj.github.io

1 3

Magnus Johansson @pgmj.bsky.social · 2d

I don't think so. This is a great development you've made! And through your references on Test Information Function I abandoned that path. As additional reliability metrics, I would like to find 1) a conditional reliability metric (non-constant); 2) a test reliability metric (sample independent)

2

Magnus Johansson @pgmj.bsky.social · 2d

This is really neat. I have borrowed the reliability() function to my `easyRasch` package, and use plausible values instead of fully Bayesian estimation to produce similar estimates/CIs, see code example below. RMU point estimates are similar to EAP reliability.

pgmj.github.io/easyRasch/re...

Giacomo Bignardi @bignardi.bsky.social · 11d

New preprint with @rogierk.bsky.social @paulbuerkner.com - we introduce "relative measurement uncertainty" - a reliability estimation method that's applicable across a broad class of Bayesian measurement models (e.g., generative-, computational- and item response theory-models osf.io/h54k8

OSF

osf.io

2 1 8

Magnus Johansson @pgmj.bsky.social · 3d

CSV will not retain factor levels/labels, etc. I prefer to use .parquet or .qs2 formats, both open source and available in R packages `arrow` and `qs2`. The latter is R specific and has the added benefit of being able to save any R object to a compressed file.

3

Magnus Johansson @pgmj.bsky.social · 3d

Det är mest troligt s.k. attention-control-frågor, som ibland läggs in i (långa) frågebatterier för att identifiera respondenter som bara kryssar random svar.

7

Andrew Heiss @andrew.heiss.phd · 4d

I really really really liked The Oatmeal’s take on AI here theoatmeal.com/comics/ai_art

A cartoonist's review of AI art - The Oatmeal

This is a comic about AI art.

theoatmeal.com

3 23 69

Jourhavande Filosof @systerfrida.se · 4d

”AI isn't going to wake up, become superintelligent and turn you into paperclips – but rich people with AI investor psychosis are almost certainly going to make you much, much poorer.”

pluralistic.net/2025/09/27/e...

Pluralistic: The real (economic) AI apocalypse is nigh (27 Sep 2025) – Pluralistic: Daily links from Cory Doctorow

pluralistic.net

1 6 10

Dr Mircea Zloteanu ☀️ 🌊🌴 @mzloteanu.bsky.social · 4d

#statstab #432 PsychOpen Gold

Thoughts: instead of submitting to greedy and unhelpful publishers, try this list of fully open and free journals in psychology.

#OpenScience #openaccess #apcs #goldaccess #pedagogy

psychopen.eu

PsychOpen GOLD: Open Access Publishing

We are a Diamond Open Access platform for psychology research. Peer-reviewed, free-to-read journals with no publication fees, promoting open science.

psychopen.eu

1 2 9

Magnus Johansson @pgmj.bsky.social · 6d

scholar.google.com/scholar?hl=s...

1

Robin Blythe @rbly.bsky.social · 7d

Love this paper by Hinne doi.org/10.1177/2515... but curious why BMA isn't compatible with competing theories: "BMA is less useful when...each candidate model may represent a different theory of a physical process...the models are not a nuisance factor; they are the focus of the analysis." #statsky

A Conceptual Introduction to Bayesian Model Averaging - Max Hinne, Quentin F. Gronau, Don van den Bergh, Eric-Jan Wagenmakers, 2020

Many statistical scenarios initially involve several candidate models that describe the data-generating process. Analysis often proceeds by first selecting the ...

doi.org

1 2 5

Ian Hussey @ianhussey.mmmdata.io · 8d

My article "Data is not available upon request" was published in Meta-Psychology. Very happy to see this out!
open.lnu.se/index.php/me...

LnuOpen | Meta-Psychology

open.lnu.se

5 37 100

Ada Palmer @adapalmer.bsky.social · 9d

Exceptional Doctorow piece on the solar rollout, why it's unstoppable, & why many old downsides of solar aren't factors anymore. The entire material mining needed 4 a global solar transition = only 17% of the fossil fuel mining we do every year! & then we're done! doctorow.medium.com/https-plural...

Decarbonization at a distance

A post-American century that runs on sunshine.

doctorow.medium.com

7 120 360

Magnus Johansson @pgmj.bsky.social · 11d

I'm sure there are ways it is brilliant, but my first methods check in EFA/CFA psychometric papers is always whether rule-of-thumb critical values are used to assess model fit. I don't think we should trust model fit interpretations using rule-of-thumb cutoffs in any paper.

1 2

Giacomo Bignardi @bignardi.bsky.social · 11d

New preprint with @rogierk.bsky.social @paulbuerkner.com - we introduce "relative measurement uncertainty" - a reliability estimation method that's applicable across a broad class of Bayesian measurement models (e.g., generative-, computational- and item response theory-models osf.io/h54k8

OSF

osf.io

2 7 18

Magnus Johansson @pgmj.bsky.social · 11d

I strongly recommend to stop using rule of thumb cutoffs such as Hu and Bentler since they are unlikely to be appropriate. Dynamic/adaptive cutoffs should be used, see for instance pubmed.ncbi.nlm.nih.gov/36787513/

dynamic : An R Package for Deriving Dynamic Fit Index Cutoffs for Factor Analysis - PubMed

To evaluate the fit of a confirmatory factor analysis model, researchers often rely on fit indices such as SRMR, RMSEA, and CFI. These indices are frequently compared to benchmark values of .08, .06, ...

pubmed.ncbi.nlm.nih.gov

1

Matti Vuorre @matti.vuorre.com · 16d

100% this, especially with Wiley given their anti-preprint stance.

Francesco Poli @francescopoli.bsky.social · 16d

Why are we still sending our work to Wiley and other publishing companies so that they can profit from it? There's so many better options now, for example: psychopen.eu/journals/

1 5 14

Magnus Johansson @pgmj.bsky.social · 17d

Interesting paper on how software could help analysts. ”Causal clarity in statistical software” (Korf et al., 2025). I like many of the ideas, such as clarifying estimand, using multiple estimators, and no output of coefficients for adjustment variables doi.org/10.1093/ije/... #stats #rstats

Causal clarity in statistical software

Imagine running a simple regression in any statistical software of choice—but this time, you only get a point estimate of the regression coefficient. There

doi.org

6

Emil Hvitfeldt @emilhvitfeldt.bsky.social · 18d

I'm exited to announce a new resource about making slides with quarto and revealjs. This book is the combination of all the work I have done in this area, reordered and polished up

There isn't a lot of new information yet, but this format allows me to add more easily

slidecrafting-book.com
#quarto

Screenshot of first page of slidecrafting-book.com website

11 64 180

Anna Alexandrova @annaalexandrova.bsky.social · 20d

I hope Melvin Bragg's retirement will not end In Our Times. He has a lovely voice and manner but it's the people he invites that hold the show. I'll take this chance to post a thread of some of the more memorable episodes for me. Starting with Schaffer, Worrall, and @michelamassimi.bsky.social

BBC Radio 4 - In Our Time, The Scientific Method

Melvyn Bragg and his guests discuss the Scientific Method.

www.bbc.co.uk

3 16 72