Lightnews — Scholar-powered news

Mark Rubin

@markrubin.bsky.social

Many analysts are not always equal analysts

"Should every analyst's contribution be weighted equally?"

From "Ten Strategies To Improve Many-Analysts Studies" by Alexandra Sarafoglou, @balazsaczel.bsky.social et al.

Preprint: osf.io/preprints/ps...

#MetaSci

Strategy 4. Integrate analytic quality and expertise. As many-analyst approaches mature, there may be good reasons to introduce more nuance to the wisdom of the crowds principle. Should every analyst’s contribution be weighted equally? How can we account for pre-existing differences between analysts —such as varying levels of expertise in the research topic, methodology and statistics, or differences in the time available to dedicate to the analysis— in a way that supports the overall quality of the project? Rather than solely using these factors as potential exclusion criteria (see e.g., Aczel et al., 2021; Gould et al., 2025), they could also serve as grounds for giving greater weight to submissions from highly rated teams. One could make a strong case for taking factors such as subject-matter and methodological expertise into account when weighing the contributions of individual analysts (see e.g., Aczel et al., 2021). This can be done either when calculating an overall meta-analytic effect size across teams (Bartoš, Hoogeveen, et al., 2025) or by incorporating the weights into a meta-regression analysis

November 28, 2025 at 8:28 AM

Mark Rubin

@markrubin.bsky.social

Thanks! Yes, I think many people assume that methodological and statistical rigor should precede concerns about theory. But another interest question here is the extent to which theoretical specification informs (contextualizes) methodological and statistical rigor.

E.g., doi.org/10.1177/2378...

In 2018, Silberzahn, Uhlmann, Nosek, and colleagues published an article in which 29 teams analyzed the same research question with the same data: Are soccer referees more likely to give red cards to players with dark skin tone than light skin tone? The results obtained by the teams differed extensively. Many concluded from this widely noted exercise that the social sciences are not rigorous enough to provide definitive answers. In this article, we investigate why results diverged so much. We argue that the main reason was an unclear research question: Teams differed in their interpretation of the research question and therefore used diverse research designs and model specifications. We show by reanalyzing the data that with a clear research question, a precise definition of the parameter of interest, and theory-guided causal reasoning, results vary only within a narrow range. The broad conclusion of our reanalysis is that social science research needs to be more precise in its “estimands” to become credible.

November 24, 2025 at 4:44 PM

Mark Rubin

@markrubin.bsky.social

Part of the Topical Issue “Theory Specification and Theory Building in Psychology”
doi.org/10.1027/2151...

#MetaSci #PhilSci #SocialPsyc #PsycSci

Psychological theories should meet scientific criteria such as objectivity, coherence, and nontautology. We argue that theories formulated as verbal narratives often fail to meet these standards and propose that all psychological theories should be formally specified to ensure compliance with these requirements. We recommend: (i) requiring authors of new theories to provide sufficient formal specification before publication by changing standards and extending publication manuals; (ii) increasing incentives for the specification of existing theories by introducing a dedicated category of theory specification papers; (iii) encouraging researchers who aim to empirically test a theory to specify the relevant aspects before doing so, if required; (iv) fostering the development of improved theory specification methodologies through methodological debates and consensus-building processes; and (v) enhancing psychology education by incorporating fundamental philosophy of science concepts and formal theory specification techniques. We review theory specification approaches in psychology that can be used to implement these recommendations.

November 23, 2025 at 12:48 PM

Mark Rubin

@markrubin.bsky.social

Building previously hidden moderators into a theory as boundary conditions can lead to overspecification and reduced parsimony.

New work by Johannes Ziegler and @mariogollwitzer.bsky.social

Open Access: doi.org/10.1027/2151...

In the wake of the replication crisis, it has been argued that many psychological theories do not sufficiently specify under which boundary conditions a hypothesized empirical effect occurs versus does not occur, which may explain why conceptual replications are sometimes unsuccessful. This reasoning implies that one path toward theory building is to identify all boundary conditions and to integrate them into the respective theory underlying the effect. In this article, we argue that identifying boundary conditions is undoubtedly worthwhile, but it comes with the risk of overspecifying theories. Overspecification occurs when certain assumptions of a hypothesis or theory can be dropped without noticeable loss in explanatory power. We argue that identifying boundary conditions for an effect to occur can be fruitfully combined with testing for overspecification of a hypothesis or theory. Such a combined strategy can improve cumulative theory building and, thus, increase the efficiency and integrability of psychological research.

November 23, 2025 at 12:48 PM

Mark Rubin

@markrubin.bsky.social

@jwsherman.bsky.social’s article…

“When the goal is to test a theoretical prediction, convenience samples are sufficient, keeping in mind that no claims should be made as to the generalizability of the findings across peoples.”

doi.org/10.1037/amp0...

#PhilSci #SocialPsyc #PsycSci #AcademicSky

Attention to issues of sample diversity and generalizability has increased dramatically in the past 15 years, as psychological scientists have confronted the limitations of relatively homogeneous samples. Though this reckoning was perhaps overdue and has undoubtedly shined a light on some poor research practices, recommendations surrounding sample diversity are sometimes applied to research that does not aim for generalizability across peoples. In this article, I seek to promote discussion about when and why sample diversity and generalizability matter. In doing so, I address problems with language surrounding generalizability, the broader question of generalizability beyond samples, challenges for determining sufficient generalizability, and the inherent question of moderation in psychological science, given the reality of limited time and resources. I then discuss the important roles that basic research plays in understanding group differences, producing generalizable knowledge, and developing applied interventions. Finally, I address issues of equity surrounding sample diversity, emphasizing the distinction between WEIRD samples and convenience samples and the importance of convenience samples for globalizing psychological science.

November 18, 2025 at 9:23 PM

Mark Rubin

@markrubin.bsky.social

“The diverse perspectives of diverse researchers may be particularly important in our efforts to identify and interpret variation across human cultural groups.”

doi.org/10.1037/amp0...

Kroupin, Henrich, & @michael.muthukrishna.com's commentary on...

In this commentary, we supplement Sherman’s (2025) defense of convenience sampling, reviewing the complementary role of broad generalization and diverse samples. Specifically, Sherman’s commentary could be misinterpreted as downplaying or disavowing the importance of broad generalization, despite the latter being necessary if we are to capture more than a narrow sliver of human cognitive variation. Moreover, we argue that stating the generalizability of our findings explicitly is key to both accurate interpretation and effective translation into applied work—a principle which holds even when our studies are not aiming to produce generalizable conclusions. We close with a review of practical ways in which broad generalization may be achieved. These include developmental, comparative, or computational approaches, as well as theoretical frameworks and “inconvenient” samples that capture cross-cultural variation.

November 18, 2025 at 9:23 PM

Mark Rubin

@markrubin.bsky.social

“Most basic research is designed to test a theory, not demonstrate an effect.”

doi.org/10.1037/amp0...

@jwsherman.bsky.social’s response to...

This is a reply to the commentary Kroupin et al. (2025) offered in response to my article on generalizability in basic research (Sherman, 2025). In their response, Kroupin et al. assert that cultural moderation enjoys special status among the many potential moderators of human psychology and behavior. They propose that cultural universality should be a default and explicit assumption of psychological theory until proven otherwise and that testing that generalizability is fundamental to psychological research. In reply, I again assert that, for many types of basic research, that is simply not the case, with other potential moderators playing a more substantial role. I also challenge the assertion that the primary function of basic research is to provide existence proofs of behavioral outcomes, arguing that theory testing, whether via existence proofs or other types of research, is the primary goal of basic research.

November 18, 2025 at 9:23 PM

Mark Rubin

@markrubin.bsky.social

This is the relevant bit re. p-excuses....

"Most p-excuses are used in psychology, in which the phenomenon was initially analyzed (Olsson-Collentine et al. 2018). The least p-excuses are used in health."

November 18, 2025 at 1:20 PM

Mark Rubin

@markrubin.bsky.social

I agree, but I think they're using the word "alternative" there not as H1 vs H0 but as a different type of H0. Confusing, but potentially less problematic (i.e., H0 doesn't have to be 0. It can be anything).

November 17, 2025 at 10:03 PM

Mark Rubin

@markrubin.bsky.social

Yes, sorry, I couldn't fit that last bit into the post. But I didn't think that last part was restricted to multiverse settings? Isn't it a general point about not being overly reliant on preregistration when other sensible approaches exist?

"While preregistration promotes transparency and reduces hindsight bias, it does not automatically improve research quality. Registering one analysis plan from multiple valid alternatives can be arbitrary and does not necessarily strengthen inferential validity12. Preregistration cannot by itself correct flawed analytical choices, highlighting the importance of thoughtful design and justification alongside preregistration. In cases where multiple reasonable analysis paths exist, a multiverse analysis—examining the
robustness of findings across different preprocessing and statistical approaches—may provide more informative insights29,30. Ultimately, while preregistration can serve as a valuable mechanism for enhancing transparency and rigor within an ’error statistical’ framework21, it should not be
regarded as universally necessary, particularly when it risks favoring a single
arbitrary perspective over a more comprehensive examination of the data31."

November 15, 2025 at 9:10 AM

Mark Rubin

@markrubin.bsky.social

"It is typically epistemically harmful to hide dissent, even when there exists an agent purposefully producing biased dissent."

New work by @coatesmatthew.bsky.social

Open Access: doi.org/10.1017/psa....

#PhilSci #Epistemology

There has been increased attention on how scientific communities should respond to spurious dissent. One proposed method is to hide such dissent by preventing its publication. To investigate this, I computationally model the epistemic effects of hiding dissenting evidence on scientific communities. I find that it is typically epistemically harmful to hide dissent, even when there exists an agent purposefully producing biased dissent. However, hiding dissent also allows for quicker correct epistemic consensus among scientists. Quicker consensus may be important when policy decisions must be made quickly, such as during a pandemic, suggesting times when hiding dissent may be useful.

November 14, 2025 at 10:25 PM

Mark Rubin

@markrubin.bsky.social

Durham, UK, tonight

www.lumiere-festival.com

November 14, 2025 at 9:51 PM

Mark Rubin

@markrubin.bsky.social

"p-Excuses"

E.g., p = .056 is “marginally significant” or “close to being significant”

🔹 Most p-excuses are used in psychology.

🔹 The least p-excuses are used in health.

Preprint: osf.io/preprints/me...

#MetaSci #AcademicSky 🧪

Researchers face incentives to write up their empirical findings in a way that maximizes publication success. We analyze the language of journal articles and its association with statistical significance to explore questionable research practices at the stage of writing up articles, using 140,606 articles from health, biology, psychology, economics, and multidisciplinary journals over 32 years. For most disciplines, a higher share of non-significant main findings is associated with more hedging and negative striking words, fewer positive striking words, and fewer superlatives. We neither find evidence that authors upsell ambiguous results using sensational language nor that ambiguous results are written up less readably. Contrarily, articles with a higher share of statistically significant main findings are written up more sensationally. We find that emphasis on (marginal) statistical significance increases with the share of non-significant main findings, reflecting a dichotomized interpretation of p-values based on arbitrary thresholds. Particularly, p-excuses give the impression of statistical significance when the finding is actually not significant, consistent with the notion of 'spin'. This study provides empirical insights that might help researchers to self-reflect on writing up empirical findings. More training and fewer incentives to sell findings using sensational language and spin can help to improve academic writing.

November 14, 2025 at 3:43 PM

Mark Rubin

@markrubin.bsky.social

See also Hahn and Harris (2014):

“‘Bias’ is neither necessarily irrational nor bad in any wider sense.”

psycnet.apa.org/record/2014-...

In this chapter, we provide a historical overview of research on bias in human cognition, ranging from early work in psychology through the detailed, quantitative examinations of belief revision in the 1960s, the Heuristic and Biases program initiated by Kahneman and Tversky, and bias focused research in personality and social psychology. Different notions of "bias" are identified and compared with the notion of bias in statistics, machine learning, and signal detection theory. Comparison with normative models then forms the basis for a critical look at the evidence that people succumb to motivated reasoning aimed at enabling them "to believe what they want to believe."

November 11, 2025 at 3:23 PM

Mark Rubin

@markrubin.bsky.social

Two definitions of bias:

(1) Bias is functional: Biases are necessary for cognition.

(2) Bias is error: Biases hamper cognition and should be eliminated

I distinguish two meanings of the term bias in the social sciences. In the first, biases are functional: they are necessary, and simultaneously enable and constrain perception and cognition. In the second, biases are viewed as errors and ideally should be reduced to zero. In the functional view, bias is value-neutral, neither good nor bad. This pragmatic perspective accepts that cognition must operate under conditions of uncertainty (rather than the certainty of a “small world”) and intractability (where the optimal solution cannot be calculated). Biases enable cognition to deal with these situations where probability theory offers no guidance, for instance, through intelligent heuristics. In contrast, the error view assigns a negative value to bias. It assumes that cognition deals with problems where the true state of the world is known with certainty – at least to some authority. This distinction has profound implications for research design: the two views lead not only to different answers, but also to different questions. Researchers who adopt the error view take the deviation between judgment and true state as the explanandum, not the judgment itself. As a consequence, the functional question – What does a bias achieve? – is virtually never asked, nor is the possibility considered that certain biases might lead to better judgments. This is one reason why less-is-more effects –conditions under which ignoring information yield more accurate inferences – were discovered only recently. Ultimately, views about the nature of bias can themselves become a bias in research on biases.

November 11, 2025 at 3:23 PM

Mark Rubin

@markrubin.bsky.social

"Basic scientists consistently enhance scientific output quality, yet research and newly joining researchers have shifted toward applied orientations."

Preprint: arxiv.org/abs/2509.01530

#PhilSci #MetaSci #AcademicSky 🧪

Despite broad acclaim for basic research, science is undergoing an applied shift that marginalizes basic scientists. This gap reflects an incomplete understanding of their distinctive roles, which prevents translating philosophical appreciation into effective support. We introduce a scalable metric--the application score--to position research along the basic-applied spectrum and apply it to 62 million publications (1970-2023) to reveal the distinctive contributions of basic scientists. We find a structural asymmetry: involvement of basic scientists substantially increases citation impact, even more so in applied contexts, while applied scientists show no such effect in basic domains. This asymmetric effect arises from their intellectual leadership in conceptualization, writing, and experimental design, amplified in large, multidisciplinary, and intermediate career teams. Yet basic scientists remain concentrated in historically prestigious institutions, while new entrants shift toward applied work, indicating critical undersupply. These findings provide large-scale evidence for the indispensable role of basic scientists, guiding policy and institutional strategy to sustain the foundations of discovery and innovation.

November 8, 2025 at 11:37 AM

Mark Rubin

@markrubin.bsky.social

"While ingroup evaluation may not always be positive, this does not preclude strong ingroup identification."

Open Access: doi.org/10.1080/1529...

Social identities play a crucial role in shaping how individuals interpret their position in society and navigate social interactions to serve both personal and ingroup interests. Existing social identity theorizing and research have offered only limited insights into how social identification operates among stigmatized group members, whose experiences may differ both in content and complexity from non-stigmatized group members. The present research addressed this gap by investigating the multicomponent ingroup identification model across 18 stigmatized groups in the U.S. (N = 3.969). Confirmatory factor analyses showed only an acceptable model fit for the overall sample and substantial variability in model fit and factor structure across subsamples – challenging assumptions of a universal social identification structure. Multilevel analyses revealed that between-group heterogeneity can be explained by group-level factors (socioeconomic status and group entitativity) and individual belief systems (permeability and legitimacy), aligning with core assumptions of Social Identity Theory. Our findings suggest important differences in social identification between stigmatized groups and point to both group- and individual-level determinants of social identification. We discuss the implications for existing measurement models and highlight the need for further theorizing and empirical work on social identification in the context of stigmatized identities.

November 7, 2025 at 5:50 PM

Mark Rubin

@markrubin.bsky.social

“Replication is an ethnography of science, a reflection by the scientists themselves on the scientific process and the challenges of producing results, while being engaged in that very process.”

New ethnographic work by Brenninkmeijer et al.

Open Access: doi.org/10.1177/0306...

🧵

Ethnography and other empirical studies of replication played a significant role in the sociology of scientific knowledge (SSK) during the 1970s and 1980s. Collins and other proponents of SSK highlighted that exact replication was impossible, knowledge was often tacit and hard to explicate, and that results were always socially negotiated. We revisit these observations and compare them with our ethnography of current replication practices in psychology. We highlight the diversity of negotiations, tacit knowledge, and practices surrounding replication more generally. We also examine how replication encourages and necessitates reflection on the research process—on what should count as the same experiment and the same result, on the role of tacit knowledge (such as skill, experience, judgment, and feeling) in science, and on what replication entails and whether it is worth the effort. Researchers who conduct replications can be seen as ethnographers of their science, reflecting on the scientific process and the challenges of producing results.

November 1, 2025 at 2:01 PM

Mark Rubin

@markrubin.bsky.social

Should 'sisters' be doing it by themselves when it comes to gender equality initiatives? Not really, no!

"It [also] matters whether the approach is arguing for a 'we-change' or a 'she-change'.”

Open Access: doi.org/10.1111/pops...

BSky author: @shellkryan.bsky.social

While gender equality initiatives have historically been spearheaded by women, male allies' contribution is increasingly recognized—and challenged. Our article examines the pivotal yet neglected intersection of women's leadership and allyship for gender equality. Across two experiments with community samples (total N = 801), we investigate how message framing (common cause vs. women's issue; Experiment 1), intergroup/male versus intragroup/female allies (Experiments 1–2), and transformative versus tokenistic allyship (Experiment 2) affect female leaders' capacity to mobilize both men and women for gender equality. We demonstrate that common cause (vs. women's issue) messages more readily mobilize men for collective action, whereas women's mobilization remains high irrespective of message framing. We also show that a female leader supported by an intergroup/male rather than an intragroup/female ally is more likely to be seen as “one of us,” have greater influence, and be more effective at mobilizing both men and women for collective action (Experiments 1–2). Critically, men are mobilized by transformative allyship (regardless of ally gender) and disengage from both the leader and the cause in response to tokenistic allyship (Experiment 2). While male allies can be important, our results suggest that transformative allyship is essential for mobilization across gender boundaries.

October 30, 2025 at 9:57 PM

Mark Rubin

@markrubin.bsky.social

"Academic freedom supports methodological pluralism and critical debate which are the most important drivers of scientific progress. When dissent is permitted and diverse viewpoints can be openly discussed, the self-correcting mechanisms of science become more effective."

doi.org/10.1016/j.ij...

🧪

Academic freedom is a widely discussed concept recognized as a key element of the academic system. While the intrinsic value of academic freedom is widely recognised, there is limited empirical evidence demonstrating its impact on research productivity and the quality of scientific output. This study examines the relationship between academic freedom and scientific performance using a global panel dataset covering 119 countries from 1996 to 2021. Research quality was measured using two field-normalized indicators: the mean field-normalized citation rate and the proportion of a country’s publications among the top 10% of most cited papers. Linear mixed-effects models reveal that academic freedom at the country level is associated with the citation impact of future papers published by researchers from a country. These effects are robust across multiple time lags (2, 3, 4, and 5 years). The analysis also shows that the positive effect of academic freedom on citation impact is primarily evident in countries where academic freedom is already at moderate or high levels. These results reinforce the idea that academic freedom is not only a normative ideal but also a structural factor that supports high-quality science.

October 29, 2025 at 10:46 PM

Mark Rubin

@markrubin.bsky.social

For more on Zanna’s work, see:
www.durham.ac.uk/staff/zanna-...

October 24, 2025 at 3:27 PM

Mark Rubin

@markrubin.bsky.social

Excellent presentation by @zannaclay.bsky.social today @durhampsych.bsky.social tracing the origins of our social minds from great apes and young children.

#Psychology #evosky #AcademicSky 🧪

As a highly social primate, humans have evolved a sophisticated set of socio-cognitive, affective and communicative abilities that enable us to navigate our social worlds. Given that such traits do not fossilise, studying our closest living relatives –the great apes – can offer us a unique window through which we can explore their evolutionary origins. In parallel, examining a system in its earlier stages of development, before its components are mature, allows us to reconstruct its foundations, understand how it is organised and how it may have evolved. Throughout my career, I have had the enormous privilege of studying the behaviour and minds of chimpanzees and bonobos, our closest ape relatives, while immersing myself in their social lives. More recently, I have extended this lens to young human children and their caregivers, including across cultures. In this inaugural lecture, I will reflect on my research journey, exploring the evolutionary and developmental origins of language, empathy and culture, and how these threads intertwine to illuminate what it means to be human.

October 24, 2025 at 3:27 PM

Mark Rubin

@markrubin.bsky.social

Thoughtful response by Makel et al. to concerns raised about the value of replication in qualitative research

Preprint: doi.org/10.35542/osf...

Abstract
In 2022, we argued that replication is relevant to qualitative research. In this manuscript, we update and clarify that idea to make three more specific claims about replication in qualitative research: (1) it is possible, (2) it can be relevant, and (3) it can be useful. We provide practical
examples and draw attention to two areas we believe need continued consideration: when to conduct a replication, jingle-jangle problems. We conclude that quantitative tools should not automatically be assumed to apply to qualitative research or forced upon qualitative researchers. However, this should not be conflated with the more extreme position—one we disagree with—that methods from quantitative research should never be applied to qualitative research.
Keywords: qualitative research; direct replication, conceptual replication; replication

October 23, 2025 at 8:37 PM

Mark Rubin

@markrubin.bsky.social

I liked that response too.

Also, Giudice and Gangestad's (2021) point that "multiverse-style analyses can generate analytic 'black holes': massive analyses that swallow true effects of interest."

doi.org/10.1177/2515...

$By inflating the size of the analysis space, the combi- natorial explosion of unjustified specifications may, ironically, exaggerate the perceived exhaustiveness and authoritativeness of the multiverse while greatly reducing the informative fraction of the multiverse. At the same time, the size of the specification space can make it harder to inspect the results for potentially relevant findings. If unchecked, multiverse-style analyses can generate analytic “black holes”: massive analyses that swallow true effects of interest but, because of their perceived exhaus- tiveness and sheer size, trap whatever information is present in impenetrable displays and summaries.$

October 21, 2025 at 4:55 PM

Mark Rubin

@markrubin.bsky.social

"Statistical presentations should not be interpreted as what the data show but rather as what the analysts would argue the data show based on their preferred approach to the data."

Preprint: arxiv.org/pdf/2508.10168

Paywall: doi.org/10.1007/978-...

The study of associations and their causal explanations is a central research activity whose methodology varies tremendously across fields. Even within specialized subfields, comparisons across textbooks and journals reveals that the basics are subject to considerable variation and controversy. This variation is often obscured by the singular viewpoints presented within textbooks and journal guidelines, which may be deceptively written as if the norms they adopt are unchallenged. Furthermore, human limitations and the vastness within fields imply that no one can have expertise across all subfields and that interpretations will be severely constrained by the limitations of studies of human populations. The present chapter outlines an approach to statistical methods that attempts to recognize these problems from the start, rather than assume they are absent as in the claims of “statistical significance” and “confidence” ordinarily attached to statistical tests and interval estimates. It does so by grounding models and statistics in data description, and treating inferences from them as speculations based on assumptions that cannot be fully validated or checked using the analysis data.

October 17, 2025 at 5:12 PM

Add to Home Screen

Light up
your news

Add to Home Screen

Light upyour news

Sign in to Lightnews

Sign up to start reading

Connect Bluesky

Connect with Bluesky

Light up
your news