JK Flake
@jkflake.bsky.social
Methodology prof in psychology at the University of British Columbia working on Schmeasurement. Into big team science, open science, and latent variables. Flake lab does not study flakes.
And this is why meta analysis is screwed
Lisa Bero's first slide at #AIMOS2025 on the evidence base weakening because of commercial influence on research, e.g. false claims of efficacy #metascience
November 18, 2025 at 11:57 PM
And this is why meta analysis is screwed
Sorry I gave them so much hell on this one but happy to see some solid discussion about the necessity of qualitative approaches. Bravo.
We put forth several recommendations to help tackle this problem, including better curation of measures, careful attention to measurement validation (particularly with qualitative approaches), and adoption of SEM approaches that can probe for method effects.
September 11, 2025 at 2:41 AM
Sorry I gave them so much hell on this one but happy to see some solid discussion about the necessity of qualitative approaches. Bravo.
Reposted by JK Flake
Our lab has three RRs of simulations underway. It’s a shame to tell my students that our top journal will not consider them for publication. Let’s change that!
Help us show there is support for offering registered reports at one of Psychology's leading method's journal! chng.it/TwwnVBScVb
Adopt Registered Reports at Psychological Methods
Can you spare a minute to help this campaign?
chng.it
August 22, 2025 at 2:43 PM
Our lab has three RRs of simulations underway. It’s a shame to tell my students that our top journal will not consider them for publication. Let’s change that!
Help us show there is support for offering registered reports at one of Psychology's leading method's journal! chng.it/TwwnVBScVb
Adopt Registered Reports at Psychological Methods
Can you spare a minute to help this campaign?
chng.it
August 21, 2025 at 6:44 PM
Help us show there is support for offering registered reports at one of Psychology's leading method's journal! chng.it/TwwnVBScVb
Reposted by JK Flake
"Preregistration" lumps three distinct practices into one:
- Study registration
- Protocol
- Analysis plans (SAPs)
To advance open science, it's critical that we distinguish them.
🌟 🌟 Now in-press paper from Evan Mayo-Wilson, @seangrant.bsky.social, David Moher, me: osf.io/preprints/me...
- Study registration
- Protocol
- Analysis plans (SAPs)
To advance open science, it's critical that we distinguish them.
🌟 🌟 Now in-press paper from Evan Mayo-Wilson, @seangrant.bsky.social, David Moher, me: osf.io/preprints/me...
OSF
osf.io
August 14, 2025 at 6:23 PM
"Preregistration" lumps three distinct practices into one:
- Study registration
- Protocol
- Analysis plans (SAPs)
To advance open science, it's critical that we distinguish them.
🌟 🌟 Now in-press paper from Evan Mayo-Wilson, @seangrant.bsky.social, David Moher, me: osf.io/preprints/me...
- Study registration
- Protocol
- Analysis plans (SAPs)
To advance open science, it's critical that we distinguish them.
🌟 🌟 Now in-press paper from Evan Mayo-Wilson, @seangrant.bsky.social, David Moher, me: osf.io/preprints/me...
Reposted by JK Flake
AMPPS Call for Papers: Replicability and Reproducibility in Methodological Research. Proposals due September 15. @jkflake.bsky.social
www.psychologicalscience.org
August 5, 2025 at 3:02 PM
AMPPS Call for Papers: Replicability and Reproducibility in Methodological Research. Proposals due September 15. @jkflake.bsky.social
Reposted by JK Flake
Folks, it's time to submit your papers to PCI Psychology. Here is our invitation to you, taken from our editorial osf.io/preprints/ps...
June 12, 2025 at 12:27 PM
Folks, it's time to submit your papers to PCI Psychology. Here is our invitation to you, taken from our editorial osf.io/preprints/ps...
Reposted by JK Flake
Check out this awesome new paper about #OpenScience in developing countries! Paper is forthcoming at AMPPS- very proud that we can promote this work. Congrats to the large and international authorship team, and esp @hcp4715.bsky.social for leading the charge. @psychscience.bsky.social
🎉 After 2 years of work, our #OpenScience guide for researchers in developing countries is LIVE!
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
June 20, 2025 at 9:42 PM
Check out this awesome new paper about #OpenScience in developing countries! Paper is forthcoming at AMPPS- very proud that we can promote this work. Congrats to the large and international authorship team, and esp @hcp4715.bsky.social for leading the charge. @psychscience.bsky.social
Reposted by JK Flake
A new preregistration template for simulation studies is now live on OSF. Thanks to the creators and the preregistration template working group for proposing and shepherding this to make it available to users.
See the post if you would like to propose a new template.
www.cos.io/blog/introdu...
See the post if you would like to propose a new template.
www.cos.io/blog/introdu...
Introducing the Simulation Studies Preregistration Template: Q&A with Björn S. Siepe, František Bartoš, and Samuel Pawel
Initially submitted through COS’s open call for community-designed preregistration templates, the Simulation Studies Template is now part of the expanding collection of preregistration resources avail...
www.cos.io
July 29, 2025 at 1:49 PM
A new preregistration template for simulation studies is now live on OSF. Thanks to the creators and the preregistration template working group for proposing and shepherding this to make it available to users.
See the post if you would like to propose a new template.
www.cos.io/blog/introdu...
See the post if you would like to propose a new template.
www.cos.io/blog/introdu...
Reposted by JK Flake
Experimentology is out today!!! A group of us wrote a free online textbook for experimental methods, available at experimentology.io - the idea was to integrate open science into all aspects of the experimental workflow from planning to design, analysis, and writing.
July 1, 2025 at 6:26 PM
Experimentology is out today!!! A group of us wrote a free online textbook for experimental methods, available at experimentology.io - the idea was to integrate open science into all aspects of the experimental workflow from planning to design, analysis, and writing.
Reposted by JK Flake
🚨 Calling all dog behavior & cognition researchers - ManyDogs Project is launching ManyDogs 2!
🔍🐕🦺We're studying overimitation: Do dogs copy irrelevant actions just b/c their favorite human does them?
Email [email protected] to collaborate globally and contribute to reproducible science!
🔍🐕🦺We're studying overimitation: Do dogs copy irrelevant actions just b/c their favorite human does them?
Email [email protected] to collaborate globally and contribute to reproducible science!
July 22, 2025 at 2:19 AM
🚨 Calling all dog behavior & cognition researchers - ManyDogs Project is launching ManyDogs 2!
🔍🐕🦺We're studying overimitation: Do dogs copy irrelevant actions just b/c their favorite human does them?
Email [email protected] to collaborate globally and contribute to reproducible science!
🔍🐕🦺We're studying overimitation: Do dogs copy irrelevant actions just b/c their favorite human does them?
Email [email protected] to collaborate globally and contribute to reproducible science!
Reposted by JK Flake
I'm very happy to see meta-science on methods research getting more attention in the form of a great new call for papers!
I recently gave some talks on the existing literature on the quality and transparency of sim studies & ideas for future improvement. Slides: bsiepe.github.io/talks/2025-0...
I recently gave some talks on the existing literature on the quality and transparency of sim studies & ideas for future improvement. Slides: bsiepe.github.io/talks/2025-0...
July 21, 2025 at 3:21 PM
I'm very happy to see meta-science on methods research getting more attention in the form of a great new call for papers!
I recently gave some talks on the existing literature on the quality and transparency of sim studies & ideas for future improvement. Slides: bsiepe.github.io/talks/2025-0...
I recently gave some talks on the existing literature on the quality and transparency of sim studies & ideas for future improvement. Slides: bsiepe.github.io/talks/2025-0...
Are you working on issues of replication and reproducibility of methodological research in psychology? Consider submitting a proposal to my call for papers, happy to chat about your ideas or answer any questions! www.psychologicalscience.org/publications...
<em>AMPPS</em> Call for Papers on Replicability and Reproducibility in Methodological Research
Methodological reform has shaped the last decade of psychological research. Researchers have undertaken replication studies, journals and funders have emphasized registration and data sharing, and aut...
www.psychologicalscience.org
July 18, 2025 at 9:30 PM
Are you working on issues of replication and reproducibility of methodological research in psychology? Consider submitting a proposal to my call for papers, happy to chat about your ideas or answer any questions! www.psychologicalscience.org/publications...
Reposted by JK Flake
🎉 After 2 years of work, our #OpenScience guide for researchers in developing countries is LIVE!
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
June 19, 2025 at 3:06 PM
🎉 After 2 years of work, our #OpenScience guide for researchers in developing countries is LIVE!
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
Check out our new preprint: osf.io/preprints/ps...
We map the opportunities 🌟 + barriers 🧱 facing Global South researchers.
*(1/4)*
Reposted by JK Flake
PCI Psychology is here!! 🎉🥳
After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org
#PsychSciSky #scipub
After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org
#PsychSciSky #scipub
June 11, 2025 at 11:03 PM
PCI Psychology is here!! 🎉🥳
After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org
#PsychSciSky #scipub
After over a year of hard work by so many people, we are thrilled to announce that we are open for submissions! Join us in making publishing more efficient, equitable, and open: psych.peercommunityin.org
#PsychSciSky #scipub
Robustness analysis is going to be a big part of the future of how we think about evidence synthesis and rigor - exciting new journal where you can turn your blog posts of other ways the analyses could have been done into papers! Excited to see some submissions come in!
We are live!
Introducing the "Journal of Robustness Reports" – a Diamond Open-Access journal dedicated to publishing short reanalyses of empirical findings.
Check out our website and blog post about the journal:
🌐 scipost.org/JRobustRep
📄 www.bayesianspectacles.org/introducing-...
Introducing the "Journal of Robustness Reports" – a Diamond Open-Access journal dedicated to publishing short reanalyses of empirical findings.
Check out our website and blog post about the journal:
🌐 scipost.org/JRobustRep
📄 www.bayesianspectacles.org/introducing-...
![[Editorial: Introducing the Journal of Robustness Reports]](https://cdn.bsky.app/img/feed_thumbnail/plain/did:plc:4bdpxswsa5p65ni7dd2f7f5y/bafkreia6p2iuq3bb4q2v5bsemv4wuw6q43635ppvcv7t2cfx7pqr74alxu@jpeg)
May 26, 2025 at 4:09 PM
Robustness analysis is going to be a big part of the future of how we think about evidence synthesis and rigor - exciting new journal where you can turn your blog posts of other ways the analyses could have been done into papers! Excited to see some submissions come in!
In the workshop we'll follow an example of developing a survey to measuring pickle fanatism for marketing purposes to learn about how to develop and evaluate survey questions for a variety of purposes. The workshop spans qualitative and quantitative methods. Two days of measurement fun!
@jkflake.bsky.social has found a way to merge her passions for science and pickles. Find out how in her course "Scale Development and Psychometrics" offered at Rocky Mountain Methodology Academy in Calgary, July 21-22, 2025. haskayne.ucalgary.ca/ccram/academy A trip to Banff is included.
May 23, 2025 at 3:53 PM
In the workshop we'll follow an example of developing a survey to measuring pickle fanatism for marketing purposes to learn about how to develop and evaluate survey questions for a variety of purposes. The workshop spans qualitative and quantitative methods. Two days of measurement fun!
HELP: Many of you will be familiar with the dataset that is often used to teach multilevel modeling with variable such as "MATHACH" and "SES" -- these data are used in papers, textbooks, and software packages. You may have the dataset on your computer, BUT, is it public use data?
High School & Beyond (HS&B) - Available Data
The HS&B descriptive report describes the educational attainment, employment outcomes, and family formation of the 1980 sophomore class in 1992, ten years after most of the students in that cohort gra...
nces.ed.gov
April 28, 2025 at 8:18 PM
HELP: Many of you will be familiar with the dataset that is often used to teach multilevel modeling with variable such as "MATHACH" and "SES" -- these data are used in papers, textbooks, and software packages. You may have the dataset on your computer, BUT, is it public use data?
Reposted by JK Flake
Chuffed and excited to announce a new journal section in Cortex:
*Methods and Assumptions* is open for submissions starting today!
Read the opening editorial here (OA): www.sciencedirect.com/science/arti... 1/
*Methods and Assumptions* is open for submissions starting today!
Read the opening editorial here (OA): www.sciencedirect.com/science/arti... 1/
Methods and Assumptions: A new section in Cortex
www.sciencedirect.com
April 3, 2025 at 2:36 PM
Chuffed and excited to announce a new journal section in Cortex:
*Methods and Assumptions* is open for submissions starting today!
Read the opening editorial here (OA): www.sciencedirect.com/science/arti... 1/
*Methods and Assumptions* is open for submissions starting today!
Read the opening editorial here (OA): www.sciencedirect.com/science/arti... 1/
Reposted by JK Flake
Attention metascience & AI enthusiasts: SSHRC is offering $140K/2yr postdoc fellowships, and I'd be happy to sponsor someone to work in my lab at UofT. If you're interested in how AI might transform scientific practice (for better or worse), apply by April 8. Let's question everything together.
March 6, 2025 at 11:10 PM
Attention metascience & AI enthusiasts: SSHRC is offering $140K/2yr postdoc fellowships, and I'd be happy to sponsor someone to work in my lab at UofT. If you're interested in how AI might transform scientific practice (for better or worse), apply by April 8. Let's question everything together.
Reposted by JK Flake
Please share! Not 100% official yet, but I may get to recruit a Ph.D. student for Fall 2025. If you are interested in psychological methods, math, and/or some combination of those, send me a message so we can chat!
March 13, 2025 at 1:16 PM
Please share! Not 100% official yet, but I may get to recruit a Ph.D. student for Fall 2025. If you are interested in psychological methods, math, and/or some combination of those, send me a message so we can chat!
I have to fix it in my head. This survey validates me.
March 25, 2025 at 10:54 PM
I have to fix it in my head. This survey validates me.
Reposted by JK Flake
I’ve long used FiveThirtyEight’s interactive “Hack Your Way To Scientific Glory” to illustrate the idea of p-hacking when I teach statistics. But ABC/Disney killed the site earlier this month :(
So I made my own with #rstats and Observable and #QuartoPub ! stats.andrewheiss.com/hack-your-way/
So I made my own with #rstats and Observable and #QuartoPub ! stats.andrewheiss.com/hack-your-way/
March 20, 2025 at 6:30 PM
I’ve long used FiveThirtyEight’s interactive “Hack Your Way To Scientific Glory” to illustrate the idea of p-hacking when I teach statistics. But ABC/Disney killed the site earlier this month :(
So I made my own with #rstats and Observable and #QuartoPub ! stats.andrewheiss.com/hack-your-way/
So I made my own with #rstats and Observable and #QuartoPub ! stats.andrewheiss.com/hack-your-way/
Hope to see more papers like this that question the performance and trade-offs of more complicated and newer methods with a sample of datasets (not just simulations that can benefit from cherry picking). Combats the publication bias towards new methods and demonstrates on-going comparative work.
🎓Paper time!✨ #ICLR spotlight.
Concluding of 5 years of research on missing values handling for prediction:
Beware of diminishing returns in imputation for prediction.
1/8
Concluding of 5 years of research on missing values handling for prediction:
Beware of diminishing returns in imputation for prediction.
1/8
March 18, 2025 at 2:18 PM
Hope to see more papers like this that question the performance and trade-offs of more complicated and newer methods with a sample of datasets (not just simulations that can benefit from cherry picking). Combats the publication bias towards new methods and demonstrates on-going comparative work.