New paper! Despite qualitative researchers claiming that generalizability is not a goal, their published work does indeed contain generalizable claims. We document this and provide recommendations for how all researchers can better calibrate their generalizability claims. doi.org/10.1177/2515...
September 22, 2026 at 4:08 PM
New paper! Despite qualitative researchers claiming that generalizability is not a goal, their published work does indeed contain generalizable claims. We document this and provide recommendations for how all researchers can better calibrate their generalizability claims. doi.org/10.1177/2515...
The findings in one figure: nearly every article included all three types of generalizability claims.
September 22, 2026 at 4:08 PM
The findings in one figure: nearly every article included all three types of generalizability claims.
Yes, we cover transferability in the paper as well. Technically different from statistical-probabilistic generalizability, but often sort of lumped together under "generalizability"
September 22, 2026 at 6:13 PM
Yes, we cover transferability in the paper as well. Technically different from statistical-probabilistic generalizability, but often sort of lumped together under "generalizability"
Can AI simulations of human research participants advance cognitive science? In @cp-trendscognsci.bsky.social, @lmesseri.bsky.social & I analyze this vision. We show how “AI Surrogates” entrench practices that limit the generalizability of cognitive science while aspiring to do the opposite. 1/
AI Surrogates and illusions of generalizability in cognitive science
Recent advances in artificial intelligence (AI) have generated enthusiasm for using AI simulations of human research participants to generate new know…
www.sciencedirect.com
October 21, 2025 at 8:24 PM
Can AI simulations of human research participants advance cognitive science? In @cp-trendscognsci.bsky.social, @lmesseri.bsky.social & I analyze this vision. We show how “AI Surrogates” entrench practices that limit the generalizability of cognitive science while aspiring to do the opposite. 1/
New paper on the generalizability of IR experiments 🌐📊
Just published on APSR First View: "The Generalizability of IR Experiments beyond the United States " by Lotem Bassan-Nygate, Jonathan Renshon, Jessica Weeks, and Chagai Weiss. www.cambridge.org/core/journal...
December 24, 2024 at 4:13 AM
New paper on the generalizability of IR experiments 🌐📊
You may have heard of Questionable Research Practices and "HARKing" as threats to replicability.
In a new paper led by @sakshighai.bsky.social, we argue that a similar practices -- Questionable Generalizability Practices and "MASKing" -- undermine generalizability.
www.nature.com/articles/s44...
In a new paper led by @sakshighai.bsky.social, we argue that a similar practices -- Questionable Generalizability Practices and "MASKing" -- undermine generalizability.
www.nature.com/articles/s44...
January 30, 2025 at 6:34 PM
You may have heard of Questionable Research Practices and "HARKing" as threats to replicability.
In a new paper led by @sakshighai.bsky.social, we argue that a similar practices -- Questionable Generalizability Practices and "MASKing" -- undermine generalizability.
www.nature.com/articles/s44...
In a new paper led by @sakshighai.bsky.social, we argue that a similar practices -- Questionable Generalizability Practices and "MASKing" -- undermine generalizability.
www.nature.com/articles/s44...
Do big team science studies guarantee the global generalizability of findings?
At the risk of not overgeneralizing ourselves, we reanalysed one big team science study on temporal discounting.
Together with peerless @psforscher.bsky.social & @hcp4715.bsky.social
www.nature.com/articles/s41...
At the risk of not overgeneralizing ourselves, we reanalysed one big team science study on temporal discounting.
Together with peerless @psforscher.bsky.social & @hcp4715.bsky.social
www.nature.com/articles/s41...
Big-team science does not guarantee generalizability - Nature Human Behaviour
Nature Human Behaviour - Big-team science does not guarantee generalizability
www.nature.com
June 7, 2024 at 9:46 AM
Do big team science studies guarantee the global generalizability of findings?
At the risk of not overgeneralizing ourselves, we reanalysed one big team science study on temporal discounting.
Together with peerless @psforscher.bsky.social & @hcp4715.bsky.social
www.nature.com/articles/s41...
At the risk of not overgeneralizing ourselves, we reanalysed one big team science study on temporal discounting.
Together with peerless @psforscher.bsky.social & @hcp4715.bsky.social
www.nature.com/articles/s41...
someone pass me the Mikes Hard Conversations About The Non-Generalizability of Automation
Nvidia announcing a "moonshot" for embodied "human-level" "AI," with a cute, easily linked-into, culturally resonant name ("GR00T"). I am pre-exhausted.
arstechnica.com/information-...
arstechnica.com/information-...
Nvidia announces “moonshot” to create embodied human-level AI in robot form
As companies race to pair AI with general-purpose humanoid robots, Nvidia's GR00T emerges.
arstechnica.com
March 20, 2024 at 9:14 PM
someone pass me the Mikes Hard Conversations About The Non-Generalizability of Automation
New paper alert! In this paper we were interested to formally test some of the assumptions made by computational psychiatrists - assumptions about parameter recoverability, reliability and validity. www.cambridge.org/core/journal...
The recoverability, reliability, and generalizability of reward processing parameters and relation to mental health symptoms | Psychological Medicine | Cambridge Core
The recoverability, reliability, and generalizability of reward processing parameters and relation to mental health symptoms - Volume 56
www.cambridge.org
July 28, 2026 at 4:05 PM
New paper alert! In this paper we were interested to formally test some of the assumptions made by computational psychiatrists - assumptions about parameter recoverability, reliability and validity. www.cambridge.org/core/journal...
🧵 Can you predict whether behavioral interventions will work in new contexts?
… probably not, right?
In our new paper with @dggoldst.bsky.social @dilipsoman.bsky.social @susanmichie.bsky.social, we dive into how to better understand and improve generalizability. rdcu.be/ewD2z
Summary👇
… probably not, right?
In our new paper with @dggoldst.bsky.social @dilipsoman.bsky.social @susanmichie.bsky.social, we dive into how to better understand and improve generalizability. rdcu.be/ewD2z
Summary👇
Generalizability of choice architecture interventions
Nature Reviews Psychology - Choice architecture interventions (or ‘nudges’) aim to guide behaviour by changing the proximal physical, social or psychological environment. In this...
rdcu.be
July 28, 2025 at 10:53 AM
🧵 Can you predict whether behavioral interventions will work in new contexts?
… probably not, right?
In our new paper with @dggoldst.bsky.social @dilipsoman.bsky.social @susanmichie.bsky.social, we dive into how to better understand and improve generalizability. rdcu.be/ewD2z
Summary👇
… probably not, right?
In our new paper with @dggoldst.bsky.social @dilipsoman.bsky.social @susanmichie.bsky.social, we dive into how to better understand and improve generalizability. rdcu.be/ewD2z
Summary👇
Very important considerations as we think about issues of replication, generalizability, and the desiderata of our scientific activity.
Trace contaminants in media can have significant effects on microbial growth! Here's the latest: www.nature.com/articles/s41...
Several years ago (shit, nearly 10) we showed that a trace contaminant of a manufacturing precursor in thiamin was required for SAR11: www.nature.com/articles/ism...
Several years ago (shit, nearly 10) we showed that a trace contaminant of a manufacturing precursor in thiamin was required for SAR11: www.nature.com/articles/ism...
October 14, 2023 at 10:50 PM
Very important considerations as we think about issues of replication, generalizability, and the desiderata of our scientific activity.
New paper just dropped:
Szaszi, Barnabas, Daniel G. Goldstein, Dilip Soman, Susan Michie. (2025). Generalizability of Choice Architecture Interventions. Nature Reviews Psychology.
Thanks to @szaszibarnabas.bsky.social for doing the lion's share!
www.nature.com/articles/s44...
Szaszi, Barnabas, Daniel G. Goldstein, Dilip Soman, Susan Michie. (2025). Generalizability of Choice Architecture Interventions. Nature Reviews Psychology.
Thanks to @szaszibarnabas.bsky.social for doing the lion's share!
www.nature.com/articles/s44...
Generalizability of choice architecture interventions - Nature Reviews Psychology
Choice architecture interventions (or ‘nudges’) aim to guide behaviour by changing the proximal physical, social or psychological environment. In this Review, Szaszi and colleagues show that the avera...
www.nature.com
July 23, 2025 at 6:21 PM
New paper just dropped:
Szaszi, Barnabas, Daniel G. Goldstein, Dilip Soman, Susan Michie. (2025). Generalizability of Choice Architecture Interventions. Nature Reviews Psychology.
Thanks to @szaszibarnabas.bsky.social for doing the lion's share!
www.nature.com/articles/s44...
Szaszi, Barnabas, Daniel G. Goldstein, Dilip Soman, Susan Michie. (2025). Generalizability of Choice Architecture Interventions. Nature Reviews Psychology.
Thanks to @szaszibarnabas.bsky.social for doing the lion's share!
www.nature.com/articles/s44...
This book changed my mind about what a random-effects meta-analysis says about generalizability. I used to think it could *show* generalizability, now I understand it *asserts* generalizability.
So happy to announce that my book with @tara-slough.bsky.social is now out at Cambridge Elements!!
External Validity and Evidence Accumulation
Link: www.cambridge.org/core/element...
External Validity and Evidence Accumulation
Link: www.cambridge.org/core/element...
External Validity and Evidence Accumulation
Cambridge Core - Political Theory - External Validity and Evidence Accumulation
www.cambridge.org
December 5, 2024 at 5:14 PM
This book changed my mind about what a random-effects meta-analysis says about generalizability. I used to think it could *show* generalizability, now I understand it *asserts* generalizability.
We write, the “conditions upon which AI tools are built inherently exclude the same people who are already underrepresented in CogSci… AI Surrogates perpetuate an illusion of generalizability where researchers believe these models represent a broader swathe of humanity than they actually do.” 5/
October 21, 2025 at 3:47 PM
We write, the “conditions upon which AI tools are built inherently exclude the same people who are already underrepresented in CogSci… AI Surrogates perpetuate an illusion of generalizability where researchers believe these models represent a broader swathe of humanity than they actually do.” 5/
See discussion here - when thinking about external validity (whether a solution in one domain will work in a second domain) you want to think through the *mechanisms*. ssir.org/articles/ent...
The Generalizability Puzzle (SSIR)
Rigorous impact evaluations tell us a lot about the world, not just the particular contexts in which they are conducted. Open access to this article is made possible by MIT.
ssir.org
September 22, 2026 at 3:32 PM
See discussion here - when thinking about external validity (whether a solution in one domain will work in a second domain) you want to think through the *mechanisms*. ssir.org/articles/ent...
AI Surrogates are offered as an efficient solution. But this technology ultimately entrenches the illusion of generalizability because it inherently excludes the same people who are already underrepresented in cognitive science. 5/
October 21, 2025 at 8:24 PM
AI Surrogates are offered as an efficient solution. But this technology ultimately entrenches the illusion of generalizability because it inherently excludes the same people who are already underrepresented in cognitive science. 5/
I really love @carlzimmer.com’s reporting but at some point in time we have to realize the “science is broken and dying” hyperbole is overselling results the same as we see anywhere else. It puts together loose observations from unique contexts and ignores mechanisms and generalizability.
August 6, 2025 at 12:40 PM
I really love @carlzimmer.com’s reporting but at some point in time we have to realize the “science is broken and dying” hyperbole is overselling results the same as we see anywhere else. It puts together loose observations from unique contexts and ignores mechanisms and generalizability.
Second, individual laboratory experiments probe an insufficient range of stimuli and contexts to defend broad claims about cognition and behavior. Some describe this as a “generalizability crisis”. We trace the history of this critique... 6/
October 21, 2025 at 8:24 PM
Second, individual laboratory experiments probe an insufficient range of stimuli and contexts to defend broad claims about cognition and behavior. Some describe this as a “generalizability crisis”. We trace the history of this critique... 6/
How outrage drives the spread of online misinformation.
New work from @killianmcloughlin.bsky.social with Bluesky luminaries @klonick.bsky.social, @mjcrockett.bsky.social, @williambrady.bsky.social and colleagues.
www.science.org/doi/10.1126/science.adl2829
New work from @killianmcloughlin.bsky.social with Bluesky luminaries @klonick.bsky.social, @mjcrockett.bsky.social, @williambrady.bsky.social and colleagues.
www.science.org/doi/10.1126/science.adl2829
Misinformation exploits outrage to spread online
We tested a hypothesis that misinformation exploits outrage to spread online, examining generalizability across multiple platforms, time periods, and classifications of misinformation. Outrage is high...
www.science.org
November 29, 2024 at 6:13 PM
How outrage drives the spread of online misinformation.
New work from @killianmcloughlin.bsky.social with Bluesky luminaries @klonick.bsky.social, @mjcrockett.bsky.social, @williambrady.bsky.social and colleagues.
www.science.org/doi/10.1126/science.adl2829
New work from @killianmcloughlin.bsky.social with Bluesky luminaries @klonick.bsky.social, @mjcrockett.bsky.social, @williambrady.bsky.social and colleagues.
www.science.org/doi/10.1126/science.adl2829
Very few of them are actual numbers people is the problem. If they understood a single thing about validity and generalizability they wouldn’t be making such confident pronouncements. Moneyball and sports betting has ruined quant and filled it up with degenerate gamblers with zero regard for truth.
October 23, 2025 at 2:19 AM
Very few of them are actual numbers people is the problem. If they understood a single thing about validity and generalizability they wouldn’t be making such confident pronouncements. Moneyball and sports betting has ruined quant and filled it up with degenerate gamblers with zero regard for truth.
It feels a certain way knowing that @mjcrockett.bsky.social and I published the rebuttal to this article… six months ago.
July 9, 2026 at 12:35 PM
It feels a certain way knowing that @mjcrockett.bsky.social and I published the rebuttal to this article… six months ago.
👉 "We ... consider two requirements of group-to-individual generalizability ... and suggest that most processes in cognitive neuroscience do not meet these assumptions. Consequently, interindividual findings are inappropriate for intraindividual inferences"
www.sciencedirect.com/science/arti...
www.sciencedirect.com/science/arti...
Group-to-individual generalizability and individual-level inferences in cognitive neuroscience
Much of cognitive neuroscience research is focused on group-averages and interindividual brain-behavior associations. However, many theories core to t…
www.sciencedirect.com
February 3, 2025 at 10:43 PM
👉 "We ... consider two requirements of group-to-individual generalizability ... and suggest that most processes in cognitive neuroscience do not meet these assumptions. Consequently, interindividual findings are inappropriate for intraindividual inferences"
www.sciencedirect.com/science/arti...
www.sciencedirect.com/science/arti...
Find the paper here, and summary of findings next!
www.science.org/doi/10.1126/...
www.science.org/doi/10.1126/...
Misinformation exploits outrage to spread online
We tested a hypothesis that misinformation exploits outrage to spread online, examining generalizability across multiple platforms, time periods, and classifications of misinformation. Outrage is high...
www.science.org
November 28, 2024 at 7:07 PM
Find the paper here, and summary of findings next!
www.science.org/doi/10.1126/...
www.science.org/doi/10.1126/...
Today I officially start at Northwestern! I am so happy to be here and so grateful to everyone who made this possible
I've got
new courses on persuasion and replication to teach
a new research agenda on generalizability and meta-analysis
new collaborators to find
Yale gratitude posts to follow...
I've got
new courses on persuasion and replication to teach
a new research agenda on generalizability and meta-analysis
new collaborators to find
Yale gratitude posts to follow...
July 1, 2025 at 1:46 PM
Today I officially start at Northwestern! I am so happy to be here and so grateful to everyone who made this possible
I've got
new courses on persuasion and replication to teach
a new research agenda on generalizability and meta-analysis
new collaborators to find
Yale gratitude posts to follow...
I've got
new courses on persuasion and replication to teach
a new research agenda on generalizability and meta-analysis
new collaborators to find
Yale gratitude posts to follow...
Generalizability problems arise at three stages of experimental design. Here's how we can fix the issues
go.nature.com/4v0pZJm
go.nature.com/4v0pZJm
Make science more reliable: study people as they go about their lives
Researchers have long discussed the replication crisis. Now it’s time to tackle another problem in the behavioural sciences: generalizability.
go.nature.com
June 22, 2026 at 12:58 PM
Generalizability problems arise at three stages of experimental design. Here's how we can fix the issues
go.nature.com/4v0pZJm
go.nature.com/4v0pZJm