📢 Frequentist != NHST 📢
February 13, 2025 at 4:21 AM
📢 Frequentist != NHST 📢
good bayesians don't let their friends do NHST with bayes factors
"There are signs that the Bayes Factor is increasingly taking on a role analogous to that of the p-value, serving as a device for the automation of decisions rather than as a tool for substantive inference."
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
The Rise of Null Hypothesis Significance Testing (NHST): Institutional Massification and the Emergence of a Procedural Epistemology
It has long been a puzzle why, despite sustained reform efforts, many applied scientific fields remain dominated by Null Hypothesis Significance Testing (NHST), a framework that dichotomizes study res...
arxiv.org
March 17, 2026 at 4:54 PM
good bayesians don't let their friends do NHST with bayes factors
"There are signs that the Bayes Factor is increasingly taking on a role analogous to that of the p-value, serving as a device for the automation of decisions rather than as a tool for substantive inference."
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
The Rise of Null Hypothesis Significance Testing (NHST): Institutional Massification and the Emergence of a Procedural Epistemology
It has long been a puzzle why, despite sustained reform efforts, many applied scientific fields remain dominated by Null Hypothesis Significance Testing (NHST), a framework that dichotomizes study res...
arxiv.org
March 17, 2026 at 7:58 AM
"There are signs that the Bayes Factor is increasingly taking on a role analogous to that of the p-value, serving as a device for the automation of decisions rather than as a tool for substantive inference."
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
Excellent new preprint by Carol Ting on the rise of NHST.
#Methodology #stats
I was reminded today that it's important to be clear: p-values, NHST, frequentism... aren't the problem.
It's the culture of thoughtless uses of them.
It's the culture of thoughtless uses of them.
May 12, 2025 at 3:18 AM
I was reminded today that it's important to be clear: p-values, NHST, frequentism... aren't the problem.
It's the culture of thoughtless uses of them.
It's the culture of thoughtless uses of them.
#statstab #624 Severe Testing:
Error Statistics versus Bayes Factor Tests
Thoughts: This seems to be the framework most NHST researchers are adopting.
#nhst #hypothesis #hypothesistesting #severetestting #equivalence #sesoi #methods #frequentist
doi.org/10.1086/736950
Error Statistics versus Bayes Factor Tests
Thoughts: This seems to be the framework most NHST researchers are adopting.
#nhst #hypothesis #hypothesistesting #severetestting #equivalence #sesoi #methods #frequentist
doi.org/10.1086/736950
Severe Testing: Error Statistics versus Bayes Factor Tests | The British Journal for the Philosophy of Science: Vol 0, No ja
doi.org
September 24, 2026 at 6:25 PM
#statstab #624 Severe Testing:
Error Statistics versus Bayes Factor Tests
Thoughts: This seems to be the framework most NHST researchers are adopting.
#nhst #hypothesis #hypothesistesting #severetestting #equivalence #sesoi #methods #frequentist
doi.org/10.1086/736950
Error Statistics versus Bayes Factor Tests
Thoughts: This seems to be the framework most NHST researchers are adopting.
#nhst #hypothesis #hypothesistesting #severetestting #equivalence #sesoi #methods #frequentist
doi.org/10.1086/736950
my example intervals are all prime: 89, 97. none of my intervals are frequentist though so should never be used for sig tests.
but I do encourage other values just to force discourse about how arbitrary and unthinking scientists are about NHST. at this point it seems like the NHST brain worms won
but I do encourage other values just to force discourse about how arbitrary and unthinking scientists are about NHST. at this point it seems like the NHST brain worms won
August 13, 2025 at 6:13 AM
my example intervals are all prime: 89, 97. none of my intervals are frequentist though so should never be used for sig tests.
but I do encourage other values just to force discourse about how arbitrary and unthinking scientists are about NHST. at this point it seems like the NHST brain worms won
but I do encourage other values just to force discourse about how arbitrary and unthinking scientists are about NHST. at this point it seems like the NHST brain worms won
New paper: Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing, with @laabho.bsky.social and @jnwulff.bsky.social. We – strongly – push back on criticisms against NHST and argue for proper application. papers.ssrn.com/sol3/papers.... >
Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing
In empirical research, Null Hypothesis Significance Testing (NHST) is the most common methodology to interpret research findings and make inferences, particular
papers.ssrn.com
April 23, 2025 at 5:55 AM
New paper: Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing, with @laabho.bsky.social and @jnwulff.bsky.social. We – strongly – push back on criticisms against NHST and argue for proper application. papers.ssrn.com/sol3/papers.... >
So are we doing Bayes vs NHST next?
November 15, 2024 at 5:03 PM
So are we doing Bayes vs NHST next?
Sam Gershman writes beautifully about how theory-free neuroscience prevents the field from reaching its promise. Beautiful and true. Most folks do not test hypotheses. Running a NHST does not a hypothesis make. www.thetransmitter.org/theoretical-...
Breaking the barrier between theorists and experimentalists
Many neuroscience students are steeped in an experiment-first style of thinking. Let’s not forget how theory can guide experiments.
www.thetransmitter.org
February 24, 2025 at 6:13 PM
Sam Gershman writes beautifully about how theory-free neuroscience prevents the field from reaching its promise. Beautiful and true. Most folks do not test hypotheses. Running a NHST does not a hypothesis make. www.thetransmitter.org/theoretical-...
I believe at this time we can afford more nuance and precision than this. Most null results are uninformative because most experiments are uninformative because we use NHST to replace the scientific process. We can't keep avoiding the root problem forever. Also the focus on efficiency is jarring.
Scientists overwhelmingly recognize the value of sharing null results, but rarely publish them in the research literature
go.nature.com/450KElr
go.nature.com/450KElr
Researchers value null results, but struggle to publish them
Survey finds that fear of reputational harm and a lack of support and publication platforms are among respondents’ key concerns.
go.nature.com
July 23, 2025 at 4:27 PM
I believe at this time we can afford more nuance and precision than this. Most null results are uninformative because most experiments are uninformative because we use NHST to replace the scientific process. We can't keep avoiding the root problem forever. Also the focus on efficiency is jarring.
New pre-print out. We tackle the most common criticisms of NHST and show how to adress them. With NHST 🤯
New preprint with @lakens.bsky.social and @jnwulff.bsky.social: Blaming the Thermometer for the Fever: Separating Misapplication from Method in NHST
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing
In empirical research, Null Hypothesis Significance Testing (NHST) is the most common methodology to interpret research findings and make inferences, particular
ssrn.com
April 20, 2025 at 8:10 PM
New pre-print out. We tackle the most common criticisms of NHST and show how to adress them. With NHST 🤯
New preprint with @lakens.bsky.social and @jnwulff.bsky.social: Blaming the Thermometer for the Fever: Separating Misapplication from Method in NHST
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
Blaming the Thermometer for the Fever: Separating Misapplication from Method in Null Hypothesis Significance Testing
In empirical research, Null Hypothesis Significance Testing (NHST) is the most common methodology to interpret research findings and make inferences, particular
ssrn.com
April 20, 2025 at 6:29 PM
New preprint with @lakens.bsky.social and @jnwulff.bsky.social: Blaming the Thermometer for the Fever: Separating Misapplication from Method in NHST
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
NHST gets blamed—mostly for problems it did not cause.
We separate misuse from method and suggest potential paths forward
ssrn.com/abstract=521...
#statstab #335 Bayesian New Statistics
Thoughts: An influential paper with a great overview of different approaches to research.
#bayesian #nhst #nhbt #estimation #testing #frequentist
link.springer.com/content/pdf/...
Thoughts: An influential paper with a great overview of different approaches to research.
#bayesian #nhst #nhbt #estimation #testing #frequentist
link.springer.com/content/pdf/...
May 2, 2025 at 6:20 PM
#statstab #335 Bayesian New Statistics
Thoughts: An influential paper with a great overview of different approaches to research.
#bayesian #nhst #nhbt #estimation #testing #frequentist
link.springer.com/content/pdf/...
Thoughts: An influential paper with a great overview of different approaches to research.
#bayesian #nhst #nhbt #estimation #testing #frequentist
link.springer.com/content/pdf/...
Things that apparently manage to mess with the brains of otherwise smart psychological researchers:
- NHST
- the “crud factor”
- vague appeals to “theory”
- NHST
- the “crud factor”
- vague appeals to “theory”
November 16, 2024 at 3:33 PM
Things that apparently manage to mess with the brains of otherwise smart psychological researchers:
- NHST
- the “crud factor”
- vague appeals to “theory”
- NHST
- the “crud factor”
- vague appeals to “theory”
It’s insane that I got all the way through getting a PhD before I encountered equivalence tests. These should be part of every intro stats curriculum that covers NHST.
#statstab #205 Equivalence Tests Using {marginaleffects}
Thoughts: Easily testing for "no effect" is important. Here's a tutorial by @carlislerainey using @VincentAB r pkg.
#equivalencetests #equivalence #TOST #marginaleffects #tutorial #nullresults #r
www.carlislerainey.com/blog/2023-08...
Thoughts: Easily testing for "no effect" is important. Here's a tutorial by @carlislerainey using @VincentAB r pkg.
#equivalencetests #equivalence #TOST #marginaleffects #tutorial #nullresults #r
www.carlislerainey.com/blog/2023-08...
Equivalence Tests with {marginaleffects}
Reproducing the Clark and Golder (2006) example from Rainey (2014)
www.carlislerainey.com
October 18, 2024 at 11:28 PM
It’s insane that I got all the way through getting a PhD before I encountered equivalence tests. These should be part of every intro stats curriculum that covers NHST.
A lot has been written on the ills of NHST rituals. But has anyone specifically studied the kind of knowledge/evidence/epistemic progress wasted in studies that are exclusively subjected to NHST? In other words, what are we not learning that we could/should have been learning from experiments?
August 21, 2025 at 2:35 PM
A lot has been written on the ills of NHST rituals. But has anyone specifically studied the kind of knowledge/evidence/epistemic progress wasted in studies that are exclusively subjected to NHST? In other words, what are we not learning that we could/should have been learning from experiments?
Has anyone tried teaching *only* Bayesian inferential #stats for undergrad psych (or other UG sosci programs)?
Like completely foregoing NHST, confidence whatevers, p values, CLT...
I'd be very interested in seeing a syllabus of such a course/ program!
Like completely foregoing NHST, confidence whatevers, p values, CLT...
I'd be very interested in seeing a syllabus of such a course/ program!
March 5, 2026 at 8:53 AM
Has anyone tried teaching *only* Bayesian inferential #stats for undergrad psych (or other UG sosci programs)?
Like completely foregoing NHST, confidence whatevers, p values, CLT...
I'd be very interested in seeing a syllabus of such a course/ program!
Like completely foregoing NHST, confidence whatevers, p values, CLT...
I'd be very interested in seeing a syllabus of such a course/ program!
In my experience, people doing NHST very, very seldom actually think about what it is they are testing
July 23, 2025 at 4:46 AM
In my experience, people doing NHST very, very seldom actually think about what it is they are testing
Teaching only (or mainly) NHST is especially horrible these days when “big data” is super prevalent.
You’re going to end up with people who are frustrated that p-values are the only tool they have in analyses of datasets with millions and millions of rows.
You’re going to end up with people who are frustrated that p-values are the only tool they have in analyses of datasets with millions and millions of rows.
September 16, 2025 at 3:50 AM
Teaching only (or mainly) NHST is especially horrible these days when “big data” is super prevalent.
You’re going to end up with people who are frustrated that p-values are the only tool they have in analyses of datasets with millions and millions of rows.
You’re going to end up with people who are frustrated that p-values are the only tool they have in analyses of datasets with millions and millions of rows.
Change my mind: with current statistical software there is no need to report t/z statistics for frequentist NHST tests. Just show me the p-value, people rarely need to look things up in tables so just skip ‘em, you won’t lose anything.
August 9, 2025 at 7:17 PM
Change my mind: with current statistical software there is no need to report t/z statistics for frequentist NHST tests. Just show me the p-value, people rarely need to look things up in tables so just skip ‘em, you won’t lose anything.
There's also the related tendency to want to prune all small parameters in the name of parsimony and NHST
December 24, 2024 at 2:57 PM
There's also the related tendency to want to prune all small parameters in the name of parsimony and NHST
I have also thought this point (and the related posts above/below) matters. What is the role of journals, really? Why should we expect them to be indifferent to the nature of evidence? (I'm setting aside all issues regarding NHST here and highlighting something more general.) Great questions to ask.
Is that the purpose of journals? I guess that's why I'm confused. I feel like journals should contain the most useful and informative studies, not all studies ever done. That does mean selecting on precision, not significance, but doesn't mean journals should be representative of all studies.
February 13, 2026 at 12:41 AM
I have also thought this point (and the related posts above/below) matters. What is the role of journals, really? Why should we expect them to be indifferent to the nature of evidence? (I'm setting aside all issues regarding NHST here and highlighting something more general.) Great questions to ask.
What is the "null" in NHST even for??
🤦
🤦
February 11, 2026 at 6:00 PM
What is the "null" in NHST even for??
🤦
🤦
The sad part is that the metascience framework popularized in psychology quite literally formalizes NHST as a model of scientific practice. No lessons learned
You can put a lot of takes on this project, but here's mine:
Null hypothesis significance testing is an abject failure at providing any sort of useful framework for statistical research in the social sciences.
Null hypothesis significance testing is an abject failure at providing any sort of useful framework for statistical research in the social sciences.
February 11, 2026 at 6:57 PM
The sad part is that the metascience framework popularized in psychology quite literally formalizes NHST as a model of scientific practice. No lessons learned