Replicability is often a misleading metric of progress or lack thereof (exceptions may exist but will be very specific) and a misguided aim for science. You can have replicability without knowledge or progress. And you can have knowledge without replicability.
May 27, 2025 at 9:31 PM
Replicability is often a misleading metric of progress or lack thereof (exceptions may exist but will be very specific) and a misguided aim for science. You can have replicability without knowledge or progress. And you can have knowledge without replicability.
lil ghost tackles the replicability crisis
September 22, 2026 at 2:38 PM
lil ghost tackles the replicability crisis
And not publishing in a peer reviewed journal so the claim can be tested for replicability.
September 22, 2025 at 6:48 PM
And not publishing in a peer reviewed journal so the claim can be tested for replicability.
Inspired by Gelman & Stern's "the difference between significant and not significant is not itself statistically significant," we show something similar for replicability. Results with mean replicability rates as different as 20% vs 80% cannot be reliably distinguished, even with more replications.
April 30, 2026 at 1:37 AM
Inspired by Gelman & Stern's "the difference between significant and not significant is not itself statistically significant," we show something similar for replicability. Results with mean replicability rates as different as 20% vs 80% cannot be reliably distinguished, even with more replications.
Replication remains valuable for evidence accumulation, calibration, and characterizing variability.
But replicability rate is NOT a demarcation criterion. Replicability rate-based verdicts on fields are INVALID.
If there is a crisis in science, it cannot be established by these methods.
But replicability rate is NOT a demarcation criterion. Replicability rate-based verdicts on fields are INVALID.
If there is a crisis in science, it cannot be established by these methods.
April 30, 2026 at 1:37 AM
Replication remains valuable for evidence accumulation, calibration, and characterizing variability.
But replicability rate is NOT a demarcation criterion. Replicability rate-based verdicts on fields are INVALID.
If there is a crisis in science, it cannot be established by these methods.
But replicability rate is NOT a demarcation criterion. Replicability rate-based verdicts on fields are INVALID.
If there is a crisis in science, it cannot be established by these methods.
As a scientist, replicability is so important. Can confirm: slappy end does not feel good.
Sharks very rarely injure humans, but when they do, most injuries originate from this part of a shark, the “bitey end.”
However, you should also be aware of the other part of a shark, the “slappy end.” Many marine biologists (including me) have been hurt far worse by that end!
However, you should also be aware of the other part of a shark, the “slappy end.” Many marine biologists (including me) have been hurt far worse by that end!
December 4, 2024 at 5:28 PM
As a scientist, replicability is so important. Can confirm: slappy end does not feel good.
I make open source software because I want:
- agency
- correctness
- transparency
- trustworthiness
- longevity-by-replicability
- collective self-sovereignty
- agency
- correctness
- transparency
- trustworthiness
- longevity-by-replicability
- collective self-sovereignty
that and like, the reason (imo) for open-source is because we want the thing to be correct
February 28, 2026 at 1:23 PM
I make open source software because I want:
- agency
- correctness
- transparency
- trustworthiness
- longevity-by-replicability
- collective self-sovereignty
- agency
- correctness
- transparency
- trustworthiness
- longevity-by-replicability
- collective self-sovereignty
There are many good arguments for why AI/ML will struggle to predict replicability.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
December 2, 2025 at 6:58 PM
There are many good arguments for why AI/ML will struggle to predict replicability.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
All our work on replicability in the last decade has taught me that it's not a good use of resources to try and predict replicability in the first place since it's neither a good goal to achieve nor diagnostic of specific problems. It's a search for shortcuts, easy answers to complex questions.
There are many good arguments for why AI/ML will struggle to predict replicability.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
www.pnas.org/doi/full/10....
I'm curious why, in light of these arguments and now data, COS is continuing to pursue this project.
December 3, 2025 at 2:29 AM
All our work on replicability in the last decade has taught me that it's not a good use of resources to try and predict replicability in the first place since it's neither a good goal to achieve nor diagnostic of specific problems. It's a search for shortcuts, easy answers to complex questions.
✨Our paper is out in Nature today! ✨
In this work, we investigated how we can leverage study designs to improve the replicability of brain-wide association studies (BWAS)
👇👇👇
www.nature.com/articles/s41...
In this work, we investigated how we can leverage study designs to improve the replicability of brain-wide association studies (BWAS)
👇👇👇
www.nature.com/articles/s41...
Study design features increase replicability in brain-wide association studies - Nature
Optimizing study design is critical for increasing standardized effect sizes and replicability, and the features that increase replicability in cross-sectional and longitudinal brain-wide association ...
www.nature.com
November 28, 2024 at 2:20 AM
✨Our paper is out in Nature today! ✨
In this work, we investigated how we can leverage study designs to improve the replicability of brain-wide association studies (BWAS)
👇👇👇
www.nature.com/articles/s41...
In this work, we investigated how we can leverage study designs to improve the replicability of brain-wide association studies (BWAS)
👇👇👇
www.nature.com/articles/s41...
Call for collaborators!
We are launching the Replicability Project: Health Behavior (RPHB) to replicate a sample of findings relevant to public health, epidemiology, health policy, and behavioral science.
Teams receive funding for conducting replication studies.
Details: www.cos.io/rphb
We are launching the Replicability Project: Health Behavior (RPHB) to replicate a sample of findings relevant to public health, epidemiology, health policy, and behavioral science.
Teams receive funding for conducting replication studies.
Details: www.cos.io/rphb
Replicability Project: Health Behavior (RPHB)
The Center for Open Science (COS) is launching the Replicability Project: Health Behavior (RPHB), which is a large-scale, multi-team effort to help support a transparent and trustworthy foundation in ...
www.cos.io
April 10, 2025 at 1:23 PM
Call for collaborators!
We are launching the Replicability Project: Health Behavior (RPHB) to replicate a sample of findings relevant to public health, epidemiology, health policy, and behavioral science.
Teams receive funding for conducting replication studies.
Details: www.cos.io/rphb
We are launching the Replicability Project: Health Behavior (RPHB) to replicate a sample of findings relevant to public health, epidemiology, health policy, and behavioral science.
Teams receive funding for conducting replication studies.
Details: www.cos.io/rphb
A short preprint describing our adoption of more consistent and precise use of terminology related to reproducibility, robustness, replicability, repeatability, and credibility.
osf.io/preprints/me...
osf.io/preprints/me...
December 3, 2025 at 6:51 PM
A short preprint describing our adoption of more consistent and precise use of terminology related to reproducibility, robustness, replicability, repeatability, and credibility.
osf.io/preprints/me...
osf.io/preprints/me...
it is funny that file replicability is probably one of the earliest potentially postcapitalist technologies to emerge but the entirety of the internet eventually became structured around being a privatized system of enclosures to constrain and prevent file "piracy" and free information access
September 1, 2025 at 1:42 PM
it is funny that file replicability is probably one of the earliest potentially postcapitalist technologies to emerge but the entirety of the internet eventually became structured around being a privatized system of enclosures to constrain and prevent file "piracy" and free information access
Our paper "High replicability of newly discovered
social-behavioral findings is achievable" appears today in Nature Human Behaviour.
Just as the large replications provided an inflection point in identifying replicability challenges, this paper does so for solutions.
www.nature.com/articles/s41...
social-behavioral findings is achievable" appears today in Nature Human Behaviour.
Just as the large replications provided an inflection point in identifying replicability challenges, this paper does so for solutions.
www.nature.com/articles/s41...
November 9, 2023 at 5:05 PM
Our paper "High replicability of newly discovered
social-behavioral findings is achievable" appears today in Nature Human Behaviour.
Just as the large replications provided an inflection point in identifying replicability challenges, this paper does so for solutions.
www.nature.com/articles/s41...
social-behavioral findings is achievable" appears today in Nature Human Behaviour.
Just as the large replications provided an inflection point in identifying replicability challenges, this paper does so for solutions.
www.nature.com/articles/s41...
Using the STRANGE framework to improve replicability in animal behaviour research @mikemwebster.bsky.social et al. show structured reporting on subjects' background, rearing & trappability helps explain mixed findings and boosts replicability www.sciencedirect.com/science/arti...
Using the STRANGE framework to improve replicability in animal behaviour research and allied fields
Many research fields rely on assessing the behaviour of wild or captive animals, using observational and experimental protocols. This work is highly s…
www.sciencedirect.com
September 23, 2026 at 11:03 AM
Using the STRANGE framework to improve replicability in animal behaviour research @mikemwebster.bsky.social et al. show structured reporting on subjects' background, rearing & trappability helps explain mixed findings and boosts replicability www.sciencedirect.com/science/arti...
Replicability doesn't mean accuracy though. It just means it's not inventing you a new lie but repeating one it already told.
May 2, 2025 at 1:37 AM
Replicability doesn't mean accuracy though. It just means it's not inventing you a new lie but repeating one it already told.
We’re all testing for replicability
So far it appears to be a success
So far it appears to be a success
November 7, 2025 at 7:14 PM
We’re all testing for replicability
So far it appears to be a success
So far it appears to be a success
Had a great time at the CSS workshop in Mannheim organized by @rubac.bsky.social & @valeriehase.bsky.social. You can find my slides on "Reproducibility & Replicability in Computational Social Science Research" here: doi.org/10.6084/m9.f...
#reproducibility #replicability #computationalsocialscience
#reproducibility #replicability #computationalsocialscience
Reproducibility & Replicability in Computational Social Science Research
Presentation on
doi.org
May 16, 2025 at 11:30 AM
Had a great time at the CSS workshop in Mannheim organized by @rubac.bsky.social & @valeriehase.bsky.social. You can find my slides on "Reproducibility & Replicability in Computational Social Science Research" here: doi.org/10.6084/m9.f...
#reproducibility #replicability #computationalsocialscience
#reproducibility #replicability #computationalsocialscience
- Replicability is not estimated consistently across studies. Different estimates have different properties and interpretations. Estimates vary greatly depending on the choice of replicability metric.
- Replications do not track truth but I've written a lot about this elsewhere so I'll stop here.
- Replications do not track truth but I've written a lot about this elsewhere so I'll stop here.
May 27, 2025 at 9:31 PM
- Replicability is not estimated consistently across studies. Different estimates have different properties and interpretations. Estimates vary greatly depending on the choice of replicability metric.
- Replications do not track truth but I've written a lot about this elsewhere so I'll stop here.
- Replications do not track truth but I've written a lot about this elsewhere so I'll stop here.
Love to see this conversation starting!
"Study design features increase replicability in brain-wide association studies"
Full article led by Kaidi Kang and Simon Vandekar + awesome News+Views by @roselynechauvin.bsky.social & @ndosenbach.bsky.social!
👇
www.nature.com/articles/s41...
#neuroskyence
"Study design features increase replicability in brain-wide association studies"
Full article led by Kaidi Kang and Simon Vandekar + awesome News+Views by @roselynechauvin.bsky.social & @ndosenbach.bsky.social!
👇
www.nature.com/articles/s41...
#neuroskyence
November 27, 2024 at 6:28 PM
Love to see this conversation starting!
"Study design features increase replicability in brain-wide association studies"
Full article led by Kaidi Kang and Simon Vandekar + awesome News+Views by @roselynechauvin.bsky.social & @ndosenbach.bsky.social!
👇
www.nature.com/articles/s41...
#neuroskyence
"Study design features increase replicability in brain-wide association studies"
Full article led by Kaidi Kang and Simon Vandekar + awesome News+Views by @roselynechauvin.bsky.social & @ndosenbach.bsky.social!
👇
www.nature.com/articles/s41...
#neuroskyence
The Graphics Replicability Stamp Initiative (GRSI, www.replicabilitystamp.org), a community-driven initiative to promote replicability in Graphics research, is seeking volunteers.
More details in the 'Volunteering' section of the home page.
More details in the 'Volunteering' section of the home page.
Graphics Replicability Stamp Initiative
www.replicabilitystamp.org
November 21, 2025 at 12:16 PM
The Graphics Replicability Stamp Initiative (GRSI, www.replicabilitystamp.org), a community-driven initiative to promote replicability in Graphics research, is seeking volunteers.
More details in the 'Volunteering' section of the home page.
More details in the 'Volunteering' section of the home page.
The political slant of psych research (e.g., left vs. right) is not related to replicability, although more slant (in any direction) may be associated with less replicability
journals.sagepub.com/doi/full/10....
journals.sagepub.com/doi/full/10....
May 30, 2025 at 8:26 PM
The political slant of psych research (e.g., left vs. right) is not related to replicability, although more slant (in any direction) may be associated with less replicability
journals.sagepub.com/doi/full/10....
journals.sagepub.com/doi/full/10....
Case in point, perhaps it's prudent to not conclude replicability has improved based on N=12, ignoring the standard error of the replicability estimates, varying populations, and inconsistent replicability metrics.
December 6, 2025 at 5:33 PM
Case in point, perhaps it's prudent to not conclude replicability has improved based on N=12, ignoring the standard error of the replicability estimates, varying populations, and inconsistent replicability metrics.
There is no "science-wide replication crisis" because every word in that phrase carries unfounded assumptions. 🧵
1. There is no evidence that progress has generally stalled in the sciences. Also, no one has actually tried to estimate replicability across fields.
1. There is no evidence that progress has generally stalled in the sciences. Also, no one has actually tried to estimate replicability across fields.
May 27, 2025 at 9:31 PM
There is no "science-wide replication crisis" because every word in that phrase carries unfounded assumptions. 🧵
1. There is no evidence that progress has generally stalled in the sciences. Also, no one has actually tried to estimate replicability across fields.
1. There is no evidence that progress has generally stalled in the sciences. Also, no one has actually tried to estimate replicability across fields.