In 2015, a team of 270 researchers tried to replicate 100 published psychology studies. Only 36 replicated cleanly. The result, published as the reproducibility crisis in research, sent a shock through every empirical discipline. I have spent the last decade working with psychometric questionnaires, and I can tell you the crisis did not surprise those of us close to the data. It should change how all of us do research, though — and a decade of reproducibility crisis research has shown me exactly how.
What reproducibility crisis research actually showed
The Open Science Collaboration’s landmark paper in Science was careful and devastating. Effect sizes in the replications were roughly half the size of the originals. It was not fraud, mostly. It was a system that rewarded novelty over verification: journals preferred surprising results, researchers ran small samples, and null findings went into file drawers. When Nature surveyed 1,576 researchers the following year, more than 70 percent had tried and failed to reproduce another scientist’s experiments, and more than half could not reproduce their own.
Why psychology was hit first
Psychology took the first hit because it was the first field to look honestly. The ingredients were all there: small samples, flexible analysis choices, publication bias toward positive results, and constructs measured with questionnaires that are noisier than we like to admit. Other fields have since found the same patterns. A 2022 review across disciplines estimated that irreproducible preclinical research costs the US alone around $28 billion a year. Cancer biology’s replication effort fared no better than psychology’s.
What I changed in my own work
Working on the PHQ-9 and DASS-21 psychometrics taught me where the bodies are buried. Here is what the crisis changed for me, concretely:
- Pre-registration: I write down the analysis plan before touching the data. This single habit eliminates the temptation to try twenty models and report the one that “worked”.
- Full reporting of factor analyses: too many papers report only the final clean model. I now report the messy intermediate steps, the dropped items, the fit indices that were ugly. Readers deserve to see the sausage being made.
- Sharing data and code: where ethics approval permits, anonymised data and analysis scripts go public with the paper. If I cannot share it, I say exactly why.
- Replication as a virtue: I now treat successful replication of my own earlier findings as a publishable result in its own right, not as boring housekeeping.
- Humility about p-values: a p of 0.04 in a small sample is a whisper, not a finding. Effect sizes, confidence intervals and Bayesian thinking now do the heavy lifting in my write-ups.
For a concrete example of how fragile published measures can be, see Validating a Mental Health Questionnaire: A Step-by-Step Guide.
What this means for readers of research
You do not need to become a statistician, but a few habits help. Distrust single studies, especially small surprising ones. Look for converging evidence across labs. Check whether the authors shared their data. And remember that “peer-reviewed” means “checked by two or three busy volunteers”, not “proven true”.
The bottom line
The reproducibility crisis was not the death of science; it was science working as intended, slowly and painfully correcting itself. The fields that faced it honestly are stronger now: pre-registration is normal, data sharing is expected, and replication studies are fundable. The crisis taught me that good research is not research that finds something exciting. Good research is research that someone else can check.
Frequently asked questions
What is the reproducibility crisis in simple terms?
It is the finding that a large fraction of published scientific results cannot be reproduced when other researchers repeat the experiments, usually because of small samples, publication bias and flexible analysis choices.
Is psychology research still trustworthy?
More so than before the crisis, arguably. The field adopted pre-registration, open data and replication as standard practice faster than most disciplines, and the strongest findings have survived repeated testing.
Sources
- Open Science Collaboration, Science (2015): Estimating the reproducibility of psychological science
- Nature (2016): 1,500 scientists lift the lid on reproducibility
- Freedman et al., PLOS Biology (2015): The economics of reproducibility in preclinical research