Why Most Research Findings Are False

Dr. John P. A. Ioannidis is a Professor of Medicine, Health Research and Policy at Stanford University.  He holds the C.F. Rehnborg Chair in Disease Prevention and is the director of the Stanford Prevention Research Center (SPRC) at Stanford University School of Medicine.  He is a professor of Statistics and one of two directors of the Meta-Research Innovation Center at Stanford (METRICS).

In an article published in 2005, titled "Why Most Research Findings Are False", Dr. Ioannidis explains the increasing concern that most current published research findings are false.  There are many factors that play into this problem such as research design, definitions, outcomes, financial interests, prejudice, and the "chase for statistical significance".  In fact, Dr. Ioannidis states that for many current scientific fields, published research findings may be little more than "accurate measures of the prevailing bias".

Dr. Ioannidis and other methodologists, have pointed out that the high rate of non-replication of research discoveries is based on a convenient, yet ill-founded strategy of claiming conclusive research results after a single study where statistical significance is claimed based on the p-value of <0 .05.="" p="">

Dr Ioannidis is known for defining the proteus phenomenon, which is the greater tendency for the early repeat studies done to contradict the original study.  The proteus phenomenon was named for the Greek God Proteus, who had the ability to change form.

Research findings are less likely to be true in research studies with:
1. Smaller sample size
2. Smaller the effect size
3. Greater the number and lesser the selection of tested relationships
4. Greater the flexibility in designs, definitions, outcomes and analytical modes
5. Greater the financial and other interests and prejudices
6. Hotter the scientific field (more teams interested in studying a field)

While it is impossible to know with 100% certainty what the truth is in any research question, Dr. Ioannidis makes some recommendations on how we can improve the scientific research.

1. Seek better powered evidence (large studies or low-bias meta analysis) with acknowledgment and avoidance of bias.
2. Find ways to evaluate the all the evidence generated by multiple teams studying the same topic.  Find ways to develop and follow protocols as done in randomized clinical trials.
3. Rather than focusing on p-value for statistical significance, improve the understanding of the range of R values - pre-study odds.  This helps us understand the chances that we are testing a true, rather than an un-true relationship.

Dr. Ioannidis is not the only one concerned about the effect of bias in the research.


Pannuchi and Wilkens have said "Bias can occur in the planning, data collection, analysis, and publication phases of research. Understanding research bias allows readers to critically and independently review the scientific literature and avoid treatments which are suboptimal or potentially harmful. A thorough understanding of bias and how it affects study results is essential for the practice of evidence-based medicine."  They go on to discuss how bias can affect a study before, during and after the research is complete.

An article in Radiology by Gregory Sica MD, MPH reviews types of study design, types of selection bias, types observational bias, and suggests some ways to reduce bias in research.

Mullane and Williams ask if bias in biomedical research has become the rule rather than the expectation.  They indicate that bias in research is amplified by competition and difficulty getting funding, pressures for maintaining laboratories and staff, desire for career advancement, and monetization of science for personal gain.  They also mention the increasing body of research publications that cannot be repeated or require corrections or retractions.

Other Resources
Bias in Research (Sandra Burge PhD) - an online learning module from UTHSC San Antonio
Continuum Between Fraud and "Sloppy Science" - Scientific American

Dr. Ioannidis will be the keynote speaker at the Nowzari Symposium on Friday, Nov. 7th 2014 in Beverly Hills, CA.  See you there!

Source:

1.  Ioannidis JPA (2005) Why Most Published Research Findings Are False. PLoS Med 2(8): e124. doi:10.1371/journal.pmed.0020124
2. Pannuchi, C Wilkens, E Identifying and Avoiding Bias In Research. Plast Reconstr Surg. Aug 2010; 126(2):619-625.
3. Sica, G Bias In Research Studies. Radiology. Mar 2006; 238:3.

4. Mullane, Williams. Bias in Research: The Rule Rather Than The Exception. Editors Update. Sep 2013. Issue 40.
5. Moonesinghe R, Khoury MJ, Janssens ACJW (2007) Most Published Research Findings Are False—But a Little Replication Goes a Long Way. PLoS Med 4(2): e28. doi:10.1371/journal.pmed.0040028