Nature’s news article published a few days ago shows that the debate on the reproducibility of scientific experiments has reached a new level. Apparently, the issue got the attention of the science leadership at the National Institutes of Health (NIH), and they are currently discussing potential solutions. This is very important as the NIH is the main funder of academic biomedical research in US, distributing nearly $31 billion per year in research grants.
I previously wrote about the hair-rising figures on the reproducibility of scientific experiments. It appears that only 10% to 30% of published scientific studies are actually reproducible. Epithets like “crazy”, “unbelievable” and “impossible” pop up in the mind of every scientist when they read about these numbers, because science is not science without reproducibility.
The two main projects that determined these crazy numbers were conducted by the pharma/biotech companies Amgen (California) and Bayer (Germany). More than a year ago,their studies were published in Nature, and further publicized in the Wall Street Journal and other general media. Back then, contrary to my expectations, the skies did not immediately fall on the earth. The reaction in the academic community was quite mute, there was no outcry among the science-related organizations, and business went as usual. It seems that everyone pretended that nothing serious happened and the science debate went back to its familiar venues of “not enough funding for science,” “we don’t have enough PhDs,” “open access vs. subscriptions” and so on, and nobody actually cares about the new evidence that put into question the entire foundation on which the science and scientists exist. Some articles cutely called it science’s “dirty little secret.”
However, it seems that the reproducibility problem was not missed by the NIH leaders and they are now gathering information to determine the next steps. The Nature article brings a few serious names including Francis Collins, the current Director of NIH and one of the leading people behind the human genome sequencing, and Harold Varmus, the former NIH director, a Nobel Prize winner, who is also a co-founder of the successful open access publication PLoS. The potential solutions mentioned include random audits of published studies. This would be interesting! An introduction of such new methods of quality control may create significant changes into the practice of science and scientific publication. It remains unclear who will pay for such audits, for which the cost is estimated to reach $25,000 per published study. Given nearly 1,000,000 biomedical articles published every year, you do the math…
Here at JoVE, we have dealt with the problem of reproducibility since 2006. Actually, our main mission is “to increase reproducibility and productivity of scientific research using video publication.” As an ideal solution, we would offer every published study to be accompanied by a video-demonstration of the methods and procedures used in this study. This would increase both reproducibility and productivity at the same time, as video allows more reliable and faster transfer of knowledge than the traditional text format. Maybe it is something we need to offer to NIH.
Blog