Why the Economist is Wrong about Science Reproducibility

Phil Meagher
Post Image

Word about the Economist’s two posts on reproducibility spread fast through the JoVE office on Friday morning. And while we were delighted to see a major news outlet recognizing science’s reproducibility issue, the Economist failed to emphasize the true matter at hand. Science doesn’t have a scientist problem, science has a communication problem.

On October 17th and 19th, the Economist published a video and an article (“Is science wrong? Human after all,” and then “Unreliable research: Trouble at the lab”) discussing at length the problems behind Amgen and Bayer’s claims that only about 10-30% of published scientific experiments are reproducible. Correctly concluding that these findings present a serious problem for modern science, the article provides several reasons for low-reproducibility. But this is where the Economist gets it wrong.

The Economist states there are several reasons behind the reproducibility problem, but their main argument has to do with two causes. They point to career motivations that result in pressure to produce positive results, and they say scientists’ weak knowledge of statistics leads to the publishing of false-positive results. In other words, the journal states that scientists lie, and that scientists don’t know what they are doing.

But every practicing scientist knows that the reality is different. While it is true that there are some unethical individuals among scientists too, there is no massive fraud in academic research. And yes, some scientists, especially biologists, do not have a strong knowledge of statistical analysis, but you don’t have to be a statistician when you see a 4-fold difference between your experiment and the control to make a correct conclusion. The problem is that even when results are valid they are very difficult to reproduce—regardless of whether or not they were published in the most prestigious and rigorous science journals.

While traditional text-based science journals are sufficient to describe the results of scientific studies (for example, showing that gene X regulates cell-division in cancer cells), they are incompetent when it comes to unambiguously describing the methods used to achieve those results (how exactly cells and gene X were manipulated in the lab). Complex experiments can be reproduced only after visual demonstration by its original authors. Reproducibility comes with actually seeing how an experiment is performed—be it from flying across the world to have it demonstrated first hand, or from viewing it through video publication, for example, through JoVE. Unlike reading, visualization eliminates the errors of interpretation, or this case, misinterpretation.

“…[Video] is particularly important for observing experimental procedures, which usually are over-simplified in most of the traditional journals,” says Dr Pinfen Yang, Associate Professor, Marquette University, “In fact, only videos could adequately convey many procedures. For example, I learned how to inject zebrafish embryos and mate green algae by watching video published JoVE over and over again.”

The Economist could have found this simple truth only if their journalists bothered to speak with the graduate students and postdocs who actually do work in experimental labs with their own hands. Instead, they preferred to do the finger-pointing and offer administrative solutions that will not change the core problem—the format of science communication.

But just as the information age has shed light onto irreproducibility, it has shed light onto one  solution: video.