Data vs. Methods: The Challenge of Article Reproducibility

Moshe Pritsker

The growing debate on reproducibility in science articles (10-30% according to the last studies) makes me think again about my days in the lab. As a grad student at Princeton in the early 2000s, I remember spending weeks trying again and again to reproduce a method to culture embryonic stem (ES) cells in a serum-free media, which was published in Cell (one of the most prestigious and selective journals in biological sciences). In the end I had to travel to the lab outside of USA that published the article to see how they do the experiment so I can learn.

Then, for my project on genetic screening in ES cells, I had to learn how to isolate a plasmid (a circular piece of DNA) from these cells. Getting plasmids out of bacteria is a common technique, but getting plasmids out of eukaryotic cells such as ES cells is a very specific case and different problem. There were a lot of references in the literature; however, most of them were giving partial information with further references to articles in obscure, hard to find journals. Compiling pieces of information together and repeating the experiment for a few weeks, I was able to master this method and get my experiment going.

Reproducibility remains a crucial part of maintaining reliability in science. Looking around myself I saw other scientists, post-docs and Ph.D. students suffering from the same problem. They came to science to solve great problems (cancer, stem cell therapy, etc…) and instead they were spending their time to reproduce experiments that sometimes were published 10-20 years ago.

What are the reasons for this phenomenon? Based on my own experience of working in different university labs for nearly 10 years and articles published on this subject elsewhere, I think the roots of the problem are in the current structure of scientific research and scientific publishing.

For most scientists, their work is paid by money from the government grants, distributed on a competitive basis through systems like the National Institute of Health (NIH). To obtain a grant, scientists have to impress their peers, especially those included on the grant committees. As such, they have to show that the results of their previous work were good, and that preliminary data indicates good results in the future.

The key words are “results” and “data”. Results of the scientists’ work are typically communicated through articles in scientific journals, employing special selection processes such as peer review. A publication in top tier journals (e.g. Nature, Science or Cell) is accepted as the most solid proof that a specific study generated novel and important scientific results.

These journals employ highly selective editorial and peer review processes and their article acceptance rate is often less than 10%. Submitted articles are judged based on their novelty, importance and validation. Often time, a publication in one of these journals (or lack of it) can determine the entire career of a scientist. As we all know it: “Publish or perish.”

 


In addition, describing novel concepts and technologies, a published article provides a foundation for work of other scientists. Sometimes, if published findings are considered interesting and important, experiments described in the article are attempted to be reproduced in multiple laboratories at different institutions to integrate the new knowledge and technologies in their work. Therefore the scientific publication serves two roles:

  • As a communication tool allowing transfer of knowledge between scientists
  • As a “currency of science” providing basis for distribution of funds and promotion

In summary, to be successful, a scientist is required to produce as much important scientific data as possible that would merit a publication in a prestigious scientific journal. This “publish or perish” system creates incentives and competition in the complex environment of academic science. Its success is proven by the rapid growth of scientific knowledge in the 20th century and until now. However, here comes the problem of reproducibility.

As every practicing experimental scientist knows it is often very difficult or even impossible to reproduce experiments described in published articles, even after multiple trials. Such inability to reproduce findings and experiments is mostly attributed to the lack of a super-special knowledge required to do this experiment. If this is the case, why is this knowledge is not fully included in the article?

Apparently, the system does not have a strong feedback mechanism that would pressure scientists, science journals or funding bodies to ensure the reproducibility of the published studies. One of the explanations is that science journals mainly focus on quality of data in submitted articles. At the same time, little attention is given to the quality of method’s description. However, to reproduce an experiment, having a clear and particular description of methods is necessary. What is wrong with publication of methods in scientific articles today? There are different opinions, see the list below:

  1. Many journals have limitation on the length of the article that limits the length of Methods section, preventing detailed description of the methods.

  2. Scientists are not incentivized to provide a high-quality description of the Methods as their submitted articles are judged based on the quality and importance of data

  3. Scientific publishing standards and requirements for originality of publication prevent authors from describing same methods in different articles. So often, the Material and Method section include descriptions like “The procedure was performed as described in Johnson et al (Journal of Something Important, 2005).” Such references are often given even parameters of the experiments in the two studies were “slightly” different.

  4. Authors are counter-incentivized to provide a clear description of their methods, as they are in competition with other scientists for money and recognition. Publishing data is sufficient to claim a specific scientific “territory”(e.g. I found gene XXX responsible for disease YYY), but avoiding publication of method enables them to create an effective “barrier to entry” for other scientists.

  5. The current text-based format of scientific articles is not sufficient to effectively transfer knowledge of complex experimental methods. Whenever possible, scientists prefer to learn experimental methods mainly by watching other scientists. New media-rich formats should be introduced into scientific publishing.

Being the co-founder and publisher of the world’s first scientific video journal (JoVE), I am obviously biased toward option 5 as a solution. But let’s talk about solutions in the next blog to avoid making this post too long.

In the meantime, what do you think? Any other ideas why the science article reproducibility is so low? Comment here or email me directly at moshe.pritsker@jove.com.