Data Vs. Methods: How to Increase Reproducibility?

Recently, I wrote about the low reproducibility of science articles, a problem plaguing every experimental scientist, and confirmed by massive studies taken by biotech/ pharma companies. There is a certain agreement that this phenomenon is due to the inadequate description of  methods and materials in science articles, and the obsession of science journals and funding bodies with data, often paying little attention to methods. There are no winners in the data vs. methods battle as data is not useful if it cannot be reproduced. In the end, science funding is wasted as scientists are in the endless cycle of learning and relearning techniques instead of innovating the field.

It is time to be productive and take initiative to fix the problem instead of adding to the cacophony of complaints. Let’s talk about some possible practical solutions:

1. Science journals should not limit the space devoted to materials and methods section. While this was a problem in the pre-Internet era, such limitations were justified by printing costs. They no longer make no sense for online publishing where printing costs are not as significant.
2. Methods sections should be required to include a step-by-step description of research procedures such as those available in methods journals (e.g. Nature Protocols, Current Protocols, JoVE, and others). When I co-founded JoVE, this was a priority and something I yearned for in my own research.
3. Funding bodies such as NIH should change the funding criteria to include a requirement for reproducibility of the research financed by the taxpayer money. This will further increase transparency associated with these grants.
4. Scientists should publish special articles on methods developed in their lab in a visualized format, similar to one applied by JoVE or, in a more limited fashion by medical journals New England Journal of Medicine and Journal of Bone & Joint Surgery. Visual demonstration is a better way to teach a new technique than just reading the text alone. Scientists mostly learn in the lab visually and by demonstration, for example they ask their more experienced colleagues to show them a new experimental technique

All of these solutions will be hard to implement. For solutions 1 and 2, a massive organizational change in the scientific publishing industry is required. Solution 3 seems to be even more difficult as it will require the government bureaucracy, which is not known for being the most flexible institution.

Solution 4 is dear to my heart as a co-founder of JoVE, and it may be the most effective, practical first step. There is also the possibility that I am biased, however JoVE was started to particularly address the issue of reproducibility. Video publication mimics the common “show-me” practice applied by scientists and students in the lab when they learn new methods. Video publication is much more effective than a text description, even the most detailed descriptions available. JoVE’s success shows that this new approach is accepted by scientists – at this time we publish 50 video articles per month. While JoVE is a valuable solution, what is needed is a cultural change by scientists and the institutions that fund them.

What do you think? Did I miss something? Let me know if you could think about another solution by commenting here or writing me directly at moshe.pritsker@jove.com.