More scrutiny is needed of irreproducible science

Science goes forward by building brick-by-brick on what went before. That’s why it is a problem if experiments performed in one laboratory cannot be replicated in another.

In April, a group of influential UK biomedical funding agencies, including the Academy of Medical Sciences, the Biotechnology and Biological Sciences Research Council, Medical Research Council (MRC) and the Wellcome Trust, held a meeting to discuss the problem, and last week released their findings.

Evidence is mounting that an increasing volume of scientific research is irreproducible. A recent study by the Open Science Foundation, published in the journal Science this year, described attempts to replicate the findings of 100 papers in psychological research journals. While 97 of the original studies reported significant results, this was only the case in 36 of the repeated experiments.

According to the report, “if too many results are irreproducible, it could hinder scientific progress, delay translation into clinical applications and waste valuable resource. It also threatens the reputation of biomedical science and the public’s trust in its findings.”

Science “waste” is the kind of thing that is hard to put a figure on, but there has been a recent attempt in the US. In June, one analysis said as much as $28 billion is spent each year on preclinical research that cannot be reproduced by other researchers. While some challenged the number, saying it overstated the extent of any problem, it was effective in drawing attention to the issue.

Irreproducible data is not necessarily a smoking gun for fraud, said Jim Smith, MRC deputy chief executive and director of strategy. “It doesn’t even mean the research is wrong, or was badly carried out either.”

It just might be that scientists have not explained how they set up their experiments in enough detail. Smith thinks researchers should err on the side of telling us more and more about the way they approached their research. 

“We need to hear more. Because you never know for sure what is an important detail or not,” he said.

As one example of the level of detail that is required, a recent study says that when reporting studies using mice it is a very good idea to report whether a man or a woman handled them. “Research has shown that mice are more anxious if handled by a man than a woman …. which can have an impact on the results,” said Smith.

As the gatekeepers of new scientific knowledge, journals have a role too, Smith thinks. The attention paid to the issue has increased. “Not so long ago, a lot of journals tended to truncate descriptions of methods. Now they have come around some more,” said Smith. For example, in 2013 Nature promised to implement new measures improving the consistency and quality of reporting in life-sciences articles, including giving more space to methods sections, and has assembled an archive called “Challenges in Reproducible Research”.

Renewed attention from funders

Smith would not go so far to say irreproducible science represents a crisis. He prefers to call it a “problem”.

“On the whole, medical research works very well. We’re making new drugs and treatments,” he said. 

It also self-corrects: researchers are trained to be highly critical and point out the limitations of one another’s work. It would just be nicer if we could do this quicker, said Smith.

To help get at the issue, the MRC has made a few changes to its processes. Applications that do not provide enough detail to judge the rigour of animal experiments are returned to researchers. In addition, MRC boards and panels are instructed not to look at where a paper was published but at its content.

In June, the US National Institutes of Health announced measures to improve reproducibility in bioresearch. Reviewers are being asked to carefully consider the strength of the scientific premise; the rigour of the study design; the proposal’s consideration of the sex of research animals or human subjects; and whether reagents have been authenticated.

Smith has some other ideas. “It’s a fine line you have to walk, isn’t it? You want to make sure researchers have the freedom to do their research. We don’t want to overlay them,” he said, although slightly slower science might make researchers more careful.  It is also important biomedical research is not turned into “a giant results-verifying machine”, he added.

The MRC might consider doing more analysis of the outcomes of grants, in order to record negative as well as positive results. Extra training to drive home the good scientific practice is another idea.

“What about a journal that specialises in publishing negative results?” Smith mused, “there’s a tendency to publish positive results and bury inconvenient ones,” It is important that the public are made aware of negatives.”

It is rare for most journals to publish papers on experiments that did not work but there is a publication called the Journal of Negative Results in BioMedicine, which shares failures. There is also a magazine called The Journal of Irreproducible Results, but this is aimed at taking the air out of serious science (it says it “targets hypocrisy, arrogance, and ostentatious sesquipedalian circumlocution”) rather than highlighting non-replicable research. 

Another idea involves not rushing to celebrate “eureka” moments before research has been validated by repetition. There is a special onus on journalists here, Smith said.