The authors started with previous estimates of the amount of irreproducible preclinical research, which soar as high as 89%. To be conservative, they decided to use 50% for their calculations. Industry, academia, and nonprofits spend $58.4 billion on this research each year. Half of that—$28.2 billion—was their estimated cost of irreproducible research.
It’s a fairly simple calculation, Freedman says, but the number is a good starting point for discussion. “The intent was to be a little bit provocative, but also to be as accurate as we could be.” After the paper came out, some critics said the authors overestimated the problem. But Freedman says even if the estimate is high—and he’s not convinced it is—the problem is still worth the research community’s attention.
“Is it $28 billion? Even if it’s $10 billion, it’s an enormous number,” he says.Gotta say, I'm a little skeptical of that 89% number, or the 50% number. But Freedman is right - even if it's 5 billion dollars, it's probably too much.
[That said, I think there's an argument to be made that the "opposite" view is important - how much irreproducible science should we accept in the peer-reviewed literature? I am prepared to argue that the correct number is not "absolutely zero." Sure, it's probably not $10 billion, either.]
Training the next generation of scientists may be the best way to avert problems that lead to irreproducible research. The current model of a principal investigator mentoring graduate students doesn’t always lead to the changes needed to improve the quality of research, Freedman says. For example, he met a fifth-year cell biology graduate student who didn’t know about a simple assay that could solve the problem of cell misidentification. “We really need to reexamine how we are training students,” he says.
Many researchers consider irreproducibility a problem just in academia, but Freedman says pharmaceutical companies also need to take it seriously. As drug companies cut back on internal R&D, more and more of their basic research is coming out of academia. So are future drug company employees.
“If I were still in pharma,” he says, “I would be pretty concerned about these issues.”So far as I can tell, pharma is concerned about these issues, isn't it? I presume there's a fair bit of work around validating biological targets, etc. It's sort of an odd statement, too - wasn't the Nature paper that kicked off a lot of the irreproducibility fuss written by a former Amgen research head?