Tuesday, September 8, 2015

The costs of irreproducibility

Also in this week's C&EN, Andrea Widener talks with Leonard P. Freedman, the president of the Global Biological Standars Institute, "a nonprofit organization that advocates for the use of standards and best practices in research." Interesting calculations they've made in a PLoS Biology paper:
The authors started with previous estimates of the amount of irreproducible preclinical research, which soar as high as 89%. To be conservative, they decided to use 50% for their calculations. Industry, academia, and nonprofits spend $58.4 billion on this research each year. Half of that—$28.2 billion—was their estimated cost of irreproducible research. 
It’s a fairly simple calculation, Freedman says, but the number is a good starting point for discussion. “The intent was to be a little bit provocative, but also to be as accurate as we could be.” After the paper came out, some critics said the authors overestimated the problem. But Freedman says even if the estimate is high—and he’s not convinced it is—the problem is still worth the research community’s attention. 
“Is it $28 billion? Even if it’s $10 billion, it’s an enormous number,” he says.
Gotta say, I'm a little skeptical of that 89% number, or the 50% number. But Freedman is right - even if it's 5 billion dollars, it's probably too much.

[That said, I think there's an argument to be made that the "opposite" view is important - how much irreproducible science should we accept in the peer-reviewed literature? I am prepared to argue that the correct number is not "absolutely zero." Sure, it's probably not $10 billion, either.]
Training the next generation of scientists may be the best way to avert problems that lead to irreproducible research. The current model of a principal investigator mentoring graduate students doesn’t always lead to the changes needed to improve the quality of research, Freedman says. For example, he met a fifth-year cell biology graduate student who didn’t know about a simple assay that could solve the problem of cell misidentification. “We really need to reexamine how we are training students,” he says. 
Many researchers consider irreproducibility a problem just in academia, but Freedman says pharmaceutical companies also need to take it seriously. As drug companies cut back on internal R&D, more and more of their basic research is coming out of academia. So are future drug company employees. 
“If I were still in pharma,” he says, “I would be pretty concerned about these issues.”  
So far as I can tell, pharma is concerned about these issues, isn't it? I presume there's a fair bit of work around validating biological targets, etc. It's sort of an odd statement, too - wasn't the Nature paper that kicked off a lot of the irreproducibility fuss written by a former Amgen research head?

(Also, Isn't it time for people to retire the "I once met a young 'un who didn't know about My Favorite Technique, so We Are All Doomed" trope?) 


  1. Dr. Lowe's had some posts on this topic, and 50% doesn't seem ridiculous (Anywhere between 35% and 65% can probably be justified, at least in some areas - I think the recently cited ones were cancer bio and experimental psychology, which might be the least reproducible areas for various reasons). Unfortunately, since there seems to be an inverse correlation between splashy results and reproducibility, and the splashiest results are the ones most likely to garner VC or pharma funding, the results least likely to be reproducible are the ones that are most likely to cost someone money. This would mean that the amount of money that irreproducible results cost is a larger slice of the funding pie than their portion of the publication pie.

  2. Meh. Here is an anecdote: I know of an "irreproducible paper" that got in a good journal that implicated a protein to be required for a certain type of cancer metastasis. In that paper, as far as I can know, it was just one key figure that was not reproducible; but its just that the particular figure (a knock-down experiment) that "proves" that this protein is required. It is likely the rest of the work is perfectly fine. So, at best its about 20%, or less of that work is irreproducible, and so 20% of the, what, 1/4 million dollars to do all that work is wrong. It's just the main conclusion of that paper is wrong.

    I imagine most of the other irreproducible work could be similar: a key experiment that got the paper into a famous journal is not reproducible, but most of the other work is fine. Its just that crap experiment made the work significant.

    So I think its more like 20% of all experiments reported are irreproducible, but 80% of the conclusions are wrong.

    I agree that the big Pharm sending research back to academia is not a good idea. We need people out there that are independent of the ferocious competition of grant-sponsored research to confirm it, and bring the bad labs down.

  3. It would be interesting to find out what percentage of this is due to lack of comprehension of basic experimental procedure and design (a surprisingly common issue that invalidates many studies), and how much of this is due to outright fraud.

  4. Wait, they still have journalists at C&EN?

  5. http://www.bloomberg.com/news/articles/2015-09-10/the-miracle-fat-pill-that-never-was-and-the-downside-it-exposed

    on cost of irreproducibility

  6. When I first saw a preclin presentation I was pretty enthusiastic about the quality of data. Every point had error bars, statistical info etc. My enthusiasm converted to "this is nuts" attitude when I learned that many of the "statistically processed" experiments consisted of two measurements of unknown quality. The data "point" was the average and the up-down error bars were the distances from the average to the two results. Nod politely, smile, walk away...