Also in this week's C&EN, a guest editorial from Richard Harris:
...Last year, Nature surveyed 1,576 scientists from all disciplines. Overall, 52% perceived a “significant reproducibility crisis,” and another 38% said there was a “slight crisis.” Another 7% said they didn’t know, and only 3% said there was no crisis.
In the case of biomedicine, I find multiple layers of causes, all of which exist to one extent or another in other fields. First is that scientists put too much faith in the ingredients they use. Some 500,000 antibodies are commercially available for experiments, but the quality of those reagents is all over the map, and labs often don’t run enough controls to identify problems. Immortal cell lines are another example, with cross-contamination a major problem. Scientists are paying less attention to more garden-variety reagents, but those are also problematic. That warning would obviously extend beyond the world of biomedicine.
Another huge area of trouble in biomedicine and psychology involves experimental design and statistical analysis. Chemists may have more predictable systems and more reproducible experimental designs, but to the extent they are trapped in the dubious analytical system built around the P value, chemists should be greatly concerned as well. This system is so often misused in science, the American Statistical Association felt compelled to publish a paper in 2016 decrying the shoddy understanding of P values.
And when John Ioannidis wrote the heavily cited essay “Why Most Published Research Findings Are False,” he did not confine his analysis to the life sciences. “For most study designs and settings, it is more likely for a research claim to be false than true,” he notes....Overall, I have rejected the "reproducibility crisis" frame for this issue (and Harris does as well) , because the word "crisis" is not really very well defined for me. (Also, I'm not trapped by P values routinely, but maybe I am and I just don't know!)
My takeaway is pretty bog-standard scientific judgment: you shouldn't believe new results in any field (especially biomedicine) without there being sufficient reproduction from other groups. Also, it would be great if science (chemists!) came up with better, faster, cheaper means of analyzing biomedical reagents (antibodies and such) and verifying identity and purity.
How will you know when another group reproduces the study? Is there some kind of journal that publishes these reproductions?
ReplyDeleteFair point.
DeletePatents are the worst - written to obfuscate key points and critical steps, and reviewed by people with half a clue at best. Journal papers are better in large part because they honestly attempt to communicate. However they are hit and miss when it comes to actually doing a thorough job of describing the procedures used.
ReplyDeleteNote to grad students writing papers:
1: When I review your papers, the first thing I do is read the abstract. The second is to search for your methods (either in the main body or SI) and check to see if I could reproduce what you did without ambiguity. If I can’t I’ll be asking for a re-write.
2: Please put at least one detailed example of a synthesis in the paper or SI directly, not buried in a reference. Unlike universities, companies generally don’t have unlimited journal access and each time you force us to fetch another article we are out another $40 or so.
Thank you for making those requirements in your reviews! I must confess, I did not think about item (2) until I started working in industry. The worst offenders are the papers that reference an earlier article, which references an earlier article...
DeleteI agree for Patents as Process ones I have tried to "repeat" often seem intentionally misleading and many of the Patent Lawyers I have interacted with must have been on the low end of the at best 50% in understanding how to integrate technical info with patentese without gutting or confusing essential details. It seems to go against the letter of the law for obtaining a patent as I read it much less the spirit of the system. Observation is unless simply wanting to synthesize a particular molecule "those skilled in the art" must mean a PhD with 10 years experience willing to devote 3-6 months to "interpret" the written procedures to establish a potentially useful process. In the end there is probability the originators have implemented a different process covered by internal trade secrets (although not sure with much more "movement" these days if that is relied upon/reliable mechanism).
Delete