On biology Twitter, this Nature essay by Lilly senior vice president Levi Garraway has received some attention, especially this passage (emphasis mine):
An incomplete thought, but one that I ponder a lot, especially as the Reproducibility Wars rage on...
...In medicine, the quest to distinguish between distinct conditions with similar symptoms is called 'differential diagnosis'. By analogy, three questions can guide a diagnosis of apparent irreproducibility: (1) Was the replication attempt backed by the requisite expertise? (2) Does a systematic comparison reveal a basis for discordant results? (3) Could the original result be wrong?
To illustrate the first question, consider the following example: extensive evidence exists that, in his prime, Tiger Woods could consistently hit a golf ball more than 260 metres straight down the fairway using his driver (the largest club in the bag). I too play golf; I have achieved credible results with my own driver (well, some of the time) and I am roughly the same height and weight as Tiger. I would very much like to be able to reproduce his results with my own hands.
Over the years, the golf industry has made untold sums of money from many golfers (myself included), who aspire to reproduce Tiger Woods' results. However, only a small percentage of us can pull this off (and I am not one of them). Does this mean that Tiger Woods' results were 'wrong', or that the remarkable physics he seemingly exemplified does not truly stand up to independent scrutiny? Of course not; it means that reproducing such results consistently requires a level of mastery that the typical golfer does not possess.
I do not believe that experimental skill is as elusive as that necessary to win the green jacket. I just want to underscore that we cannot assume that any given scientist — even a very good one — is properly equipped to reproduce an experiment if it involves new reagents, systems or biological context. Before even attempting to reproduce an 'index' experiment, a lab that lacks the needed experience often sends a trainee to another lab to work with a scientist who regularly performs the experiment. This is so, even if both labs are recognized experts.
If the question of reproducibility persists once the requisite expertise is established, the answers often reside in subtle differences in methods, cell-line properties or reagents that become apparent only after scrutiny. Two acclaimed cancer researchers required more than a year to harmonize techniques to get similar measures for experiments; success depended on cross-country visits and on reconciling minor differences in how cells were prepared.
If these steps do not work, we must consider whether the original result was really correct. And we must be prepared to accept the brutal facts, make the required corrections and move on. Great scientists are always willing to embrace the truth with humility and grace — even when it hurts....I find this passage odd, mostly because I'm an organic chemist, and the ten or twenty variables that we can control for are measurable with the myriad tools that we have to characterize physical and/or chemical properties. Biologists have many more variables, so that's probably where the difference lies. (I wonder if this means there is room to understand and communicate biological experimental technique better in biomedical science?)
An incomplete thought, but one that I ponder a lot, especially as the Reproducibility Wars rage on...
Over the past few years, biology--especially the in vivo crowd--has seen a rash of attempts to systematize reporting of experimental details. I suspect this is doomed to fail, as the variety of potentially meaningful details is enormous, but the number of details demonstrated to have an impact on any one study is small (and the number that has a real effect on any one study likely isn't much bigger).
ReplyDeleteAnd then there's the question of whether it's worth studying experimental mechanisms that are so narrowly applicable that they don't show up if you tweak the procedure even a little....
If even a very good scientist may not be able to reproduce your results, have the experiments been properly described? One of the fundamental tenets of science is that the system is reproducible. A skilled scientist should be able to set up the system as you describe and receive a comparable result. Even in the much simpler chemical world this is often a problem. We are inclined to not report details we think to be trivial or obvious. Unfortunately, these trivialities are often not so obvious to others.
ReplyDeleteI think it also important to highlight the last line you quoted. That is, "Great scientists are always willing to embrace the truth with humility and grace — even when it hurts....".
Agreed. Most synthetic chemists have had to deal with trying to reproduce a synthesis from a very old prep where few descriptive details were provided. It's annoying but at least somewhat understandable for stuff that was published back in the day of typewriters and 1 page communications. When I see that lack of descriptive detail in the era of unlimited space for SI, I get extra cranky.
DeleteI have to remind the students to write their experimental procedures so that a second year grad student in their same subfield but a different group could do the procedure without too much confusion.