Irreproducible chemistry results and unknowable secrets

The secret ingredient is: I DON'T KNOW!
Photo credit: bestweekever.tv

A few years ago, US nuclear weapons scientists experienced significant delays in reproducing a previously US-manufactured material codenamed FOGBANK for use in nuclear warheads. The material is believed to be an aerogel that participates in the second stage of a thermonuclear reaction. While the exact nature of FOGBANK has not been disclosed, the story was interesting enough to attract some attention from national media. LANL's own internal newsletter (page 20) talks about the difficulties of reproducing the material.* It will sound familiar to most chemists:

...When investigating historical records with respect to impurity levels during the Fogbank purification process, personnel discovered that in some cases the current impurity levels were much lower than historical values. Typically, lower impurity levels lead to better product quality. For Fogbank, however, the presence of a specific impurity is essential. 

Laboratory data show that the presence of one particular impurity in the Fogbank purification process plays an important role in the quality of the final material. The impurity’s presence in sufficient quantity results in a different morphology (form and structure) of the material. Although the change in morphology is relatively small, it appears to play an important role in the downstream processes. A review of the development records for the original production process revealed that downstream processes had been implicitly based on that morphology. 

[snip] Historically, it was this chemical that reacted during purification of the feed material to produce the impurity necessary for proper morphology. The historical Fogbank production process was unknowingly based on this essential chemical being present in the feed material. As a result, only a maximum concentration was established for the chemical and the resulting impurity. Now the chemical is added separately, and the impurity concentration and Fogbank morphology are managed.

I think that most chemists will agree that most chemistry is a reproducible science. X grams of starting material A, Y grams of reagent B, stir for 30 minutes at 50°C in Z mls of solvent C and voila -- product!

But I think that reactions and processes that are more complex and less understood have little details that are important; these details may be known to a few (or none.) It is sometimes extraordinarily difficult to know what is key (and what is not key) until you run into a failure like the FOGBANK scientists ran into. (Cynically, sometimes scientists will withhold these details to increase their status and preserve job security. Sometimes, I can't blame them.)

While reports and long conversations with the personnel involved are central to the deep understanding of a reaction or a process, often even they may not be aware of the "unknown unknown" that is at the heart of reproducing the desired material. Then, the paranoia really sets in.

*In other sources, it's mentioned that acetonitrile may be part of the manufacturing process. I'm amused to note how many folks consider the stuff dangerously flammable and toxic; that being said, one comment in that post has them taking the stuff supercritical. That's a little different than a bottle sitting atop an HPLC.