Process Wednesday: Backups on backups on backups on backups

I'm still reading Bert Hulshof's "Right First Time in Fine-Chemical Process Scale-up"* (almost done!). One of the more valuable things in the book is Chapter 4, where over 200 case studies of processes where some difficulty was encountered in scale-up and the solutions that were devised.

I was rather taken by an example of bromination of 4-nitrotoluene; the chemists in this case decided to perform the bromination without solvent. Naturally, there were runaway concerns, with the "temperature of no return only 25°C above the bromination temperature." Here's what the engineers did to ensure that the reaction operated within a state of control:

• "Using dual temperature probes, the first interlock, set at a working temperature range of approx. 2°C, controlled the addition of bromine. Below this temperature range the bromination was too slow and significant amounts of bromine were lost to the scrubber, while above this temperature range the rate of thermal decomposition increased." 
• "At 5°C above the upper temperature limit, a second interlock would activate a system that automatically emptied the pressurized hot water from the jacket and applied crash cooling." 
• "If the agitator failed, a third interlock would activate a system where nitrogen was vigorously bubbled through the reaction mixture to provide the agitation needed to allow efficient heat loss to the jacket."
• "If the plant would suffer a complete power failure, the crash cooling would not work because the cooling water was circulated by means of an electric pump. Although back-up emergency generators were available, there was always a slim chance that they may not start in an emergency. In order to overcome this eventuality a back-up system was devised which worked off the high-pressure city water main which not dependent on the site electricity supply." 
• "This process was run for just over seven years and none of the emergency systems were needed." 

This series of backups is in rather stark contrast to the T2 Laboratories explosion in 2007, where loss of cooling (among other factors) led to the explosion of a 2,450 gallon reactor, 4 deaths and 32 injuries and  destruction of the worksite. Here's the relevant portion of the CSB report:

...At a temperature of about 360°F (182.2°C), the process operator initiated the control system cooling program, which intermittently injected water into the jacket based on the rate of reaction temperature increase. The operating procedures used at the time of the incident included no emergency instructions for loss of cooling. 

However, earlier procedures—which included emergency instructions—directed the
operators to fully open the water supply valve and the manual bypass valve. A secondary (backup) source of water stored on site was not immediately available to the process operator in an emergency. 

...The CSB determined insufficient cooling to be the only credible cause for this incident, which is consistent with witness statements that the process operator reported a cooling problem shortly before the explosion. The T2 cooling water system lacked design redundancy, making it susceptible to single-point failures...

Interestingly, the CSB report seems to point to a problem with the cooling drain being a potential factor in the failure of cooling, i.e. the cooling water was not draining out of the reactor jacket. Presumably, operators could have performed some sort of check before starting the batch to make sure that the cooling system worked? I wonder how often the backup systems in the bromination example were checked...

*Honest, I'm not getting anything for talking about the book -- it's just interesting and full of blogging fodder.