Can we spend too much time dwelling on the past? Is it worth it? If we can learn from it and avoid repeating the same mistakes, it is definitely worth the effort.
A few months ago, I listened to a great presentation by Brian Hughitt from NASA’s Office of Safety and Mission Assurance. His presentation left on impression on me and I would like to share some highlights with you. The story is another testimonial to the importance of finding the true root-cause of a problem. However, this story is not about NASA, its about Boston’s Tunnel Project.
Boston’s Central Artery/Tunnel Project is the most expensive highway project in U.S. history. The 7.5 mile underwater tunnel to Logan International Airport used 3.8 million cubic yards of concrete (enough for a sidewalk from Boston to San Francisco and back 3 times) and 541,000 truckloads of dirt (about 15 stadiums filled to the rim with dirt.) The tunnel has been plagued by leaks, falling debris, delays and other problems linked to faulty construction. The initial price tag was $2.6 billion and it was supposed to be completed in seven years. Instead, it took nearly 15 years and repeated cost overruns drove the price tag up to $14.6 billion.
At 11:00pm on July 10, 2006 a 1991 Buick passenger car occupied by a 46 year-old male driver and his 38 year-old wife was traveling eastbound in the I-90 connector tunnel, en route to Logan International Airport. As the car approached the end of the connector tunnel, a section of the tunnel’s suspended concrete ceiling (26 tons) detached from the tunnel roof and fell onto the vehicle, crushing its right side. The driver’s wife, occupying the right-front seat, was fatally injured. The driver escaped with minor injuries.
The investigation of the accident revealed a history of findings and reports that should have led to earlier action that might have prevented this fatal incident.
The proximate cause of the structural failure has been attributed to the use of an epoxy anchor adhesive with poor creep resistance. The anchor attaches the hanger for the ceiling panels to the concrete roof of the tunnel. Post accident testing revealed that fast-set epoxy had been used and that, while both fast-set and standard-set epoxy performed similarly in short term tests, they differed dramatically under long term load.
***
Creep - Epoxy is a polymer and its stiffness is time and temperature dependent. If a load is applied suddenly, the epoxy responds like a hard solid. But if that load is then held constant, the molecules within the polymer may begin to rearrange and slide past one another, causing the epoxy to gradually deform. As the deformation increases, it becomes irreversible
***
Fast-set epoxy was used because it was more convenient and perceived to have very similar performance qualities. Why were the engineers working on this project unaware about the creep characteristics of the fast-set? Was it negligence by designers, suppliers, procedures, or all of the above? There is much to learn from the story leading up to this event.
The following fishbone diagram gives an idea of the wide variety of factors requiring investigation on an incident like this. A thorough root-cause investigation is a big investment, and the reality is that we cannot perform analysis to this detail on every incident. The size of the potential impact of the issue at hand must be considered and balanced with how much effort goes into an investigation. However, some companies like Toyota strongly believe that it is good to contain a problem right away and investigate the root-cause to prevent further impact and variability on the planned production process.
The root-cause analysis on this incident yielded some interesting findings. Let’s look at the main contributing causal factors identified by the Safety Board’s report.
The Safety Board analysis starts with the tunnel design. The tunnel design didn’t have redundancy built into the ceiling suspension. The majority of U.S. tunnels have continuous ceiling panels that extend into the concrete wall. If the hangers fail, the ceiling is self-supported. Design specifications for this tunnel did not incorporate a provision for attaching a suspended ceiling, even though it was known that one would be needed. Consequently, the tunnel had no embedded ceiling supports. The design consultant repeatedly recommended undercut anchors versus the adhesive anchors ultimately chosen. In addition, in order to save cost and time, a change was made to deviate from the original design and use heavier pre-cast concrete panels in lieu of custom-engineered laminated lightweight concrete panels.
Failure Modes and Effects Analysis (FMEA) was incomplete. Creep was not identified as a potential failure mode. Consequently, risk mitigation measures were not implemented.
The Safety Board identified a high concern with the ICBO industrial standards. The ICBO AC58 calls for either a design safety factor of 5.33 or a 120-day creep test for fast-set epoxy. The creep test is optional even though the ability to sustain a load over a period of time is a typical requirement for almost any type of fastener.
A design engineer should be provided with all of the relevant information about a product before it is used in a safety critical application. However, no documentation was provided by the supplier specifying which epoxy formulation was being supplied, and neither the contractor nor the design consultant questioned which epoxy was used. Both assumed that the epoxy provided by the supplier was suitable.
To support product qualification, the supplier provided an Evaluation Report (ER) which included bond strength tables specifying a safety factor of 5.33 for fast-set epoxy. Nothing in the ER tables or footnotes indicated that the fast-set epoxy should be limited to use with short-term loads regardless of the safety factor employed. A restriction for use in short-term load applications was contained elsewhere in the ER, but could have easily been overlooked.
The Safety Board learned during the investigation that fast-set epoxy had been tested for creep performance in 1995 and 1996 and had failed to meet the standard, but still qualified under the current AC58 standard.
The supplier should have made a clear distinction in all its literature between the relative capabilities of its standard-set and fast-set formulations. It did not do so, even though before the epoxy was provided, the company had conclusive evidence that its fast-set epoxy was susceptible to creep. The Safety Board concluded that the information provided by the supplier was inadequate and misleading.
The Safety Board also found fault with the construction contractor and the design consultant for not adequately reviewing the product qualification documentation.
But the story gets even better… I mean worse…
On September 9, 1999, (seven years before the accident) a construction contractor employee installing ventilation ductwork over the tunnel ceiling noticed that several of the anchors had begun to pull out.
The finding was documented and on November 12, 1999, a proof load test was performed on one of the anchors that had shown significant displacement (9/16”). The engineer noted that “the bolt held for a few seconds, then began to pull out with almost no resistance.
When the supplier was called to examine the anchor displacements, they seemed surprised that the anchors that had been successfully proof tested only a few months before could be failing. The fact that the fast-set epoxy was subject to creep, was apparently not considered or was not known by the representatives who evaluated the failed anchors. Even if the information about poor creep resistance was not common knowledge, some research into historical performance testing would have likely revealed it.
Installation problems (e.g., excessive preload) were postulated by the supplier as the cause. There was no evidence of further testing or research by the supplier. Shouldn’t a supplier be more proactive about one of their safety critical components failing?
The response from the builder, design agent, and project manager was to perform more proof load testing. The original design service load was calculated at 2,600 lb-force. After original bolt installation, a proof test was conducted at 25% higher than design service load (3,250 lb-force) and the bolts passed. After the slippage was found, bolts were proof tested at the maximum load of 6,350 lb-force and the bolts passed again. Analysis determined that the actual load would be well under that, at a maximum of 2,823 lb-force.
At this point the builder and project managers decided to proceed with repairs and move on confident that the bolts would hold based on the testing performed. They do so even though a couple of employees had raised some concerns on emails.
"You’ve noted the key piece of information that is missing. That is the cause of the anchor failure and how the repair procedure will overcome that… We are not trying to hold up construction, we are trying to make a determination that the installation is safe…”
Design Manager e-mail concerning response to Deficiency Report
“Glaringly absent from the Deficiency Report is any explanation why the anchors failed and what steps are proposed to ensure that this problem does not reoccur.”
Structural Engineer e-mail reply
The root cause for the anchor displacement was never clearly identified and surveillance monitoring inspections were never implemented.
But wait there is more…
On December 17, 2001, a quality control inspector submitted a Noncompliance Report which stated: “Several anchors appear to be pulling away from the concrete. The subject anchors were previously tested to the revised value of 6350 lbs, all of which passed…. Reason for failure is unknown.”
At this point, it should have been obvious that the prior remedies in response to the anchor displacement in 1999 had not been effective. This was another opportunity to inspect all the installed anchors and, more importantly, to determine the cause of the anchor displacement. Instead, the oversight team apparently considered the continuing failures as isolated instances and took no further action to address the problem in a systemic way. Even after being presented with evidence of anchor creep, project managers and overseers failed to recognize the inherent weakness in the epoxy adhesive – a weakness that could not be overcome even with the best installation practices or the most rigorous short-term proof testing.
This type of denial and cognitive dissonance is something to keep in mind when performing root-cause analysis. Our minds can filter information that conflicts with what we might already believe, making it hard to see other alternatives.
In November, 2003, the Design Agent published Inspection Manual for Tunnels and Boat Structures. The manual required each ceiling hanger component to be inspected visually or by NDT. From the time the tunnel was opened to traffic until the day of the fatal accident, no tunnel inspections were performed.
Post accident inspection of the suspended ceiling displayed large numbers of anchors that had become displaced (~25%), and that the displacement was so obvious that even a cursory examination would have revealed that structural integrity was threatened.
Investigators asked MTA officials why the inspection manual was not used. The officials stated that the inspection manual was not used because (1) a tunnel inspection database needed to be developed, (2) the inspection manual was being reviewed by the FHWA and the MTA, and
(3) MTA personnel needed time to be trained on the manual.
It is also amazing that federal regulations were not in place for tunnels. The FHWA National Bridge Inspection Program (NBIP) mandates bridge inspections at least once every two years, but there are no similar mandates for tunnel inspections. The FHWA’s Road Tunnel Design Guidelines does not address the design, construction, and inspection of tunnels.
The result of the investigation was that the panel support system had to be completely replaced with Hilti mechanical fasteners which had to be individually proof loaded.
In summary,
The investigation of this incident yielded incredible findings. Contributing causal factors included:
• Lack of Redundancy in Design
• Inadequate Regulations
• Lack of Awareness and Documentation on Product Performance
• Lack of Adherence to Quality Assurance Procedures
What sticks in my mind the most is that even though some emails expressed a concern, the root-cause was never identified and both builder and supplier went on with their business when the hanger anchors, a critical safety component, appeared to be failing.
When it comes to safety, we need to perform adequate FMEA, we cannot afford to focus on project dates, and we definitely cannot afford to move on without identifying the root-cause of a failure. At a minimum, surveillance inspections should have been implemented to monitor the situation.
Paraphrasing Brian Hughitt, "The study of incidents like this one, incidents of “un-quality”, helps us become better students of quality."