Tag: defense industry war stories

The Apache Main Rotor Blade Failures

By Joe Berk

Like that photo you see above?  Yeah, me, too.  I took it on the parade grounds at Fort Knox, Kentucky, a few years ago.   I used to run the Composite Structures plant that made rotor blades for the Apache helicopter.  It was one of the best jobs I ever had.

We recently reposted (under the Wayback Machine banner) our blog about the Gator mine system, and in it I promised to write about the Apache main rotor blade failures.  This is another defense industry failure analysis war story that crosses company lines and supplier/customer boundaries, and I’m not entirely sure that there wasn’t some nefarious behavior going on at McDonnell Douglas.  I’ll tell you what happened and you tell me.

The UH-1 Huey was a Vietnam War workhorse. It was extremely susceptible to small arms fire and you could hear it coming miles away.

During the Vietnam War, the Army (my alma mater) found that the Huey helicopter had a few shortcomings.  I guess that’s to be expected; it was the first time the Army used helicopters in a major way in a real war.  The Huey was susceptible to small arms fire (and big arms fire, too, for that matter) and it was noisy.  On a clear night, you could hear a Huey coming in from a long way out with its characteristic “wop wop wop” signature as it beat the air into submission.  That “wop wop wop” sound was actually the rotor tips breaking the sound barrier on the left side of the helicopter, so the Army knew it had to do something to get the blade tip speed below the speed 0f sound on its next-gen helicopter.  Another big problem was small arms fire; a single .30-caliber AK-47 bullet through a Huey rotor blade would destroy the blade’s structural integrity (and there were a lot of AK-47 rounds in the air in those days).  When that happened, the helicopter and its crew were lost.  There could be no autorotation (you can’t autorotate without a blade) and you couldn’t bail out.  The next-gen helicopter blades would have to be impervious to small arms fire.

Fixing the blade tip speed problem was simple.  Instead of having two blades like the Huey, the Apache went to four blades.  That cut the rotor speed and let the blade tips go subsonic.  “Wop wop wop” no more.  Easy peasy.

The structural integrity issue was the more significant challenge. The engineers at McDonnell Dougas (the Apache prime contractor) designed a blade that had four spars that ran longitudinally (with the length of the blade) contructed of AM455 stainless steel (a special blend used on the Apache and, at the time, nowhere else).  The spars had overlapping epoxy-bonded joints that ran the length of the blade.  The idea was that a hit anywhere on the blade (up to and including a 23mm high explosive Russian anti-aircraft round, roughly the explosive equivalent of a hand grenade) would damage that spar, but the remaining three spars would hold the blade together.   It worked.  An Apache blade actually took a blade hit from an Iraqi ZSU-23/4 and made it back to base.

A cross section of the McDonnell Douglas Apache blade showing the four spars.

So here’s the problem:  The Army specified a blade life of 2200 hours (blades on a helicopter are like tires on a car…they wear out), but our blades were only lasting about 800 hours before the blades’ bondline epoxy joints holding the spars together starting unzipping.   It wasn’t a catastrophic failure (the helicopter could still fly home), but the blades had to be repaired.  The Army would send the blades back to McDonnell Douglas, and McDonnell Douglas sent them back to us at Composite Structures for refurbishment.  If they couldn’t be repaired, we sold McDonnell Douglas a new blade (back in the 1990s, each blade cost just north of $53,000, and McDonnell Douglas put a hefty markup on that when they sold the blade to the Army).  When they could be repaired, we still charged a hefty fee.

When I entered the picture as the plant manager, I learned that both Composite Structures (my company) and McDonnell Douglas (my customer) had made half-assed efforts to fix the blade problem, but neither company was financially motivated to eliminate it.  We were making good money selling and repairing blades and so was McDonnell Douglas.   The Army, however, was taking it in the shorts.

This was also a major problem for me as the manufacturing guy.  I didn’t like having to make two blades to get one good one.  We were rejecting one of every two blades we made for spar disbonds in the factory.  You read that right:  We had to make two blades to get one good one.  Because of this, we were in a severe past due delivery condition, and my mission was to correct that situation.  So we went to work on solving the problem.  We found and fixed plenty of problems (blade cure profile issues, cleanroom assembly shortfalls, epoxy shelf life and pot life issues, nonconforming components issues, and contamination issues), but the blade disbonds continued.  McDonnell Douglas continued to pound us for quality issues, all the while secretly smiling all the way to the bank as they continued to sell twice as many blades as they should have been selling.

We went through everything and finally concluded that there had to be a design issue with the blades; specifically, that the bondline width where the spars were glued together had too much variability.  If that glue line was small enough, we reasoned, it wouldn’t hold up and the blade would disbond.  We asked McDonnell Douglas about that (McDonnell Douglas was responsible for the design; we were building it to their engineering drawings), but they kept blowing us off.  The bondline width wasn’t dimensioned on the McDonnell Dougas drawings.  The other parts were, and McDonnell Douglas’ idea was that if the blade parts met their drawing requirements, the bondline width would be okay.   That’s what they hoped for, anyway.  But you know what they say about hope.  You can poop in one hand and hope in the other, and see which one fills up first.

A macro shot of the bondline joint. The scribe lines (in the blue Dykem) show the bondline area.

I asked for a meeting with our company and McDonnell Douglas on the blade failures, and they wouldn’t meet with us.  So I sent out another invitation, and this time I included the Army.  McDonnell Douglas was livid when the Army quickly said yes; now, the McDonnell Douglas wizards had to meet with us on this issue.  That meeting started about like I expected it to, with McDonnell Douglas tearing us a new one on the blade failures, telling us our quality was terrible, and basically letting me and the rest of the world know that, in their opinion, things had gone downhill since I had taken over as plant manager (no matter that this 50%-rejection-rate blade issue had existed for a dozen years prior to my arrival).  I patiently explained the issues we had found and corrected, and then emphasized that the problem with blade separations had continued unabated.  I then asked the McDonnell Douglas program manager about the bondline width and the fact that this apparently critical requirement was not on their engineering drawings.  He denied it was the issue and went off about our poor quality again.  When he ran out of steam, I asked the question about the bondline dimension yet again, and specifically, how narrow the bondline could be and still provide an adequate joint.  There were more accusations about our lousy quality (the guy only knew one tune and he loved singing it), and I again waited for him to finish.  When I asked the question a third time, before McDonnell Douglas lit up about our poor quality again the Army representative asked “yeah, how narrow does it have to be before the blade fails?”

The McDonnell Douglas guy stared at me like cobra looks at a mongoose (I’ve only seen this in YouTube videos, but I’m pretty sure the analogy is a good one).  He sputtered and stammered and I think I saw a little spit fly from his mouth.  “If you make it to the drawing it will be okay,” he said.  I mean, under the circumstances it was the only thing he could say.  I almost felt sorry for him, in the same way you feel sorry for a rat when a red-tailed hawk is swooping down with talons extended.  You feel bad, but you look forward to seeing the hawk doing his thing.

The Army guy sensed this was something big. “How low?” he asked again.  If there is such a thing as a perfect impersonation of a deer caught in the headlights, the McDonnell Douglas dude was nailing it.  It was what we in the literary world call a pregnant pause, one of those “what did the President know, and when did he know it?” moments.  As I type this, I can remember the scene like it happened 10 minutes ago, but it’s been close to 30 years.

“0.375 inches,” the McDonnell Douglas dude finally answered.  He actually said the zero in a half-assed attempt to add engineering gravitas to his answer. “As long as they build it to the print, they’ll be okay,” he added, with a “so there” smirk.  He was answering the Army man, but the smirk was all for me.

What the McDonnell Douglas guy didn’t know was that my guys could see the bondline width in an x-ray, and we x-rayed every blade returned for repair.  And I guess he didn’t realize how easy it was to do a tolerance analysis to show what the drawings allowed the bondline width to be.

What happened next was one of those moments I’ll remember for the rest of my life.  I looked my engineering guy and my QA guy.  They knew what I wanted.  They both left the room.  Fifteen minutes later they were back.   My engineering guy handed me the results of his tolerance analysis.  The McDonnell Douglas engineering drawings tolerance stackups allowed the bondline width to go as low as 0.337 inch.   The QA guy had even better information.  All the blades that had been returned to us for spars unzipping (which was the only reason we ever saw a blade returned) had bondline widths less than 0.375 inches (McDonnell Douglas’ admission for the lower limit) but above .337 inches.  In other words, our quality was fine.  The failed blades met the McDonnell Douglas engineering drawings but were below the value I had finally prodded McDonnell Douglas into revealing.

I could have been more diplomatic, I guess, but that wasn’t me.  I shared that information with the room.  The Army rep smiled.  “I think you guys might want to continue the meeting without me,” he said.  And then he left.

The McDonnell Douglas guy exploded as soon as the door closed.  He was apoplectic (I looked that word up; it means overcome with anger and extremely indignant, and that was him).  McDonnell Douglas had been screwing the Army for years with a deficient design and now it was out in the open.  They were potentially exposed to defective design claims from the Army (and from us) for hundreds of millions of dollars.  Think about it:  12 years of Apache blade production, a 50% failure rate in production, a blade life of only 800 hours (against the Army’s spec requirement of 2200 hours), and the fact that we and McDonnell had factored all that waste into our pricing.

Fortunately for McDonnell Douglas, the Army wasn’t interested in suing them (all they wanted was good blades).   My boss wasn’t interested in pursuing a claim against McDonnell Douglas, either, as they were our bread and butter and he wanted to keep the business.  We fixed the problem by holding the blade components to tighter tolerances (tighter than McDonnell Douglas had on their drawings) so the bondline width would always be above the magical 0.375 inch, and we never had a blade unzip in production again.  McDonnell Douglas did not correct their drawings, as it would have been an admission of guilt on their part that would absolutely guarantee a loss if the Army ever took them to court.

So there you have it:  The Apache main rotor blade failures, all caused by sloppy engineering at McDonnell Douglas.  It’s hard to believe that the blades had a 50% failure rate and didn’t meet the Army’s specified blade life for a dozen years before the problem was fixed, but that’s what happened.  It’s also hard to believe that nobody at McDonnell Douglas went to jail for it.

Never miss an ExNotes blog:

The Wayback Machine: Later, Gator…

By Joe Berk

I had a tough time choosing a title for this blog.  I went with what you see above because it reminds me of one of my favorite Dad jokes…you know, the one about how you tell the difference between a crocodile and an alligator.  If you don’t see it for a while, it’s a crocodile.  If you see it later, well, then it’s a gator.  The other choice might have been the old United Negro College Fund pitch:  A Mine is a Terrible Thing to Waste.  But if I went with that one I might be called a racist, which seems to be the default response these days anytime anyone disagrees with anyone else about anything.

Gresh likes hearing my war stories.  Not combat stories, but stories about the defense industry.  I never thought they were all that interesting, but Gresh is easily entertained and he’s a good traveling buddy, so I indulge him on occasion.  Real war stories…you know how you can tell them from fairy tales?  A fairy tale starts out with “once upon a time.”  A war story starts out with “this is no shit, you guys…”

Don’t forget to click on the popup ads…it’s how we get paid and how we keep the blog alive.

So, this is a “no shit” story.  It sounds incredible, but it’s all true.  I was an engineer at Aerojet Ordnance, and I made my bones analyzing cluster bomb failures.  They tell me I’m pretty good at it (I wrote a book about failure analysis, I still teach industry and gubmint guys how to analyze complex systems failures, and I sometimes work as an expert witness in this area).  It pays the rent and then some.

So this deal was on the Gator mine system, which was a real camel (you know, a horse designed by a committee).  The Gator mine system was a Tri-Service program (three services…the Army, the Navy, and the Air Force).  It was officially known as the CBU-89/B cluster munition (CBU stands for Cluster Bomb Unit).  The way it worked is instead of having to go out and place the mines manually, an airplane could fly in and drop a couple of these things, the bombs would open on the way down and dispense their mines (each cluster bomb contained 94 mines), the mines would arm, and voila, you had a minefield.  Just like that.

It sounds cool, but the Gator was a 20-year-old turkey that couldn’t pass the first article test (you had to build two complete systems and the Air Force would drop them…if the mines worked at a satisfactory level, you could start production).  The UNCF slogan notwithstanding, the folks who had tried to take this Tri-Service camel and build it to the government’s design wasted a lot of mines.  In 20 years, several defense contractors had taken Gator production contracts, and every one of them failed the first article flight test.  When my boss’s boss decided we would bid it at Aerojet, I knew two things:  We, too, would fail the first article flight test, and it would end up in my lap.  I was right on both counts.  We built the flight test units per the government design and just like every one else, we failed with a disappointing 50% mine function rate.  And I got the call to investigate why.

So, let’s back up a couple of centuries.  You know, we in the US get a lot of credit for pioneering mass production.   Rightly so, I think, but most folks are ignorant about what made it possible.  Nope, it wasn’t Henry Ford and his Model T assembly line.   It was something far more subtle, and that’s the concept of parts interchangeability.  Until parts interchangeability came along (which happened about a hundred years before old Henry did his thing), you couldn’t mass produce anything.  And to make parts interchangeable, you had to have two numbers for every part dimension:  The nominal dimension, and a tolerance around that dimension.  When we say we have a 19-inch wheel, for example, that’s the nominal dimension.  There’s also a ± tolerance (that’s read plus or minus) associated with that 19-inch dimension.  If the wheel diameter tolerance was ±0.005 inches, the wheel might be anywhere from 18.995 to 19.005 inches.  Some tolerances are a simple ± number, others are a + something and a – something if the tolerance band is not uniform (like you see in the drawing below).  But everything has a tolerance because you can’t always make parts exactly to the nominal dimension.

Where companies get sloppy is they do a lousy job assigning tolerances to nominal dimensions, and they do an even worse job analyzing the effects of the tolerances when parts are built at the tolerance extremes.  Analyzing these effects is called tolerance analysis.  Surprisingly, most engineering schools don’t teach it, and perhaps not so surprisingly, most companies don’t do it.  All this has been a very good thing for me, because I get to make a lot of money analyzing the failures this kind of engineering negligence causes.  In fact, the cover photo on my failure analysis book is an x-ray of an aircraft emergency egress system that failed because of negligent tolerancing (which killed two Navy pilots when their aircraft caught fire).

I don’t think people consciously think about this and decide they don’t need to do tolerance analysis.  I think they don’t do it because it is expensive and in many cases their engineers do not have the necessary skills.  At least, they don’t do it initially.  In production, when they have failures some companies are smart enough to return to the tolerancing issue.  That’s when they do the tolerance analysis they should have done during the design phase, and they find they have tolerance accumulations that can cause a problem.

Anyway, back to the Gator mine system.  The Gator system had a dispenser (a canister) designed by the Air Force, the mines were designed by the Army, and the system had an interface kit designed by the Navy.  Why they did it this way, I have no idea. It was about as dumb an approach for a development program as I have ever seen.  Your tax dollars at work, I guess.

The Navy’s Gator interface kit positioned the mines within the dispenser and sent an electronic pulse from the dispenser to the mines when it was time to start the mine arming sequence.  This signal went from coils in the interface kit to matching coils in each mine (there was no direct connection; the electric pulse passed from the interface kit coils to the mine coils).  You can see these coils in the photo below (they are the copper things).

In our first article flight test at Eglin Air Force Base, only about 50% of the mines worked.  That was weird, because when we tested the mines one at a time, they always worked.  I had a pretty good feeling that the mines weren’t getting the arming signal.   The Army liked that concept a lot (they had design responsibility for the mines), but the Air Force and the Navy were eyeing me the way a chicken might view Colonel Sanders.

I started asking questions about the tolerancing in the Navy’s part of the design, because I thought if the coils were not centered directly adjacent to the matching coils in each mine, the arming signal wouldn’t make it to the mine.  The Navy, you see, had the responsibility for the stuffing that held the mines in place and for the coils that brought the arming signal to the mines.

At a big meeting with the engineering high rollers from all three services, I floated this idea of coil misalignment due to tolerance accumulation.  The Navy guy basically went berserk and told me it could never happen.  His reaction was so extreme I knew I had to be on to something (in a Shakespearian methinks the lady doth protest too much sort of way).  At this point, both the Army and Air Force guys were smiling.  The Navy guy was staring daggers at me.  You could almost see smoke coming out his ears. He was a worm, I was the hook, and we were going fishing.  And we both knew it.

I asked the Navy engineer directly how much misalignment would prevent signal transmission, he kept telling me it couldn’t happen, and I kept pressing for a number:  How much coil misalignment would it take?  Finally, the Navy dude told me there would have to be at least a quarter of an inch misalignment between the Navy coils and those in the mine.  I don’t think he really knew, but he was throwing out a number to make it look like he did.  At that point, I was pretty sure I had him.  I looked at my engineering design manager and he left the room.  Why?  To do a tolerance analysis, of course.  Ten minutes later he was back with the numbers that showed the Navy’s interface kit tolerances could allow way more than a quarter inch of misalignment.

When I shared that with the guys in our Tri-Service camel committee, the Navy guy visibly deflated.  His 20-year secret was out.  The Army and the Air Force loved it (they both hated the Navy, and they really hated the Navy engineer).

We tightened the tolerances in our production and built two more cluster bombs.  I was at the load plant to oversee the load, assemble, and pack operation, and when we flight tested my two cluster bombs with live drops from an F-16 we had a 100% mine function rate (which had never been achieved before).  That allowed us to go into production and we made a ton of money on the Gator program.  I’m guessing that Navy weasel still hates me.

It’s hard to believe this kind of stuff goes on, but it does.  I’ve got lots of stories with similar tolerance-induced recurring failures, and maybe I’ll share another one or two here at some point.  Ask me about the Apache main rotor blade failures sometime…that’s another good one and I’ll post a blog about it in the next week or so.

Stay tuned, my friends.

Never miss an ExNotes blog: