Fail Safe Design

My engineer friend Rod, who pointed out some flaws in my math regarding the moon platform a couple of weeks ago, dropped me a note after last week’s post with some additional questions about the pneumatic system. In the midst of that email conversation, he wrote, “We try to design things such that if they fail, they fail safe. Not always possible, but always worth the consideration.” A simple statement, but worth unpacking.

The term failsafe, as defined by Merriam-Webster online, means, “incorporating some feature for automatically counteracting the effect of an anticipated possible source of failure.” A term often used to describe technical design that includes failsafe considerations is “single failure proof design.” Single failure proof designs incorporate elements such that the failure of any one element or component does not lead to damage or physical harm. Consider, for example, attaching a lighting instrument to a batten: usually, as part of this process, some kind of “safety cable” or “safety chain” is also attached to the batten, connecting the yoke of the lighting fixture to the batten; if the C-clamp fails, the safety cable provides additional security that the lighting fixture will not fall straight to the ground.

An effective (if somewhat tedious) tool for assessing the “single-failure-proofness” of an effect or mechanism is the failure-mode effect analysis, or FMEA. (Alan Hendrickson’s Mechanical Design for the Stage includes a very detailed section on developing a FMEA for mechanical effects.) In essence, a FMEA forces a technical designer to examine every individual part in a machine or effect and consider:

1. How it might fail
2. What could have caused that failure
3. What effect the failure will have on safety
4. How that failure mode could be eliminated or mitigated.

As I indicated above, this can be a tedious process—a simple effect such as the moon platform I’ve examined over the last few weeks has dozens of components simply in the pneumatic system (from hoses to fittings, valves to cylinders, etc.) to examine, for example. However, considering each of these components will often lead not only to a safer system in the end, but a more streamlined and efficient design overall. Further, many times the steps taken to mitigate the possibility of a failure include using reputable component manufacturers and/or instituting careful inspection and maintenance routines, which is relatively easy to implement. Finally, it is possible that many of the failure modes identified will result in the same effect, and finding a way to mitigate that one effect will address all of those failure modes at the same time.

Single failure proof design for the moon platform

In the case of the moon platform, after looking at the FMEA, we looked at the following failure modes very closely:

1. Battery (which supplies the power to the AC inverter, which powers the pneumatic directional control valve) loses charge
2. Battery becomes disconnected from inverter
3. Inverter fails
4. Directional control valve becomes disconnected from inverter
5. Air pressure drops low enough that the directional control valve cannot shift position
6. Air pressure drops low enough that the brakes do not hold the unit in place.

In all of these cases, the end result was the same: the brakes would not function. Immediately, we recognized that adding a secondary, manual braking mechanism would ameliorate concerns that the wagon would move when it should be stationary. In this way, we ensured that no single failure of a component would result in the moon being unsafe for actors to perform on. (Additionally, using four pneumatic cylinders, when two would have provided enough force, added another layer of redundancy to the design.)

Single failure proof design is more complicated than simply adding redundancy, however. Sometimes redundancy is not enough; a single failure proof design will also anticipate ways to control a failure such that it happens the way the designer wants it to, to ameliorate the effects of that failure. The FMEA can help to illuminate when this is required and how it might be achieved.

Our manual brakes certainly provided a backup braking system, but as we began looking at the function of the system—specifically regarding the six failure modes above—we realized that depending on the timing of any of those failures, the effects could be more or less dangerous. We determined that there were three different moments when the system could fail: first, when the moon was to be moved, with actors around the perimeter ready to engage/disengage the brakes using the switch installed in the facing; second, when the moon was in motion; third, when actors were on the moon, but no one was in a position to move the moon.

If we designed the system such that the brakes would engage when power was supplied to the valve (meaning that in a “normal state,” the brakes were disengaged), loss of power or pressure would have no appreciable effect if the moon were in motion; if it happened when the actors were ready to engage/disengage the brakes, they would be in position to immediately engage the manual brakes, and there would be no appreciable effect. However, if power loss or pressure loss happened when actors were performing on the wagon and no actors were in a place to engage the manual brakes, then the wagon would suddenly be free to move underneath the actors without warning.

Obviously, this was not the best option. We then considered what would happen if we designed the system such that its “normal state” was with the brakes on. If we designed the system such that without power supplied, or sufficient pressure to shift the directional control valve’s position, the brakes would be engaged, the potential dangers dependent on the timing of a failure were radically decreased. If any of the above failure modes occurred when the moon was stationary, there would be no appreciable effect until the actors tried to move it; if they failed at the moment the actors tried to move it (or some time before), they would be in a position to bleed the system to disengage the pneumatic brakes and engage the manual brakes at the end of the move.

The case where any of the failure modes occurred while the moon was in motion proved slightly less advantageous: in any of these cases, the pneumatic brakes would engage during the move. To mitigate this effect, we installed meter-out valves to ensure that the cylinders engaged slowly. Also working in our favor was the fact that the moves of the moon were all relatively slow (meaning that engaging the brakes mid-move would not result in a hugely jarring stop—just a slightly disconcerting one). Additionally, based on conversations with the director, it was clear that the moon would be stationary most of the time it was on stage.

We open Arabian Nights tomorrow, and it is worth noting that, indeed, the pneumatic braking system failed at least a half-dozen times during tech over the last week (culprits included a battery which would not hold sufficient charge, unexpected pressure leaks, and operator error). These failures all happened at different times--at least once for each of the different times we identified. In every instance, the actors (who'd been trained on what to do if a failure occurred) were able to bleed the system and shift to using the manual brakes without missing a beat. More importantly, at no time were any actors injured or at risk of injury: fail safe design at work.