Chapter 10 · Lesson 2 of 4 · about 12 minutes
Redundancy, voting and controlled recovery
Voting keeps the output right while one replica is wrong, provided the other replicas and the voter still work. Replicas often share a clock, a supply or a corrupted input, and a repaired copy has to rejoin in a consistent state, or one fault simply waits for the next one.
What you need to know
Three copies and a majority voter deliver the right output as long as at most one copy is wrong. For independent replicas that each fail with probability p over some period, the voted output fails with probability 3p² − 2p³. That is far below p when p is small and above p once p exceeds one half. The figure shows the same thing against time. With each replica failing at a constant rate λ and no repair, the triplicated reliability 3e^(−2λt) − 2e^(−3λt) stays above a single replica's e^(−λt) only until λt = ln 2, which is 69 days at the teaching rate of one fault per 100 days. Its mean time to failure is 5/(6λ), shorter than that of a single copy. Triple modular redundancy is therefore a technique for systems that get repaired. If the faulty copy is restored within an interval T, the voter fails only when two copies fail within the same interval, with probability of order 3(λT)², and the yearly failure rate falls with T in the same way the uncorrectable word rate fell with the scrub interval in lesson 10.1. In an SRAM-based FPGA the repair is configuration scrubbing, and the interval is the scrub time.
The equation assumes three things that implementations rarely provide: independent copies, a voter that cannot fail, and a repaired copy that rejoins in the right state. Real replicas share a clock, a reset, a supply and the input they vote on. A transient on the shared clock is a common cause, and the voter cannot outvote it. The voter is itself logic that can be upset; careful designs triplicate the voter as well and vote the outputs again downstream. A repaired copy that resumes with stale state, a counter or a state machine for example, is a second wrong copy waiting to happen. Triplicating the source code does not triplicate the hardware unless synthesis and placement preserve the three copies, so the evidence is fault injection on the implemented netlist or bitstream and a beam test of the implementation. A review of the source does not show it.
A watchdog turns a functional interrupt into a recovery, and it can be credited only for the faults it detects and the resets that restore the function. A timer serviced by a task that keeps running while the function has failed detects nothing. A monitor on the same supply or clock as the failed function disappears with it. A reset that restores the processor but leaves the peripheral configuration corrupted restores nothing the mission needs. Write the reset contract down: what is detected and within what time, what is reset and what is retained, what the safe outputs are during the reset, which boot image is used, and what happens when the reset repeats. Coverage is then measured per mode by fault injection, as the fraction of interrupts of each class that the monitor detects, times the fraction of resets that restore validated function.
Worked example 10.2. Triplication with repair, and the watchdog on U2
| Repair interval T | λT | Probability the voter fails within one interval, 3(λT)² − 2(λT)³ | Voter failures per year |
|---|---|---|---|
| 0.24 hours | 1.0 × 10⁻⁴ | 3.0 × 10⁻⁸ | 0.0011 |
| 2.4 hours | 1.0 × 10⁻³ | 3.0 × 10⁻⁶ | 0.011 |
| 1 day | 1.0 × 10⁻² | 3.0 × 10⁻⁴ | 0.109 |
| 10 days | 1.0 × 10⁻¹ | 2.8 × 10⁻² | 1.02 |
At one fault per 100 days per replica, a configuration scrub every 0.24 hours leaves 0.0011 voter failures per year from independent faults. Without repair, the triplicated system is worse than a single copy after 69 days. The number excludes common causes, which the injection campaign has to find separately. For U2's controller the interrupt rate is 0.026 per day, or 9.5 per year. A watchdog with a measured detection coverage of 90 %, and a reset that restores validated function in 98 % of the injected cases, leaves 1.12 uncovered interrupts per year (10 % undetected plus 1.8 % detected but not recovered). Each of those needs the next rung of the ladder, a ground-commanded power cycle, with its own hours of downtime. Whether 1.12 per year is acceptable depends on the function, that is, on how much recording is lost per power cycle, and chapter 11 takes that up.
Exercise
Review the fault domains as implemented, and verify how the voter, the shared resources and the repair path behave under representative faults.
Build a matrix of detection and restoration coverage by failure mode, and verify the recovery time and the safety of the outputs during recovery against what the mission needs.
Going deeper
The ideal voting equation has narrow assumptions
For three independent replicas with equal per-demand failure probability p and a perfect voter, the majority output fails with probability 3p² − 2p³. Common-cause faults, unequal channels, persistent configuration damage and the repair timing all change that expression, and the benefit can disappear when a shared path dominates. The dependencies assumed in the system model have to be checked against the hardware.
A running processor may still be wrong
A liveness check says nothing about whether the control outputs or the data are correct. The same fault can affect both the monitored function and its supervisor. A coverage model needs, for each mode, the probability of detection and the probability that the reset restores function, and it has to count false trips and resets that are themselves hazardous. Fault injection should reach internal state and interfaces, and not only clean task crashes.
A watchdog that shares the failed function's clock or reset authority may provide no independent containment.
The common mistake
The ideal equation for independent replicas says nothing about the coverage of the TMR as implemented, or about the reliability of the function it protects.
A reset cannot be assumed to restore damaged hardware, or every corrupted peripheral and every piece of retained state.
Takeaway
- What redundancy is worth depends on the independence of the copies, on containment and on repair, more than on how many copies there are.
- A device mode that is recoverable on the bench is recoverable in the system only if a recovery path exists, is available at the time, and has been verified.
Check yourself
Why is the mean time to failure of unrepaired TMR shorter than that of a single replica?
Because the voter needs two of the three copies working. With three copies failing independently, the time until the second failure is on average 5/(6λ), less than 1/λ. What TMR gives you is a high probability of surviving a short interval, and repair renews that interval.
A design review credits "TMR on the controller". What evidence is required to credit it?
Three independent copies in the implemented netlist or bitstream, with a voter that is itself protected. An analysis of the shared clock, reset and supply paths as common causes. Fault injection or beam testing on the implementation showing that the voted function survives single faults and that a repaired copy rejoins in a consistent state.
A watchdog timer is reset by the main loop every 100 ms. Which U2 interrupts does it miss?
Those in which the main loop keeps running while the memory interface has stopped. The timer is still serviced, so nothing is detected. The detector has to watch the function itself, for example data flowing and patterns verified, rather than whether a task is alive.
An SRAM-based FPGA carries triplicated logic at a fault rate of one per 100 days per replica, and the designer argues that with three copies the configuration scrub interval hardly matters. What does the worked example say?
With repair, the voter fails only when two copies fail within the same interval, with probability of order 3(λT)², so the yearly failure rate falls with the interval in the same way the uncorrectable word rate did in lesson 10.1. At one fault per 100 days, a scrub every 10 days leaves 1.02 voter failures per year, a daily scrub 0.109, and a scrub every 2.4 hours 0.011. The interval is what makes triplication useful; without repair the triplicated system falls below a single copy after 69 days. The figures exclude common causes, which the injection campaign has to find separately.
A configuration scrub has repaired the faulty copy of a triplicated state machine, and the review treats the system as restored. What else has to be shown?
That the repaired copy rejoined in a state consistent with the other two. A copy that resumes with stale state, a counter or a state machine for example, is a second wrong copy waiting to happen, and the next single fault then defeats the voter. The repair restores the copy; whether its state is resynchronized is a separate design question, and the evidence is fault injection or a beam test on the implementation showing that the repaired copy rejoins correctly.
Why does the course ask for a written reset contract before the watchdog is credited with any coverage?
Because coverage is the fraction of interrupts of each class that the monitor detects, times the fraction of resets that restore validated function, and both fractions depend on what the contract specifies: what is detected and within what time, what is reset and what is retained, the safe outputs during the reset, the boot image, and what happens when the reset repeats. A reset that restores the processor but leaves the peripheral configuration corrupted restores nothing the mission needs. For U2 the measured 90 % detection and 98 % restoration leave 1.12 uncovered interrupts per year, and that figure means nothing without the definitions of detected and restored it was measured against.
Further reading
- R. E. Lyons and W. Vanderkulk, "The use of triple-modular redundancy to improve computer reliability," IBM J. Res. Dev., vol. 6, no. 2, pp. 200-209, 1962.
- S. Mukherjee, Architecture Design for Soft Errors, Morgan Kaufmann, 2008.
- NASA, "Avionics Radiation Hardness Assurance (RHA) Guidelines," NASA/TM-20210018053, 2021. NTRS 20210018053.