space-rha.comShort courseChapter 10 · Mitigation and system behavior

Chapter 10 · Lesson 1 of 4 · about 18 minutes

Where to act, and memory protection

A mitigation can change the part, its operating conditions, the circuit, the architecture or the operations, and each of these acts at a different point between the exposure and the lost function. The choice is made against the mechanism and the function at hand, because a familiar technique helps only if it breaks the path that matters.

What you need to know

Every mitigation acts at some point on the path from the particle to the lost function, and the first thing to ask about any technique is which link of that path it breaks. A harder part, or a lower bias on the same part, reduces the event rate. Shielding changes the environment at the part; it is useful for dose and generally of little use against galactic ions. Filtering and error correction contain an event at the circuit, and redundancy and voting contain it at the architecture. Watchdogs and recovery ladders restore the function after the event. Operations can avoid the exposure or accept the loss. Each technique helps one effect and leaves the others untouched: a code does nothing for latchup, a limiter does nothing for dose, and a reset does nothing for a parameter that has drifted. The trade compares the candidate techniques against the mechanism and the function at hand, with their cost in mass, power, performance and schedule, and with the evidence each one needs before it can be credited.

An error-correcting code protects a defined word against a defined error pattern. The single-error-correcting, double-error-detecting (SEC-DED) code on a 64-bit word adds 8 check bits. It corrects one wrong bit among the 72 and detects two. It fails in two ways: when a word accumulates two upsets between corrections, and immediately when one particle puts two bits in the same word. The scrub interval controls the first failure. A scrubber reads every word, corrects single errors and writes the corrected word back, so a word rarely holds an error long enough to collect a second one. With an upset rate λ per bit per day and an interval T, each bit in a word has probability q = λT of upsetting between scrubs, the word fails with probability of order C(72, 2) q², and the number of uncorrectable words per year scales with T. Scrubbing does nothing about the second failure. Multi-cell upsets are common for heavy ions and not rare for protons in scaled DRAM. Whether the cells of one cluster land in one word depends on the physical layout, and the standard answer is interleaving, which maps physically adjacent bits to different words so that a cluster becomes several single errors. Whether the interleaving is sufficient is decided by the measured multiplicity distribution from the test (chapter 8 lesson 1), mapped through the physical-to-logical layout. If the bit interleaving distance is smaller than the largest measured cluster, the code has a bypass that no scrub interval closes.

The code and the scrubber are hardware with failure modes of their own. The correction engine, the address path and the mode registers of the memory are all outside the protected word. A SEFI in the controller stops the scrubbing and the recording together, and that is the event the watchdog of lesson 10.2 has to catch. When you credit a technique, you credit the implemented mechanism, in its configuration, under the fault patterns it was tested against. The name of the technique on a block diagram is not evidence of any of that.

Figure 10.1. Uncorrectable words per year in U2 in a sun-synchronous orbit against the scrub interval, for the accumulation of independent upsets in a 72-bit word; the horizontal line is the rate at which a single event places two bits in one word if the array is not interleaved (3 % of events, a teaching value).
Uncorrectable words per year in U2 against the scrub interval 0.1 1 10 100 1000 10-4 0.001 0.01 0.1 1 10 100 1000 104 scrub interval, hours uncorrectable words per year one event placing two bits in one word, not interleaved: 1538 per year independent upsets accumulating in one 72-bit word
Try it: the scrub interval of a SEC-DED memory

Set the device upset rate (heavy ions plus protons from chapter 9), the array size, the code word, and the scrub interval; watch the accumulation term fall with the interval while the multi-bit term does not move.

Worked example 10.1. Sizing the U2 scrubber

U2 in a sun-synchronous orbit sees 140 upsets per day (chapter 9), which is 1.3 × 10⁻⁷ per bit per day over 1,073,741,824 bits, organized as 16,777,216 words of 64 data bits with 8 check bits.

Scrub intervalProbability a word collects two or more upsets in one intervalUncorrectable words per year (accumulation)
6 minutes7.6 × 10⁻¹⁶0.00111
1 hour7.6 × 10⁻¹⁴0.0111
6 hours2.7 × 10⁻¹²0.0669
24 hours4.4 × 10⁻¹¹0.268
7 days2.1 × 10⁻⁹1.87
30 days3.9 × 10⁻⁸8.03

A daily scrub leaves 0.27 uncorrectable words per year from accumulation, and an hourly scrub leaves 0.011. Compare that with the multi-bit term. If 3 % of events put two bits in one word because the array is not interleaved, the uncorrectable rate is 4.2 per day, or 1538 per year, and no scrub interval changes it. So the design decision is interleaving first, sized from the measured cluster distribution, and the scrub interval second, chosen so that the accumulation term is small against the number of lost words the mission can tolerate. With interleaving in place, an interval of a few hours is more than enough for an average day. The interval then has to be checked against the anomaly pass, where the rate is 50 times the average for a few minutes, and against the event week of chapter 9 lesson 2.

The four teaching parts in this chapter

Four constructed parts run through chapters 6 to 12 so that each chapter's work can be followed on the same hardware: U1, a shared regulator that powers sensing, logic and the recovery controller; U2, a 1 Gbit recorder memory under SEC-DED error correction and periodic scrubbing; U3, a precision analog chain with an accuracy allocation; and U4, an optical interface that must keep its signal margin. Their response curves are stated in the build (constructed from a declared true curve with Poisson sampling), so that every number in these chapters can be traced; they are teaching cases, not commercial devices.

PartFunctionThis chapter's step
U1 · shared powerA shared regulator powers sensing, logic and the recovery controller.Compare independent power for the recovery controller, current limiting, isolation, and a different part or bias. Measure the output capacitance, the shutdown latency and the behavior of the controller itself.
U2 · recorder memoryA memory stores science data under ECC and periodic scrubbing.See how a cluster of adjacent physical bits lands in contiguous and in interleaved code words. Treat the scrub interval, an unobserved SEFI and the scrubber hardware as separate paths to a lost word.
U3 · precision analogA precision analog chain has to hold its accuracy and timing allocation.Compare the real headroom, calibration, redundancy and filtering. A first-order filter that attenuates the transient slows the wanted step by the same time constant.
U4 · optical interfaceAn optical interface has to keep enough signal margin through the mission.Compare the initial headroom, a higher drive current, redundancy, temperature and the choice of part. Extra drive current costs LED lifetime and thermal budget.

Exercise

For each proposed mitigation, write down which failure mode it targets, what it depends on, how it will be verified, and what it leaves uncovered.

Then inject error patterns into code words, including the multi-cell clusters the beam produced, and measure detection, correction and scrub completion.

Going deeper

Combining coverage across layers

Layers of mitigation often share detectors, clocks, software, supplies and assumptions, so multiplying their nominal coverage figures together overstates the combined coverage. The model has to state, for each originating failure mode, the probability that it is detected, contained and restored, including faults and damage in the mitigation hardware itself. The model is revisited whenever the implementation or its configuration changes.

The credit goes to the implemented mechanism in its tested operating envelope, not to the name of the technique.

Accumulation and single-event clusters are different

The independent-bit model shows why a shorter scrub interval leaves fewer chances for two upsets to meet in one word. A single particle that puts a correlated error into one word bypasses that timing argument entirely. How the code behaves beyond its guaranteed distance also depends on the decoder. A mission model therefore uses the measured multiplicity distribution, the physical-to-logical mapping and the observed behavior of the memory system.

Squaring a measured single-bit error rate does not give the multi-bit event rate; that rate comes from the measured multiplicity distribution.

The common mistake

No single technique covers cumulative damage, recoverable events and destructive mechanisms together.

The independent-bit teaching model leaves out correlated errors, decoder faults and failures outside the protected data path.

Takeaway

  • The trade is which technique protects the required function, with evidence that can be defended and a cost the project can carry.
  • ECC and scrubbing work when the fault patterns the memory actually sees fit the code, the interleaving and the scrub interval.

Check yourself

The scrub interval is halved. What happens to each term of the uncorrectable rate?

The accumulation term falls by about half, because it scales with the interval. The single-event multi-bit term does not change, because it needs no second upset.

A test measured 4 % of heavy-ion events as multi-cell with clusters up to 3 bits. What layout fact decides whether the code survives them?

The bit interleaving distance. If physically adjacent bits are mapped to different code words at a distance of at least the largest cluster, each cluster becomes single errors in several words. If not, a cluster lands in one word, and that word is uncorrectable at once.

A SEFI stops the memory controller for 90 s. What does the code do during that time?

Nothing. The correction engine and the scrubber are part of the stopped controller. The watchdog and the reset ladder cover the SEFI, and its downtime is counted in the availability of chapter 9 lesson 3.

A colleague estimates the multi-bit upset rate of U2 by squaring the measured single-bit rate, on the grounds that the code fails only when two upsets meet in one word. What is wrong with that?

Two independent upsets meeting in one word is only one of the two failure modes, and it is the one the scrub interval controls. The other is one particle placing two bits in the same word, which needs no second upset and whose rate comes from the measured multiplicity distribution mapped through the physical-to-logical layout. In the worked example that term, at 3 % of events in an array that is not interleaved, is 1538 per year, against 0.27 per year from accumulation with a daily scrub, so squaring the single-bit rate misses the larger term entirely.

In the worked example the probability that a word collects two upsets in one interval grows with the square of the interval, yet the uncorrectable words per year grow only in proportion to it. Why?

Each bit has probability q = λT of upsetting within an interval, so the per-interval failure probability of a word is of order C(72, 2) q², which grows as T². There are 1/T intervals in a year, so the yearly count grows as T; the table shows 0.011 per year at one hour and 0.27 at 24 hours. The rule holds for independent upsets while q is small, which it is for U2 at 1.3 × 10⁻⁷ per bit per day, and it says nothing about correlated errors from one particle, decoder faults or failures outside the protected data path.

A review multiplies the nominal coverage figures of the ECC, the scrubber and the watchdog on U2 to arrive at a combined coverage. Why is that product generally an overstatement?

Because the layers often share detectors, clocks, software, supplies and assumptions, so their failures are not independent. A SEFI in the memory controller stops the correction engine and the scrubber together, and a watchdog on the same supply or clock disappears with the function it watches (lesson 10.2). The combined model has to state, for each originating failure mode, the probability that it is detected, contained and restored, including faults and damage in the mitigation hardware itself, and it is revisited whenever the implementation or its configuration changes.

Further reading

  • A. M. Saleh, J. J. Serrano and J. H. Patel, "Reliability of scrubbing recovery-techniques for memory systems," IEEE Trans. Reliability, vol. 39, no. 1, pp. 114-122, 1990.
  • S. Mukherjee, Architecture Design for Soft Errors, Morgan Kaufmann, 2008.
  • G. M. Swift and S. M. Guertin, "In-flight observations of multiple-bit upset in DRAMs," IEEE Trans. Nucl. Sci., vol. 47, no. 6, pp. 2386-2391, 2000.
  • NASA, "Avionics Radiation Hardness Assurance (RHA) Guidelines," NASA/TM-20210018053, 2021. NTRS 20210018053.