Mission availability and reliability under SEE

Why a Markov chain, how the chain is built from the single event modes, what it assumes, how it is solved, how total dose enters, and where the model assumptions no longer apply.

Why a Markov chain for single event effects

Fault trees and reliability block diagrams describe combinations of failed components; they cannot describe repair, sequence, or the fact that a system in the middle of recovering from one fault is more vulnerable to the next. Single event effects have all three: the events arrive as a Poisson process at a rate fixed by the environment, the mitigations are repairs with characteristic times (a scrub cycle, a watchdog reset, a power cycle, a reconfiguration), and the interesting failures are the coincidences, a second SEFI during a reboot or a latchup while the redundant unit is already down. Boyd's tutorial [1] lays out why this is the case for fault tolerant systems in general and what the method demands: a state space small enough to build, transition rates that are constant in each state, and attention to coverage, the fraction of faults that the detection actually catches, which in his experience dominates the answer more often than the fault rate does. All three hold here, and the third is why the SEFI mode carries a coverage and a latency rather than just a rate.

The site's EDAC tool is already a small chain of this kind for one memory. This page generalizes it to a function delivered by redundant units and to the whole set of functions, so that the per part rates the rate tool produces and the per function requirements the SEE Analysis Workbook derives meet in one calculation.

The unit template

Each unit of a function is in one of four states: up; in a restart class outage (a SEFI or an uncorrected upset that a reset, scrub or reconfiguration clears, or an interruption that waits for ground intervention); in a power cycle outage (a latchup caught by a current limiter and cleared by removing power); or failed permanently (burnout, gate rupture, an unprotected or destructive latchup, a stuck bit that retires capacity, or any mode with no recovery). The rates out of the up state are per device per day from test data and the rate tool; the rates back are the reciprocals of mean outage durations from the design. These are the transitions drawn in the diagram on the page, and everything else the chain contains is bookkeeping about how many units are in each state.

Pooling modes into classes

A unit may have several modes that land in the same class, a SEFI at one rate with a 45 s reboot and an uncorrectable memory error at another rate with the same reboot, or a SEFI with a 90 percent watchdog coverage. The chain needs one rate and one recovery rate per class per unit, so modes are pooled: the class rate is the sum of the mode rates and the class recovery rate is the rate weighted harmonic mean of the mode durations, which preserves the mean outage time. Coverage enters as a duration: an event caught with probability c is down for the recovery time, and an uncaught one for the detection latency plus the recovery time, so the mean duration is c t_rec + (1 − c)(t_lat + t_rec). For availability this is exact; for the strict reliability question (does the function ever go down) only the rate matters, and pooling does not touch it. For SEU and SET modes an outage fraction says what part of the raw upset rate survives mitigation as a functional interruption; the EDAC tool gives that number for a memory. A protected latchup splits at the event into a power cycle outage with probability 1 − p_d and a permanent failure with probability p_d, the latent damage fraction the protection tool's verification test is meant to bound.

Redundancy, spares and switchover

A function is k of n identical units. Because the units are identical the chain is lumped: its state is the count of units in each template state (u up, r₁ restarting, r₂ power cycling, f failed) rather than the identity of each, which turns 4ⁿ states into (n + 3 choose 3), 35 for four units instead of 256, and it is exact for identical units. Two redundancy styles are modeled. Hot: all n units are powered and exposed, and the function is up when u ≥ k; a voting triplet or parallel strings. Cold: k units are active and the rest are unpowered spares that see no single events (they do accumulate dose, which is handled separately). For cold spares the failover rule matters. With failover on permanent failure only, the realistic case for most flight computers, a reboot of the active unit is an outage of the function for the reboot time, because nobody switches to the spare for 45 seconds; only when the active unit fails for good does a spare take over. With failover on any fault, the spare is brought up whenever the active unit stops and the rebooted unit becomes the spare. Each switchover succeeds with the coverage probability; a failed switchover puts the function into a ground recovery state for the entered hours, after which the spare is in service. Coverage below one is what turns a nominally redundant function into one whose reliability is set by the failover logic rather than by the parts, which is Boyd's point again in different clothing.

The storm environment

Rates are not constant over a mission: a solar particle event raises the proton flux by orders of magnitude for a few days, and heavy ion rates with it. Averaging that into the mission rate loses the coincidences, and running a separate worst day case loses the probability of it. The chain instead carries an environment state that alternates between quiet and storm with the entered frequency and mean duration, and multiplies each effect type's rate in the storm state. This is a Markov modulated Poisson process, and the storm exposure becomes part of every result: the expected number of events includes the storm contribution weighted by the time spent in storm, the reliability includes the chance that two events coincide during a storm, and the fraction of the mission in storm is reported. The multipliers are yours to set from the environment model, and they are typically 10 to 100 for upsets and transients and less for the deep structure effects, latchup and burnout, whose thresholds sit above most of a proton dominated event.

Solving the chain

With generator Q (rates in per second) the state probability row vector obeys dp/dt = pQ and p(t) = p(0) exp(Qt). The mission is long and the recovery rates are fast, so uniformization is impractical (the product of the largest rate and the mission time is of order 10⁶). The tool computes the one day transition matrix M = exp(Q · 86400 s) once by scaling and squaring with a Taylor series on the scaled matrix, then steps the vector through the mission a day at a time. A second matrix with the down states made absorbing gives the first passage distribution for the strict reliability. Chains here have at most a few hundred states (eight units with storms and switchover is 660) and solve in under a second. Expected counts of events and outages are the time integrals of state probability times transition rate, accumulated by the trapezoid rule on the daily grid; because outage states equilibrate within hours, the daily grid is adequate for the integrals even though it does not resolve an individual outage. Everything runs in the browser.

What is reported and how

Availability A(t) is the probability of being in an up state, and its mission mean is one minus expected downtime over mission time; it includes permanent loss, so a function that is likely to die halfway through has a mean availability near one half however quickly it recovers from upsets before that. Two reliabilities are reported because two different requirements exist. R permanent is the probability that fewer than k units have failed for good by time t, the right number for a function that tolerates interruptions. R no outage is the probability that the function has never been down at all, from the absorbing chain, the right number for a function whose brief loss is a loss of mission (a thruster valve driver during a burn). Expected outages are split by the cause of the transition from an up state to a down state: restart class, power cycle, permanent failure, or failed switchover; the dominant cause is where the design effort goes. Unit level resets and power cycles are counted regardless of system state, and the power cycle count is what the latchup protection has to be shown to survive.

Total dose as a lot level hazard

Total dose is not a random arrival and does not belong in the chain as a rate. It is degradation with accumulated dose, the units of a function are one lot at nearly the same dose, and so they fail together: redundancy buys almost nothing against it. The tool therefore treats TID as a survival factor common to all units of the function: with dose D(t) growing linearly at the entered rate and a lognormal lot failure dose with median D₅₀ and log sigma from a lot test or the hardness priors, S_TID(t) = 1 − Φ((ln D(t) − ln D₅₀)/σ), and the function's reliability and availability are multiplied by it. If the units are from independent lots the option computes a k of n survival instead. Coupling in the other direction, dose changing the single event rates, is real for some parts and is not modeled; the data to do it are thin.

From functions to system

Functions are combined as a series system: the mission survives if every critical function survives, and the system is up when every function is up, so the system curves are products over functions. This assumes the functions fail independently, which the storm environment violates (every function sees the same storm at the same time) and which shared resources violate (one power converter feeding two functions). The first makes the product optimistic by a small amount during storms; the second is handled by modeling the shared resource as its own function. Functions marked service count toward availability but not toward mission reliability.

The model as an ontology instance

The JSON model on the page is deliberately more than a settings file. Its vocabulary, function, unit, mode, effect type, detection coverage, recovery action and time, redundancy and failover, environment, dose hazard, is the set of terms a system level ontology of single event effects needs, and the relations are the ones the chain is generated from: a mode is caused in a unit, is detected with a coverage, is recovered by an action of a duration, and propagates to a function outage or a permanent loss depending on the redundancy. Writing the model down this way makes two things possible. The chain is generated rather than hand drawn, so changing a recovery action changes the analysis consistently everywhere. And rules can be checked mechanically: a SEFI mode with no recovery and no spare is a permanent loss; a critical function with a single unit and a latchup mode without protection has a destructive branch; a cold spare with coverage below one has a ground recovery state whose duration matters. The realist ontology literature [4] argues for deeper formalization than this; the position taken here is that the vocabulary and the generated analysis are the value, and that the format should stay light enough for a parts engineer to edit by hand.

Checks

The solver was checked against closed forms: the two state availability μ/(λ + μ) and its expected outage count; hot one of two reliability 1 − (1 − e^−λt)²; cold standby reliability e^−λt(1 + λt); cold standby with imperfect coverage e^−λt(1 + cλt) and its expected switchover failures (1 − c)(1 − e^−λt); and the storm modulated event count against the time weighted rate. All agree to the precision of the daily grid, better than 0.5 percent on the integrated quantities and to six digits on the probabilities. The example models are synthetic and are marked as such; their rates are in the range the rate tool produces for commercial parts in LEO and are not measurements of any named part.

Limits

Repair times are exponential in the chain. For availability that costs nothing, since only the mean matters; for the strict reliability of a function whose recovery is a fixed 30 s reboot, the exponential overstates the chance of a long recovery during which a second event lands, which is conservative. Units within a function are identical; a function with two different parts in parallel is two functions with a shared requirement, which the product handles approximately. Common cause single events (one SEFI in a shared clock taking down both strings) are not modeled except by making the shared part its own function. The environment has two levels; a phased mission with different orbits is run as separate models. Dose does not change the single event rates. The chain knows nothing about software, operations or the ground segment beyond the numbers you give it for coverage and recovery, and those numbers are the ones most worth arguing about.