Chapter 12 · Lesson 4 of 4 · about 13 minutes
Responding and closing the loop
The operational response is often needed before the root cause is settled. It follows the mission's fault-response authority and constraints, and it keeps the data needed to investigate. A reset restores the function, but it also erases the state you would need to find the cause, so the choice between immediate recovery and evidence capture is made in advance, for each kind of event, in a prepared response plan.
What you need to know
The response to an anomaly follows the mission's fault-protection authority, and it generally has to act before the cause is settled. A response plan written in advance settles the trade between recovery and evidence. It says which state is captured before any command is sent (the event log, the current trace buffer, the memory region with the uncorrectable word, the configuration registers), which telemetry is protected from overwrite, which safe diagnostics can be run before a reset (a read of the affected region, a parameter sweep), and which recovery step is then taken, in the ladder order established by the ground test. A reset restores the function, but it also erases the state you would need to find the cause, so the plan decides for each failure class which comes first. A workaround such as a mode restriction, a changed scrub interval or a string swap is checked for side effects and for its own dependencies before anyone relies on it, and the risk estimate is revised as the evidence improves.
Flight experience is evidence the ground test could not provide, and it improves the next assessment only if it is recorded with its limits. A rate over a year, with its live time and event definition, updates the factor of two on the proton model. A recovery that took longer than it did on the bench updates the availability estimate. A class of interrupt that the watchdog missed updates the coverage. A quiet record taken with coarse telemetry updates nothing. Repeated successful recoveries do not show that the part is undamaged. Recurring faults can share a driver, a corrupted state or progressive degradation, so the leakage and parameter trends of the affected part are read together with the event counts. Lessons are written as updates to specific assumptions in the argument (A1, A2, A3 of chapter 11), together with the configuration and exposure they apply to, so that the next project can decide whether they transfer.
The loop closes where the course began, with the mission question, the environment, the requirements, the parts and the evidence, each revisited with what flight has shown. A successful mission proves that this configuration survived this exposure, as far as the telemetry could see. It does not prove the next lot, the next architecture or the next orbit immune. The record that lets a later engineer tell the difference is the one designed in chapter 12 lesson 1.
Worked example 12.4. Capture before recovery, and the feedback register
| Failure class | Capture before any command | Safe diagnostics | Recovery rung | Evidence retained |
|---|---|---|---|---|
| U2 uncorrectable word | address, syndrome, surrounding rows; scrub counter; time and position | read the region twice; compare with the pattern | rewrite the region from the backup; no reset | cluster size for the interleaving check; whether the word recurs (stuck bit) |
| U2 interrupt | interrupt counter and controller status; current trace | none; the interface is down | the ladder, timed | rung that cleared it; time to validated function against the bench 87.5 s |
| U1 high-current event | current trace buffer, trip counter, rail voltage, mode | leakage read at the next calibration | automatic (limiter and restart); confirm by telemetry | latent-damage trend; the event position and time |
| U3 offset drift beyond budget | offset history and temperature | calibration sweep | enable the in-flight offset correction | drift rate against the LDR prediction |
| Flight observation after one year | Assumption it updates | Change to the argument |
|---|---|---|
| U2 corrected errors at about 77 per day against 140 predicted | the proton term of chapter 9 | the proton model is revised toward an energy-dependent cross section; the factor of two stays in the budget |
| 9 interrupts, all cleared by the ladder within 88 s | watchdog coverage estimated by injection (10.2) | coverage updated from the flight record; the bound on the uncovered fraction tightens |
| one U1 high-current event, recovered; leakage unchanged after six months | the U1 rate prediction (9.1), the residual risk statement (11.2) and the latent-damage open item | one event in 410 days exceeds the worst-case expectation of 1.9 × 10⁻³ by three orders of magnitude. The prediction is reopened (angular data for the depth, a proton run, solar heavy ions) and the risk statement is reissued with the flight bound. The open item gets a flight data point but stays open |
| no U3 drift beyond the thermal trend | the LDR offset prediction | consistent so far. The prediction is for end of life, and one year is one third of the mission dose, so no change |
Each line names the assumption and the configuration it was observed on, so that the next project can read the register as evidence with limits rather than as a success story.
Exercise
Draft a capture-before-recovery list for each failure class, an action log, and a follow-up test plan whose scope is limited to what would discriminate the competing hypotheses.
Write a register that maps each flight lesson to the assumption it updates, as in worked example 12.4, and issue it as a versioned update to the RHA evidence and the monitoring baseline.
Going deeper
Repeated recovery is not evidence of no damage
Recurring faults can share an environmental driver, a state that was never corrected, or progressive physical degradation. A successful restoration and a healthy part are different observations. When repeated events are reviewed, latent damage, shifted calibration and cumulative loss of margin are looked for. A follow-up ground test is designed around the flight state and the limits of the flight measurement; reproducing only the final symptom tells you little about what led to it.
The sequence of commands sent during the response is kept, so that the investigation can tell what the hardware did on its own from what it did because it was commanded.
A validation covers only what it compared
Agreement can come from model errors that cancel, and disagreement can come from missing exposure or detection information rather than from the model. The model is updated when the physical and measurement evidence supports the change. A model tuned to one anomaly and then presented as independently validated is a common mistake, so where it can be arranged, a comparison or an independent measurement is held back for the check.
The written feedback says which of three things happened: a recalibration, a corroboration of the existing model, or a validation of a new one.
The common mistake
A successful mission does not show that another lot, another architecture or another mission is immune.
Takeaway
- A prepared response plan settles in advance how to restore the mission function, keep operations safe and preserve the evidence that would separate the causes.
- Flight feedback improves the next assurance decision only when each item of evidence keeps the configuration and exposure it was observed on and the limits of its attribution.
Check yourself
A reset would clear a recorder interrupt immediately, and the ground is in contact. What does the response plan say to do first?
Capture first: the interrupt counter and controller status, the current trace and the configuration registers, all of which the reset would erase. Then run the ladder, timed, so that the recovery itself becomes a data point.
After three recoveries from U1 events in a year, the project proposes closing the latent-damage item. What is the counter-argument?
Recovery and health are different observations. Latent damage shows up as a leakage or parameter trend, which has to be read over time, and three events in a year is itself a rate that should be compared with the prediction before the risk statement is reissued.
Which flight observations are worth passing to the next project, and in what form?
Rates with their live time and event definitions, recoveries with their timing, coverage updates and parameter trends, each tied to the configuration and exposure it was observed on and to the assumption it updated. A summary without those limits is of no use to the next project.
After one year the U3 offset shows no drift beyond the thermal trend, and the project proposes recording the low-dose-rate prediction as "confirmed by flight". What does the feedback register say instead?
Consistent so far, with no change to the argument. The prediction is for end of life, and one year is one third of the mission dose, so the observation cannot yet test an offset error that was predicted to reach 12 mV at the full dose (lesson 10.4). The register records the observation against the assumption it bears on, the LDR offset prediction, with the configuration and exposure it was observed on, and the offset history and temperature keep being captured so that a later drift can be read against the thermal trend. A quiet record taken with coarse telemetry would update nothing at all.
The response plan lets the U2 uncorrectable word be read twice before anything is rewritten, but for a U2 interrupt it runs the ladder with no diagnostics at all. Why the difference?
The plan decides for each failure class which comes first, recovery or evidence, according to what the state can still tell. For the uncorrectable word the interface is up, so the address, syndrome and surrounding rows are captured and the region is read twice against the pattern; that evidence gives the cluster size for the interleaving check and tells a recurring address (a stuck bit) from a cluster, and the recovery is a rewrite from the backup with no reset. For the interrupt the interface is down and there are no safe diagnostics, so the counter, the controller status and the current trace are captured and the ladder then runs, timed, so that the time to validated function against the bench 87.5 s is itself a data point.
A model retuned after the U2 flight rate came in at 77 per day is presented at the next review as "validated by flight". What is the objection?
A model tuned to one anomaly and then presented as independently validated is a common mistake. The flight rate was used to revise the proton term toward an energy-dependent cross section, so the same record cannot also validate the revision; a validation covers only what it compared, and agreement can come from model errors that cancel. Where it can be arranged, a comparison or an independent measurement is held back for the check. The written feedback says which of three things happened, a recalibration, a corroboration of the existing model or a validation of a new one, and this case is a recalibration, with the factor of two kept in the budget.
Further reading
- NASA, "Avionics Radiation Hardness Assurance (RHA) Guidelines," NASA/TM-20210018053, 2021. NTRS 20210018053.
- K. A. LaBel et al., "Emerging radiation hardness assurance (RHA) issues: a NASA approach for space flight programs," IEEE Trans. Nucl. Sci., vol. 45, no. 6, pp. 2727-2736, 1998. doi:10.1109/23.736521.
- C. Poivey et al., "In-flight observations of long-term single-event effect (SEE) performance on Orbview-2 solid state recorders (SSR)," IEEE Trans. Nucl. Sci., vol. 50, no. 6, pp. 2169-2174, 2003.
- D. L. Hansen et al., "Correlation of prediction to on-orbit SEU performance for a commercial 0.25-µm CMOS SRAM," IEEE Trans. Nucl. Sci., vol. 54, no. 6, pp. 2525-2533, 2007. doi:10.1109/TNS.2007.908787.