space-rha.comShort courseChapter 12 · Flight monitoring and anomalies

Chapter 12 · Lesson 1 of 4 · about 14 minutes

Observability and the timeline

If all that was recorded is a daily reset count and a coarse shared-bus current, a later anomaly can be impossible to explain. The telemetry is designed around the functions and failure paths identified during RHA, and normal behavior is recorded too, because whether a change is meaningful often depends on the baseline it is compared against.

What you need to know

An anomaly can only be explained from what was recorded before and during it, so the telemetry has to be designed before launch, around the failure paths that the assurance argument identified. For the recorder function that means the corrected-error count and the syndromes of U2, with their addresses so that a cluster can be told from a stuck bit. It also means the scrubber's status, the interrupt counter with the rung that cleared each interrupt, U1's rail current and voltage at a cadence fast enough to show a trip, the limiter trip counter, U3's offset trend with temperature, U4's drive current and received signal, the mode and configuration identifiers, and the reset causes. Normal behavior is recorded as carefully as anomalies, because the baseline is what an anomaly is compared against. An error count that doubles means something only when you can set it beside the count from the previous month at the same orbit phase. Counter rollover and resolution matter as well. A daily reset count and a coarse shared-bus current are the kind of record that leaves an anomaly impossible to investigate.

The first alarm is rarely the first fault. An event propagates through the system, from a current step on a rail to a brown-out on a load, a controller reset, a lost bus transaction, a fault-protection response and finally a safe mode, and each step is logged by a different subsystem with its own clock. The timeline is built from the earliest evidence that can be supported. It keeps the time of physical occurrence separate from the detection, the onboard logging, the downlink and the ground receipt, because those can be hours apart. Clock conversions from spacecraft time to ground time are done deliberately, with the offset and drift recorded, and the timestamp resolution of each source is kept as an uncertainty interval. A sampled current that rises before a logged reset can still have risen after the true internal fault. If you want to associate the event with the anomaly region, you need the spacecraft position at the time the event occurred, which can be hours before the time anyone noticed it.

The telemetry itself is a measurement, and it has a detection efficiency of its own. A counter saturates, a reset erases a log, the telemetry drops out for a pass, a monitor was disabled during a maneuver. So a missing record can mean no event, lost telemetry, disabled monitoring or overwritten state, and the list of what each channel can and cannot see, which says which of those is possible, is written when the telemetry is designed. Recovery can erase the state that would separate the causes. That is the reason chapter 12 lesson 4 asks for capture before recovery.

Worked example 12.1. What the recorder telemetry can see, and one reconstructed timeline

Failure pathSignalCadence and resolutionWhat it can exclude
U2 upset cluster beyond the codeuncorrectable-error flag with address and syndromeper event, address to the worda stuck bit (the same address recurs) against a cluster (adjacent physical rows)
U2 interruptinterrupt counter; rung that cleared it; duration to validated functionper event, 1 sa proton-pass association needs the position at the event time
U1 high-current eventrail current at 10 ms; limiter trip counter; a 1 MS/s trace buffer released by the trip; rail voltage10 ms, 1 mA; the buffer at 1 µsa load-driven current rise (slow, in step with a mode change) against a step (within one sample, resolved in the buffer)
U3 driftoffset at the calibration points; board temperatureper orbita thermal trend (which tracks temperature and reverses with it) against a dose trend (which generally does not)
U4 margindrive current and received amplitudeper orbitcontamination or pointing (optical) against CTR loss (electrical, generally progressive, with partial annealing possible)
Scrubber alivescrub cycle counterper hourthat the assumption of lesson 10.1 holds in flight

The reconstructed sequence reads as follows. At spacecraft time T the U1 rail current steps from 0.31 A to 0.52 A within one 10 ms sample, so the occurrence is placed at T ± 10 ms. The limiter trips within the same sample: the trip counter increments, and the 1 MS/s trace buffer that the trip releases shows the step and the trip 12 µs apart, consistent with the 10 µs latency of lesson 10.3. The event log itself has only 1 s resolution. At T + 2.1 s the recorder controller logs a reset cause "supply". At T + 90 s the recorder reports ready. At T + 14 min the fault-protection log shows no action. Ground receipt is T + 6 h at the next pass. At time T the spacecraft was at 48° S, 20° W, inside the anomaly footprint of chapter 1. That position supports a radiation hypothesis without establishing it, and the alternatives of lesson 12.2 are kept open alongside it.

The four teaching parts in this chapter

Four constructed parts run through chapters 6 to 12 so that each chapter's work can be followed on the same hardware: U1, a shared regulator that powers sensing, logic and the recovery controller; U2, a 1 Gbit recorder memory under SEC-DED error correction and periodic scrubbing; U3, a precision analog chain with an accuracy allocation; and U4, an optical interface that must keep its signal margin. Their response curves are stated in the build (constructed from a declared true curve with Poisson sampling), so that every number in these chapters can be traced; they are teaching cases, not commercial devices.

PartFunctionThis chapter's step
U1 · shared powerA shared regulator powers the sensing, the logic and the recovery controller.Record the rail current and voltage, the reset cause, the mode and the timestamps. For a high-current step, weigh a radiation event against supply instability and a software-driven load.
U2 · recorder memoryA memory stores science data under ECC with periodic scrubbing.Keep the correction counts, the uncorrectable syndromes, the resets, the live time and the configuration. A missing error log may mean lost telemetry or a log erased by a reset.
U3 · precision analogA precision analog chain has to hold its accuracy and timing allocation for the application.Before drift is attributed to radiation, compare the trend with the temperature, the calibration state and the exposure history, and weigh it against ordinary aging using whatever local dose evidence exists.
U4 · optical interfaceAn optical interface has to deliver enough signal margin for the whole mission.Keep the optical and electrical telemetry, the drive current, the temperature and the calibration. Radiation-induced degradation is weighed against contamination, aging and changes at the interface.

Exercise

Before the flight interface is frozen, trace each important failure path to the sensor that would see it, the event record it would leave and the timing resolution it needs.

Build a versioned timeline that keeps the original timestamps, the converted times and their uncertainty, and that separates the observed facts from the inferred links.

Going deeper

The telemetry is a measurement with its own limits

A detector has a sensitivity, a dead time, a saturation level and a rate of misclassified events, and recovery can erase the state that would separate the causes. Any inference from flight data therefore has to account for how the event was captured and how the telemetry was transported, as well as for the physical event itself. The telemetry design should include, for each channel, a statement of what it can and cannot exclude.

A missing record may mean no event, lost telemetry, disabled monitoring or overwritten state.

Cadence can change the apparent order of events

A sampled current rise may appear before the logged reset and still have happened after the true internal fault. Interpolating through a telemetry gap gives a precision the data does not have. Uncertain event times are treated as intervals, and those intervals are carried into the exposure reconstruction and the comparison of hypotheses. When several clock domains are involved, the alignment between them is measured; a shift chosen by eye to fit the preferred story is not an alignment.

The original time tags and the conversions applied to them are kept alongside the reconciled timeline.

The common mistake

Telemetry cannot bring back physical state that was lost, and it generally cannot identify the initiating particle on its own.

Two things happening close together in time does not show that one caused the other, least of all when the telemetry is coarse or the clocks are poorly aligned.

Takeaway

  • What was recorded decides which failure hypotheses, and which assumptions of the assurance argument, flight operations can test at all.
  • Getting the time order right separates the initiating fault from the responses that followed it and from environmental coincidences.

Check yourself

Why is a shared-bus current at one-minute cadence insufficient to investigate a U1 event?

A high-current event on U1 lasts tens of microseconds to milliseconds before the limiter acts. At one-minute cadence the step is averaged into the bus current and cannot be told from a load change, and the trip counter on its own cannot order the sequence.

A reset log entry carries a ground-receipt timestamp only. What can be inferred about when the event occurred?

Only that it occurred some time before the downlink. The position at the event time cannot be recovered from that record, so neither can any association with the anomaly region.

What is the role of the scrub-cycle counter in the anomaly record?

It watches an assumption of the argument, that the scrubber runs at its interval. If the scrubber stopped, the accumulation term of lesson 10.1 no longer holds, and an uncorrectable word then has a non-radiation explanation that should be weighed first.

The recorder controller logged its "supply" reset 2.1 s after the U1 current step, and a colleague takes the reset as the first fault. What does the reconstructed timeline say?

The first alarm is rarely the first fault. An event propagates from a current step on a rail to a brown-out on a load and a controller reset, each logged by a different subsystem with its own clock, and the timeline is built from the earliest evidence that can be supported: here the 10 ms current sample places the occurrence at T ± 10 ms, the trace buffer released by the trip shows the step and the trip 12 µs apart, and the event log has only 1 s resolution. The physical occurrence, the detection, the onboard logging, the downlink and the ground receipt at T + 6 h are kept as separate times, each with its resolution as an uncertainty interval.

Why does the course insist on recording normal behavior as carefully as anomalies?

Because the baseline is what an anomaly is compared against. An error count that doubles means something only when it can be set beside the count from the previous month at the same orbit phase, and a U3 offset that moves can be separated into a thermal trend (which tracks temperature and reverses with it) and a dose trend (which generally does not) only if the offset at the calibration points and the board temperature were recorded every orbit all along. The same holds for U4, whose drive current and received amplitude per orbit are what separate contamination or pointing from CTR loss.

The telemetry shows no uncorrectable-error flag during a pass through the anomaly region, and the report counts the pass as free of events. What are the other readings of a missing record?

A missing record can mean no event, lost telemetry, disabled monitoring or overwritten state. The telemetry is a measurement with a detection efficiency of its own: a counter saturates, a reset erases a log, the telemetry drops out for a pass, a monitor is disabled during a maneuver. The statement of what each channel can and cannot see, written when the telemetry was designed, says which of those is possible here, and only then does the absence of a flag count as evidence of no event; negative evidence counts only when the missing signature should have been detectable (lesson 12.2).

Further reading

  • NASA, "Avionics Radiation Hardness Assurance (RHA) Guidelines," NASA/TM-20210018053, 2021. NTRS 20210018053.
  • R. Harboe-Sørensen et al., "Observation and analysis of single event effects on-board the SOHO satellite," IEEE Trans. Nucl. Sci., vol. 49, no. 3, pp. 1345-1350, 2002.
  • C. Poivey et al., "In-flight observations of long-term single-event effect (SEE) performance on Orbview-2 solid state recorders (SSR)," IEEE Trans. Nucl. Sci., vol. 50, no. 6, pp. 2169-2174, 2003.