Memory EDAC and Scrubbing Reliability Tool

A memory with error correction does not have an upset rate, it has an uncorrectable error rate, and that number depends on things the upset rate alone cannot tell you: how many bits of a word live in one die, how far apart those bits are physically, how often the scrubber comes round, what the code can and cannot correct, and what else the part does under irradiation besides flipping bits. This tool puts those together for SRAM, DDR, FPGA configuration memory and flash, separates detected uncorrectable errors from silent corruption from outages, and shows which lever moves the answer.
SetupResultsLeversCode comparison How to read thisMethodologyReferencesHelp

Six things go wrong, and a code handles each differently. Independent single upsets accumulate in a word between scrubs, and a t error correcting code fails when the word collects t plus one before the scrubber arrives. A single ion can upset several adjacent cells at once, and whether those land in one word is decided by the physical interleave, not by the code. A word striped across several die can only lose one die's share of its bits to one event, which is what symbol codes such as chipkill exploit. A functional interrupt takes a whole die out and presents every word with a burst the width of that die. A hard error sits in a word for the rest of the mission consuming its correction capacity. And in DRAM, a strike on the periphery can corrupt a whole row.

The numbers come from the corpus and the literature. Presets carry priors from 1,535 publicly citable memory test records: per bit cross sections by technology and node, functional interrupt cross sections by technology, the mix of recovery actions they needed, and how often reports saw block errors and interleaving. Multiple cell upset fractions by node follow Ibe and co-workers' scaling series, with the FinFET literature layered on top. Every default is editable and every one is shown with its source.

Setup

Memory and word

Code and scrubbing

Effects and their rates

What the corpus says about this technology

Heavy ion, publicly citable records only. Cross sections are saturated values per bit or per device, shown as the 10th, 50th and 90th percentile of the records. These are a sanity range for your inputs, not a substitute for a measurement on your part.

Results

Headline

By mechanism, uncorrectable words per day (mission average)

DUE is a detected uncorrectable error: the read fails and the system knows. SDC is silent corruption: the code returned wrong data as if correct, either because the error pattern was beyond its detection range or because it miscorrected. Outage is time the device spends recovering from a functional interrupt.

Reading

Levers

Uncorrectable rate against scrub period

Only the accumulation term and the column error term move with the scrub period. Where the curve goes flat, scrubbing faster buys nothing and the floor is set by multiple cell upsets, die bursts and hard errors.

Uncorrectable rate against interleave distance

The interleave decides whether a cluster along the word line becomes one uncorrectable word or several correctable single errors. Past the cluster extent the curve is flat.

Hazard through the mission

Instantaneous uncorrectable rate against mission time. Scrubbed terms are flat. Unscrubbed accumulation rises as time to the power t, and hard errors rise linearly whether or not there is a scrubber. If the curve rises, most of the risk is late in the mission and a single number for the rate hides that.

Sensitivity to the upset rate you entered

Ladbury's observation: if the raw upset rate is wrong by a factor x, the accumulation term is wrong by x to the power t plus one. A factor of two on the input is a factor of four on a SEC-DED accumulation rate and eight on DEC-TED. The die burst and functional interrupt terms scale linearly.

Expected count and probability through the mission

Expected number of uncorrectable words by each point in the mission (solid) and the probability of at least one (dashed, right axis from 0 to 1). A quadratic curve is the signature of unscrubbed accumulation with a single error correcting code.

Why this is not simply a rate

An uncorrectable error is a transition of a word into a state the code cannot fix, so what the mission accumulates is a count of transitions, and the count is a non homogeneous Poisson process: expected count is the integral of a hazard that may change with time. With a periodic scrub the array is reset every period, each period is an independent trial, and the hazard is constant, so a rate is exactly right. Without a scrub, the probability that a word holds t plus one errors by time tau grows as tau to the power t plus one, so the hazard rises as tau to the t and most of the expected errors arrive late. Hard errors do the same even under a scrubber, because a scrub does not clear them. The tool integrates the hazard explicitly and reports the expected count and the probability of at least one, which are the quantities that remain meaningful either way; the mission average rate is shown for comparison with tools that only produce a rate.

Code comparison for this memory

Every code applied to the same memory, effects and scrub settings, with the check bits each one needs. The check bits are also exposed area, which is why a stronger code is not automatically a better one.

How to read this

Start with the architecture, not the code

The two inputs that dominate everything else are how many bits of a word live in one die and how far apart the bits of a word are on the silicon. A 72 bit word across nine x8 devices puts eight bits of every word in each die; anything that takes out a die, a functional interrupt or a row error, hands the code an eight bit burst, and a code that corrects one bit cannot help. Spread the same word across eighteen x4 devices and give it a symbol code whose symbol is four bits wide, and the same events become single symbol errors, corrected on the fly. That is the whole of chipkill, and it is why the comparison table can differ by six orders of magnitude between two codes with similar overhead. Inside a die, the interleave does the same job for clusters: a cluster that spans fewer cells than the interleave distance puts one bit into each of several words, and the code corrects them one by one.

Then the scrub period

The accumulation term is the one people model and it is usually not the one that matters. It scales as the scrub period, and as the upset rate to the power t plus one, so on the scrub plot it falls steadily as the period shortens until it drops below the terms that do not care about scrubbing. Look for where the curve goes flat. If it is flat at your operating point, the scrubber is not your problem. If it is not, note that random access scrubbing at the same mean interval is about twice as bad as a periodic sweep, and that correction on the fly without writing the corrected word back is not scrubbing at all: the error stays in the array and keeps counting.

Then what the technology actually does

SRAM upsets by charge sharing between adjacent cells, so multiple cell upsets are the exposure and the interleave is the defence. DRAM arrays barely do that, because adjacent cells already belong to different words, but a strike on the periphery corrupts a whole row and a functional interrupt corrupts a whole die; in the corpus, 85 percent of DRAM records that mention block behaviour report it, and about two fifths of DRAM functional interrupts needed a power cycle. Flash cells hardly upset at all and the on die code corrects tens of bits per codeword, so the exposure is the controller interrupt, more than half of which needed a power cycle in the corpus, and the slow accumulation of hard errors. FPGA configuration memory is SRAM with a twist: an uncorrectable frame is a change of logic, not a wrong data word, and stays wrong until reconfiguration.

Silent corruption is a separate number

A code that detects double errors and is handed a triple error does not always say so. For odd weight column SEC-DED codes the published miscorrection fraction for triple errors is in the range of one half to three quarters, which means most triple errors come back as valid looking wrong data. The tool carries that fraction as an input and reports silent corruption separately from detected failure, because the system consequences are different: a detected error can be handled, a silent one propagates.

What the numbers do not include

Part to part and lot to lot variation. The environment model. The energy dependence that LET does not capture. Any correlation between events, which field studies of terrestrial DRAM find to be the dominant failure mode on the ground; on orbit, cosmic ray upsets are much closer to the independent Poisson picture the model assumes. And design bugs in the error handling path, which in flight anomaly records are more common than any of the above.

Methodology

Accumulation

Each word of n bits receives independent upsets at rate n times the per bit rate. With a periodic scrub of period T, the errors present in a word at the end of a period are Poisson with mean n times lambda times T, and the word is uncorrectable if that count exceeds t. The rate is the number of words times that tail probability divided by T. At small rates this reduces to Saleh, Serrano and Patel's closed form, mean time to failure equal to 2 over (Q times T times (n lambda) squared) for a single error correcting code, and the engine reproduces that closed form to five figures across scrub periods from an hour to a year. Random access scrubbing is applied as a factor of two on this term, from the same source. Correct on read with write back sets the effective period to the shorter of the scrub period and the access interval. No scrubbing sets it to the mission.

Time dependence

The uncorrectable process is treated as a non homogeneous Poisson process and integrated over the mission. Terms that are reset by the scrubber, and the single event terms (multiple cell upsets, interrupts, row and column errors), have a constant hazard. Accumulation without a scrub has a hazard equal to the number of words times the per word upset rate times the probability that a word holds exactly t errors at that time, which rises as time to the power t; its integral is the number of words times the probability of t plus one or more errors by the end of the mission, which is what the mission average rate reports. Hard errors give a hazard that rises linearly. The tool reports the expected count and the probability of at least one, the hazard at the start and the end of the mission, and the share of expected events in the final quarter, so that a rising hazard is visible rather than averaged away.

Multiple cell upsets

The per bit upset rate is converted to an event rate using the fraction of events that are multiple cell and their mean multiplicity. Each multiple cell event is given an extent along the word line drawn from a geometric distribution with the stated mean. With interleave distance D, a run of w cells gives each of D words either the floor or the ceiling of w over D cells, and (w mod D) words get the larger count; each word is judged against the code on the cells it received, capped by the bits of the word that sit in one die. The tool reports both events and words, because one event can produce more than one uncorrectable word, and a system sees words. A Monte Carlo of an interleaved memory with periodic scrubbing agrees with the closed form within counting statistics at rates high enough to test it, and the neglected cross term between clusters and previously accumulated errors vanishes at space rates. Extent along the bit line puts single errors into other words and is carried by the per bit rate in the accumulation term.

Die bursts, hard errors, interrupts

A functional interrupt or a row error in one die presents every affected word with a burst equal to the bits of the word in that die, judged against the code in bits or in symbols. A symbol code whose symbol is at least the device width corrects it; a bit code with t below the device width does not. Interrupts also carry a recovery time, which becomes an outage fraction. Hard errors accumulate linearly at the stated rate; a word holding one has its correction capacity reduced by one, so for a single error correcting code its next transient error is fatal, and the mission averaged contribution uses half the end of mission count. Retirement removes the term. Column errors put one bit in each of many words and are correctable individually; their contribution is the chance of a second error arriving before the scrubber does.

Codes

Each code is described by the errors it corrects, t, the errors it detects, d, and whether it acts on bits or on symbols. Error counts above d are split between detected failure and silent corruption by a miscorrection fraction: 0.6 for SEC-DED from the published range for odd weight column codes, 0.07 for single symbol correcting Reed Solomon from a Monte Carlo of the SSC baseline (lower for SSC-DSD), a placeholder of 0.5 for DEC-TED where no good number was found, and 1.0 for parity and TMR, where a two error pattern is by construction silent. Word level TMR is modelled as a three bit word per data bit with t = 1, d = 1.

Priors

The corpus figures are drawn from 1,535 publicly citable memory test records, adjudicated individually against a written specification to separate per bit from per device cross sections, functional interrupts from upsets, and hard errors from transients. Per bit cross sections use the structured saturated values by technology and node; functional interrupt cross sections, recovery actions, block error reports and interleave statements come from the adjudicated text. Multiple cell fractions by node are too sparsely reported in test reports to stand alone, so the node defaults follow Ibe and co-workers' scaling series for planar bulk SRAM, with the FinFET results of Pieper and co-workers informing the word line extent, and the corpus values shown for comparison.

References

Saleh, A. M., Serrano, J. J. and Patel, J. H., 1990. Reliability of Scrubbing Recovery Techniques for Memory Systems. IEEE Transactions on Reliability 39(1), 114. The accumulation model, periodic and random scrubbing.
Mukherjee, S. S., Emer, J. and Reinhardt, S. K., 2005. The Soft Error Problem: An Architectural Perspective. IEEE HPCA. Architectural vulnerability and the detected versus silent distinction.
Ladbury, R., 2010. SDRAM Testing: Lessons Learned. NASA NEPP SEE Symposium. Flight versus predicted rates for a 512 Mbit SDRAM and the x to the power n plus one sensitivity rule.
Ladbury, R., Shoga, M. and Koga, R., 2003. SDRAMs: Can't Live Without Them, But Can We Live With Them? NASA NEPP SEE Symposium. The DRAM effect taxonomy.
Ibe, E., Taniguchi, H., Yahagi, Y., Shimbo, K. and Toba, T., 2010. Impact of Scaling on Neutron-Induced Soft Error in SRAMs From a 250 nm to a 22 nm Design Rule. IEEE Transactions on Electron Devices 57(6), 1527. Multiple cell fraction and multiplicity versus node.
Pieper, N. J., Xiong, Y., Feeley, A., Pasternak, J., Dodds, N., Ball, D. R. and Bhuva, B. L., 2022. SRAM Multi-Cell Upset Vulnerability at the 5-nm FinFET Node. IEEE Transactions on Nuclear Science. Cluster shape and word line versus bit line extent.
Quinn, H., Graham, P., Krone, J., Caffrey, M. and Rezgui, S., 2005. Radiation-Induced Multi-Bit Upsets in SRAM-Based FPGAs. IEEE Transactions on Nuclear Science 52(6), 2455. Configuration memory multiple bit fractions by device generation.
Ostler, P. S., Caffrey, M. P., Gibelyou, D. S., Graham, P. S., Morgan, K. S., Pratt, B. H., Quinn, H. M. and Wirthlin, M. J., 2009. SRAM FPGA Reliability Analysis for Harsh Radiation Environments. IEEE Transactions on Nuclear Science 56(6), 3519. Scrubbed TMR failure model and composite mission MTTF.
Hussein, J. and Swift, G., 2015. Mitigating Single-Event Upsets. Xilinx White Paper WP395. Interleaving, frame ECC and CRC in configuration memory.
Yeleswarapu, R. and Somani, A. K., 2019. Addressing Multiple Bit/Symbol Errors in DRAM Subsystem. arXiv 1908.01806. Reed Solomon miscorrection fractions.
Richter, R., Oberlaender, K. and Goessel, M., 2008. New Linear SEC-DED Codes with Reduced Triple Bit Error Miscorrection Probability. IEEE IOLTS. Triple error miscorrection in odd weight column codes.
Dell Technologies, 2020. Memory Errors and PowerEdge Server Memory RAS Features. The x4 versus x8 symbol mapping, demand and patrol scrub.
Irom, F. and Allen, G. R., 2012. Radiation Tests of Highly Scaled, High-Density, Commercial, Nonvolatile NAND Flash Memories, Update 2012. JPL Publication 12-19. Flash cell and functional interrupt cross sections.
Hwang, A. A., Stefanovici, I. A. and Schroeder, B., 2012. Cosmic Rays Don't Strike Twice. ASPLOS. Field evidence that terrestrial DRAM errors are dominated by correlated hard faults, and the case for page retirement.

Related tools: SEE Rate Assessment, Alpha Law Directional SEU Tool, SEL Threshold Predictor, Standards and Data Sources.