# Estimating hardware error probability

Say I run a supercomputer computation on 100k cores for 4 hours on http://www.nersc.gov/users/computational-systems/edison/configuration, exchanging about 4 PB of data over the network and performing about 4 TB of I/O. The calculation is all integer, so the results are either right or wrong (no intermediate numerical errors).

Assuming the code is correct, I would like to estimate the probability that the computation is wrong due to a hardware failure. What is a good way to go about this? Are there good sources for the numbers required to make such an estimate?

-
I imagine the CPU/ram results are really stable compared to the network hooey and disk considerations. –  meawoppl Mar 4 at 2:02

## 4 Answers

Have you looked at the various exascale reports that have come out? Hardward failures are not a significant concern today -- sure, they happen, but their frequency is not sufficiently high to cause grave worry. But they are estimated to be sufficiently frequent on exascale systems with $O(10^8)$ or more cores that codes need to be prepared to react appropriately. I believe that these issues have been laid out in the reports on roadmaps towards exascale.

My recollection is that among the various failure modes, single bit flips in memory or on processor cores were not the most significant concerns. Rather, it was entire nodes going down, e.g. due to disk failure, operating system faults, etc. The current exascale designs therefore all call for periodic checkpointing of codes into flash RAM, preferably transmitting the checkpoint data off-node. Codes will then need to be able to restart on the fly from a previously saved state if the system encounters that one node has disappeared, replacing this node with a hot-start node elsewhere in the system.

-
That sounds like exactly what I need. Do you have particular examples in mind? –  Geoffrey Irving Mar 5 at 0:42
I would see if there's anything among the various DoE reports that's of interest to you. I assume you also know about exascale.org ? There should be plenty to read there for you. –  Wolfgang Bangerth Mar 5 at 3:56
Geoff, the definitive exascale report is by Peter Kogge, and is available online. Have a look at any occurrences of the word resiliency. That said, I can point you to a few people at NERSC who might have more specific information about that machine. –  Aron Ahmadia Mar 5 at 5:27
@AronAhmadia: Thanks, that document looks great. I'm accepting this answer since it should cover more of the classes of errors I'm interested in. –  Geoffrey Irving Mar 5 at 17:37
@Wolfgang: This reminds me of my cold-war days when Minuteman missiles were programmed with checkpoints, so that if a neutron flash occurred nearby, causing instantaneous shutdown of the processor, it could restart from the most recent checkpoint. If it took checkpoints at the provably right times, it was called "restart-protected". –  Mike Dunlavey Mar 5 at 21:41

I guess, you start by collecting error rates of components, such as DRAM, like this Google research on DRAM Errors in the Wild: A Large-Scale Field Study They found ~1% chance to get one uncorrectable error per year.

I'm not sure if that's what you're interested. I'd be more interested in undetectable errors. Errors such that typical error checking methods would not detect. For instance, when you send packets over the optics, they're accompanied by some sort of CRC, which allows for a small chance of an error slipping through.

UPDATE: this paper Architectures for Online Error Detection and Recovery in Multicore Processors talks about reliable multicore architecture, but they also cover different aspects of reliability of the system and has bibliography

-
Great study. It confirms a lot of intuition, old, hot, frequently used, nearly full ram is less reliable. I am somewhat surprised there is no vendor particular failings or generally worse architectures. –  meawoppl Mar 3 at 6:09

Are there good sources for the numbers required to make such an estimate?

You might try asking the admins of the cluster you're computing on. I imagine as part of their validation process they have confronted the problem of estimating the likelihood of hardware errors.

-
Thanks! Obvious in hindsight, but it had not occurred to me. –  Geoffrey Irving Mar 3 at 17:45

Sounds epic. If no one has done this experiment, you might consider running 100k separate cores doing something like rehashing a sha1 input over and over, seeing what the error rate is. (Unmeasureable I suspect), from there do the same, but have them trade hash-chain results every so often to get your network error rates. This I imagine is also very small, but I suspect you can get at least a couple using your supercluster over a few hours :)

This approach insures that every calculation is correct, as hashing is extremely sensitive to single-bit swaps, whereas even an integer only calculation might hide errors in branches, ie the entire calculation would not be elliptic on each consecutive memory state.

I have been working on a way to ensure that code has been run correctly by a external cluster who's motivation is to cheat by submitting faked results. The solution I converged on is integrating the hash into the computation with some frequency that makes cheating less efficient than doing the work.

-
Unfortunately, it's unlikely that your scheme for mining bitcoins would be approved. :) –  Geoffrey Irving Mar 3 at 6:25
Tee hee hee. Its just proof of work really. :P –  meawoppl Mar 3 at 6:28