13
$\begingroup$

Say I run a supercomputer computation on 100k cores for 4 hours on http://www.nersc.gov/users/computational-systems/edison/configuration, exchanging about 4 PB of data over the network and performing about 4 TB of I/O. The calculation is all integer, so the results are either right or wrong (no intermediate numerical errors).

Assuming the code is correct, I would like to estimate the probability that the computation is wrong due to a hardware failure. What is a good way to go about this? Are there good sources for the numbers required to make such an estimate?

Improve this question
$\endgroup$
1
  • $\begingroup$ I imagine the CPU/ram results are really stable compared to the network hooey and disk considerations. $\endgroup$
    – meawoppl
    Commented Mar 4, 2014 at 2:02

4 Answers 4

Reset to default
5
$\begingroup$

Have you looked at the various exascale reports that have come out? Hardward failures are not a significant concern today -- sure, they happen, but their frequency is not sufficiently high to cause grave worry. But they are estimated to be sufficiently frequent on exascale systems with $O(10^8)$ or more cores that codes need to be prepared to react appropriately. I believe that these issues have been laid out in the reports on roadmaps towards exascale.

My recollection is that among the various failure modes, single bit flips in memory or on processor cores were not the most significant concerns. Rather, it was entire nodes going down, e.g. due to disk failure, operating system faults, etc. The current exascale designs therefore all call for periodic checkpointing of codes into flash RAM, preferably transmitting the checkpoint data off-node. Codes will then need to be able to restart on the fly from a previously saved state if the system encounters that one node has disappeared, replacing this node with a hot-start node elsewhere in the system.

Improve this answer
$\endgroup$
6
  • $\begingroup$ That sounds like exactly what I need. Do you have particular examples in mind? $\endgroup$
    – Geoffrey Irving
    Commented Mar 5, 2014 at 0:42
  • 1
    $\begingroup$ I would see if there's anything among the various DoE reports that's of interest to you. I assume you also know about exascale.org ? There should be plenty to read there for you. $\endgroup$
    – Wolfgang Bangerth
    Commented Mar 5, 2014 at 3:56
  • 1
    $\begingroup$ Geoff, the definitive exascale report is by Peter Kogge, and is available online. Have a look at any occurrences of the word resiliency. That said, I can point you to a few people at NERSC who might have more specific information about that machine. $\endgroup$
    – Aron Ahmadia
    Commented Mar 5, 2014 at 5:27
  • $\begingroup$ @AronAhmadia: Thanks, that document looks great. I'm accepting this answer since it should cover more of the classes of errors I'm interested in. $\endgroup$
    – Geoffrey Irving
    Commented Mar 5, 2014 at 17:37
  • $\begingroup$ @Wolfgang: This reminds me of my cold-war days when Minuteman missiles were programmed with checkpoints, so that if a neutron flash occurred nearby, causing instantaneous shutdown of the processor, it could restart from the most recent checkpoint. If it took checkpoints at the provably right times, it was called "restart-protected". $\endgroup$
    – Mike Dunlavey
    Commented Mar 5, 2014 at 21:41
9
$\begingroup$

I guess, you start by collecting error rates of components, such as DRAM, like this Google research on DRAM Errors in the Wild: A Large-Scale Field Study They found ~1% chance to get one uncorrectable error per year.

I'm not sure if that's what you're interested. I'd be more interested in undetectable errors. Errors such that typical error checking methods would not detect. For instance, when you send packets over the optics, they're accompanied by some sort of CRC, which allows for a small chance of an error slipping through.

UPDATE: this paper Architectures for Online Error Detection and Recovery in Multicore Processors talks about reliable multicore architecture, but they also cover different aspects of reliability of the system and has bibliography

Improve this answer
$\endgroup$
1
  • $\begingroup$ Great study. It confirms a lot of intuition, old, hot, frequently used, nearly full ram is less reliable. I am somewhat surprised there is no vendor particular failings or generally worse architectures. $\endgroup$
    – meawoppl
    Commented Mar 3, 2014 at 6:09
3
$\begingroup$

Are there good sources for the numbers required to make such an estimate?

You might try asking the admins of the cluster you're computing on. I imagine as part of their validation process they have confronted the problem of estimating the likelihood of hardware errors.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks! Obvious in hindsight, but it had not occurred to me. $\endgroup$
    – Geoffrey Irving
    Commented Mar 3, 2014 at 17:45
2
$\begingroup$

Sounds epic. If no one has done this experiment, you might consider running 100k separate cores doing something like rehashing a sha1 input over and over, seeing what the error rate is. (Unmeasureable I suspect), from there do the same, but have them trade hash-chain results every so often to get your network error rates. This I imagine is also very small, but I suspect you can get at least a couple using your supercluster over a few hours :)

This approach insures that every calculation is correct, as hashing is extremely sensitive to single-bit swaps, whereas even an integer only calculation might hide errors in branches, ie the entire calculation would not be elliptic on each consecutive memory state.

I have been working on a way to ensure that code has been run correctly by a external cluster who's motivation is to cheat by submitting faked results. The solution I converged on is integrating the hash into the computation with some frequency that makes cheating less efficient than doing the work.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Unfortunately, it's unlikely that your scheme for mining bitcoins would be approved. :) $\endgroup$
    – Geoffrey Irving
    Commented Mar 3, 2014 at 6:25
  • $\begingroup$ Tee hee hee. Its just proof of work really. :P $\endgroup$
    – meawoppl
    Commented Mar 3, 2014 at 6:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.