4
$\begingroup$

I am currently testing some new (embedded) processors for their double-precision numeric performance.

The FPU capabilities vary wildly between platforms; For example, the ARM Cortex-A8 has a double precision FPU, takes 9 cycles per operation. The Cortex-A9 also takes 9 cycles, but is able to be fully pipelined, whereas the Cortex-A15 apparently (finally!) has a single cycle double.

Up until now, I have just used linear algebra (e.g. Eigen or ATLAS) for benchmarking, which represents a good subset of our own computational demands, but is deficient as a benchmark for well-documented reasons.

I was looking at something like the HPC Challenge, or one of the many others (see slide 15), but it's a bit like giving a starving man a menu!

So, what numerical benchmarks, if any, meet the following criteria?

  • Which other benchmarks (even if flawed/limited) are generally accepted? A good example for "general acceptance" would be the ability to find results for common (commodity) processors without much time investment using google.
  • As I am generally interested in embedded systems, I much prefer applications that can cleanly be cross-compiled - Self-tuning packages are a pain on embedded systems (e.g. ATLAS is a good example)
  • Even better, packages that only minimally depend on operating system services (or better yet, just libc or similar) allows for testing on a much wider range of systems.

Update: Some context

Linear algebra does indeed cover a large subset of our computational demands (which was listed in the OP). It does, however, benefit greatly where the operations can be pipelined. This isn't universally true for all our computational demands - there are multiple smaller solvers and algorithms of various types scattered throughout our code base. I doubt I am aware of all of them. Plus, since we typically maintain an embedded system for 5 years (or more!) after release, our current suite of applications may not represent future demands, the nature of which I cannot entirely predict (Note 1).

The second reasons is than I cannot purchase development kits, port code, build a cross-compiler, run benchmarks, etc for the literally hundreds of different SoCs that exist on the market. If there are few, well-known benchmarks that become ubiquitous (Coremark is an example, but it is not numeric-specific), then it makes the job of processor selection that much easier.

Note 1: A good example is half-way through the life of a product, a floating point application that had significant amounts of branching was developed (by an ex-HPC guy, so I hope he know what he was doing!). Had we selected a processor that was very heavily pipelined, we may have been caught out.

$\endgroup$
  • $\begingroup$ You haven't told us what kind of computations you plan to do with these processors. Since they're embedded, I'd expect that they'll be used for a very limited set of applications. If your plan is to solve lots of systems of equations, then linear algebra benchmarks really are the right thing to look at. On the other hand, if your interest is in doing signal processing with double precision FFT's then you should be benchmarking FFT code. $\endgroup$ – Brian Borchers Oct 21 '13 at 3:27
  • $\begingroup$ I will edit the post with some comments to this. $\endgroup$ – Damien Oct 21 '13 at 6:50
  • $\begingroup$ I'd be interested to see the published times for these benchmarks in the on such embedded processors (@BrianBorchers: your cell phone likely has a very similar processor in it - along with other stuff of course, the processors mentioned are not low end). $\endgroup$ – horchler Oct 21 '13 at 13:48
  • 1
    $\begingroup$ BTW, here are some benchmarks for ATLAS on the Cortex-A8 and Cortex-A9. $\endgroup$ – Damien Oct 21 '13 at 22:44
1
$\begingroup$

I would suggest to look at the SPEC CPU2006 benchmarks, specifically the Floating Point benchmarks. They are many and complex, so you may not necessarily want to run them yourself (the also cost money) but you may be able to find results at http://www.spec.org

$\endgroup$
  • $\begingroup$ I doubt that these embedded processors have been built into systems that anyone would bother running SPEC benchmarks on. $\endgroup$ – Brian Borchers Oct 21 '13 at 3:24
  • 2
    $\begingroup$ I found this recent paper (PDF). $\endgroup$ – horchler Oct 21 '13 at 13:50
  • $\begingroup$ @horchler, nice paper! Figure 7 is instructive. Certain workloads for the Cortex-A8 are described as "outliers" by the paper, which are consistent with my experience. This is the sort of thing I would like to crystallise via benchmarks - at least with respect to FP, anyway. $\endgroup$ – Damien Oct 21 '13 at 22:37
  • $\begingroup$ ... there is one dubious thing in the paper with respect to the ARM Cortex-A8/A9-- "However, unlike the VFP unit, the NEON unit is not IEEE754 compliant, and double precision operations are mapped to library calls." -- The VFP does have DP hardware and hence library calls are not necessary, but it takes time to switch between NEON mode and VFP mode and hence programs tend to be compiled for one and not the other. Since NEON is not IEEE compliant, I tend to prefer VFP (and I rarely use single-precision float anyway). $\endgroup$ – Damien Oct 21 '13 at 22:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.