3
$\begingroup$

Is there any deep learning library based on Trillinos or Petsc linear algebra?

$\endgroup$
  • 1
    $\begingroup$ Do you consider support vector machine as a form of at least machine learning? If yes, you can check this repo for multi-level support vector machine, which is developed based on PETSc: github.com/esadr/mlsvm $\endgroup$ – Alone Programmer Oct 17 at 13:41
  • $\begingroup$ thanks! However, I am more interested in modern deep networks, such as ResNet ... $\endgroup$ – computational_scientist Oct 17 at 18:01
2
$\begingroup$

When you talk about distributed linear algebra, I take this to mean between nodes.

This probably wouldn't be a good design choice.

Why would you distribute the problem?

  1. Maybe it will make training faster!
  2. Maybe I can learn on bigger datasets!

Let's talk about (1).

The K80's internal bandwidth is about 480GB/s (gigabytes per second). The Haswell architecture is about 102GB/s. The first-generation TPUs had a bandwidth of about 34GB/s.

In comparison, Infiniband, which is the interconnect used in many supercomputers, has a bandwidth of just 25GB/s.

If we look at a roofline chart of deep learning models, this provides some useful information:

Roofline chart of deep learning kernels

The vertical axis is performance while the horizontal axis is memory reuse (operations per byte once it is loaded from memory).

The diagonal lines are regions where the kernels are limited by the speed of memory. The horizontal lines indicate limitations from the speed of floating-point operations. Using Infiniband at the speed I state means drawing a new diagonal line:

Rooflines for deep learning kernels with Infiniband

There are roughly two groups of kernels: those that are memory-limited (on the left-hand side) and those that are computation-limited (on the right-hand side). For the kernels on the RHS, the only way to make them faster is to have faster floating-point operations. That's why we built the TPU: it was physically impossible to accelerate the kernels in any other way.

The memory-bound kernels can be accelerated in two ways:

  1. By increasing arithmetic intensity (memory reuse), thereby sliding them up and to the right so that they're blocked by one of the horizontal rooflines.
  2. By increasing the speed of the memory they're bound by. This is often achieved by improving cache utilization or the locality of an operation.

As you can see, Infiniband does neither of these things. Therefore, this kind of distributed operation is unlikely to be useful for accelerating deep learning.

What about the second possibility: training larger models? Larger models require greater training time and training is expensive. Couple this with having to train slower and it's a no-go.

What about limiting yourself to a single node? Well, in that case there are simpler libraries than PETSc (e.g. MKL) that perform this same operation, so there's no need to pull in the PETSc dependency.

To address one of the comments, since 1990 FLOPS per socket have increased at about 50-60% per year while interconnect bandwidth has increased at only 20% per year and interconnect latency has only decreased at about 20% per year. See this chart, for instance:

Performance of FLOPS, memory, and network over time

This means that if you want peak performance you need to avoid interconnects and this will become more true over time. Research in communication avoiding algorithms seeks to find ways of achieving good performance in the face of this reality. In deep learning, distributed training methods have a similar motivation.

Aside from the algorithmic costs of communication, it's expensive in terms of energy:

Energy cost of data movement in 2010: On-Chip vs Off-Chip

Collectively, these trends are also changing the way we design computers. Summit, for instance, clusters a lot of compute power per node to reduce costs.

$\endgroup$
  • 1
    $\begingroup$ Considering that there is a paper from FB researchers where they basically reinvent the implementation of popular MPI collectives, there seems to be some interest in distributed learning. Also, since learning seems to be largely matrix-matrix, any MPI traffic can be very nicely hidden. Doing distributed matrix-matrix (at a large fraction o the on-node performance has been settled decades ago. $\endgroup$ – Victor Eijkhout Oct 17 at 19:07
  • $\begingroup$ If you disagree, feel free to aggregate the appropriate information and post an answer of your own. I'll be happy to learn. The blue line in that chart's pretty convincing to me. $\endgroup$ – Richard Oct 17 at 20:23
  • $\begingroup$ The performance gap between, for instance, processor and memory performance has grown about 1000x since 1980. That sort of thing has implications. Questions pertaining to hardware don't really get settled and arguments from "decades ago" don't automatically apply. Hardware changes and so too must our thinking. $\endgroup$ – Richard Oct 17 at 20:46
  • 1
    $\begingroup$ @Richard: I think you ought to take what Victor Eijkhoud says with serious consideration -- he's not exactly a "nobody" in the field. If he declares that something is true in linear algebra, you better believe it. $\endgroup$ – Wolfgang Bangerth Oct 17 at 23:36
  • 1
    $\begingroup$ @Richard I'll try to find that paper. Processor/memory/network performance: sure. But that's entirely beside the point. If your granularity is high enough, you can usually hide the communication (this is known as "latency hiding") behind non-communicating computation. (I imagine your TPU uses prefetch streams to overcome memory latency. Same idea.) The reason I'm not writing an answer is that I'm insufficiently familiar with DL to know if libraries such as PETSc can help much. $\endgroup$ – Victor Eijkhout Oct 18 at 17:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.