# Deep learning using Distributed linear algebra

Is there any deep learning library based on Trillinos or Petsc linear algebra?

• Do you consider support vector machine as a form of at least machine learning? If yes, you can check this repo for multi-level support vector machine, which is developed based on PETSc: github.com/esadr/mlsvm Commented Oct 17, 2019 at 13:41
• thanks! However, I am more interested in modern deep networks, such as ResNet ... Commented Oct 17, 2019 at 18:01
• Perhaps relevant: github.com/LLNL/lbann
– user20857
Commented Mar 9, 2022 at 15:59

When you talk about distributed linear algebra, I take this to mean between nodes.

This probably wouldn't be a good design choice.

Why would you distribute the problem?

1. Maybe it will make training faster!
2. Maybe I can learn on bigger datasets!

The K80's internal bandwidth is about 480GB/s (gigabytes per second). The Haswell architecture is about 102GB/s. The first-generation TPUs had a bandwidth of about 34GB/s.

In comparison, Infiniband, which is the interconnect used in many supercomputers, has a bandwidth of just 25GB/s.

If we look at a roofline chart of deep learning models, this provides some useful information:

The vertical axis is performance while the horizontal axis is memory reuse (operations per byte once it is loaded from memory).

The diagonal lines are regions where the kernels are limited by the speed of memory. The horizontal lines indicate limitations from the speed of floating-point operations. Using Infiniband at the speed I state means drawing a new diagonal line:

There are roughly two groups of kernels: those that are memory-limited (on the left-hand side) and those that are computation-limited (on the right-hand side). For the kernels on the RHS, the only way to make them faster is to have faster floating-point operations. That's why we built the TPU: it was physically impossible to accelerate the kernels in any other way.

The memory-bound kernels can be accelerated in two ways:

1. By increasing arithmetic intensity (memory reuse), thereby sliding them up and to the right so that they're blocked by one of the horizontal rooflines.
2. By increasing the speed of the memory they're bound by. This is often achieved by improving cache utilization or the locality of an operation.

As you can see, Infiniband does neither of these things. Therefore, this kind of distributed operation is unlikely to be useful for accelerating deep learning.

What about the second possibility: training larger models? Larger models require greater training time and training is expensive. Couple this with having to train slower and it's a no-go.

What about limiting yourself to a single node? Well, in that case there are simpler libraries than PETSc (e.g. MKL) that perform this same operation, so there's no need to pull in the PETSc dependency.

To address one of the comments, since 1990 FLOPS per socket have increased at about 50-60% per year while interconnect bandwidth has increased at only 20% per year and interconnect latency has only decreased at about 20% per year. See this chart, for instance:

This means that if you want peak performance you need to avoid interconnects and this will become more true over time. Research in communication avoiding algorithms seeks to find ways of achieving good performance in the face of this reality. In deep learning, distributed training methods have a similar motivation.

Aside from the algorithmic costs of communication, it's expensive in terms of energy:

Collectively, these trends are also changing the way we design computers. Summit, for instance, clusters a lot of compute power per node to reduce costs.

• Considering that there is a paper from FB researchers where they basically reinvent the implementation of popular MPI collectives, there seems to be some interest in distributed learning. Also, since learning seems to be largely matrix-matrix, any MPI traffic can be very nicely hidden. Doing distributed matrix-matrix (at a large fraction o the on-node performance has been settled decades ago. Commented Oct 17, 2019 at 19:07
• If you disagree, feel free to aggregate the appropriate information and post an answer of your own. I'll be happy to learn. The blue line in that chart's pretty convincing to me. Commented Oct 17, 2019 at 20:23
• @Richard: I think you ought to take what Victor Eijkhoud says with serious consideration -- he's not exactly a "nobody" in the field. If he declares that something is true in linear algebra, you better believe it. Commented Oct 17, 2019 at 23:36
• @Richard I'll try to find that paper. Processor/memory/network performance: sure. But that's entirely beside the point. If your granularity is high enough, you can usually hide the communication (this is known as "latency hiding") behind non-communicating computation. (I imagine your TPU uses prefetch streams to overcome memory latency. Same idea.) The reason I'm not writing an answer is that I'm insufficiently familiar with DL to know if libraries such as PETSc can help much. Commented Oct 18, 2019 at 17:45
• I'll note without comment that the Benchmark used for the top500 list is based upon distributed linear algebra - top500.org/project/linpack Commented Jul 15, 2020 at 11:13

Since you are interested in ResNet, you may want to check out this repo: https://github.com/steffi7574/LayerParallelLearning

It is based on the idea of "parallel-in-layer" and uses XBraid to distribute the layers. It is not exactly PETSc or Trillinos, but it is close.

I have looked into distributed learning, more specifically, model parallelism, which would be related to distributed linear algebra. But I was not able to find a nice ML framework to do that. For instance, implementing a scalable algorithm described in that repo would be very challenging in any ML framework.

There is another recent one called TSAdjoint developed by Hong Zhang et al: https://arxiv.org/abs/1912.07696

This package is distributed along with PETSc and it is designed to train neural ODE efficiently.