Going to try to move some of my scipy/numpy calculation to a new GPU, how to avoid disappointing results?

Question

update: I've refactored the question based on helpful advice in the linked meta.

I'm a heavy user of Python's NumPy and SciPy (and not much else) and for years I could run anything I need on my laptop.

Now I will start to write simulations with 100's to 10,000's of elements that go way beyond what a single CPU can do in a reasonable amount of time.

One potentially economical way to add cores is to use a PC and add a CUDA GPU, and CuPy is one way to move the heavy calculations to it:

CuPy's interface is highly compatible with NumPy and SciPy; in most cases it can be used as a drop-in replacement. All you need to do is just replace numpy and scipy with cupy and cupyx.scipy in your Python code. The Basics of CuPy tutorial is useful to learn first steps with CuPy. CuPy supports various methods, indexing, data types, broadcasting and more. This comparison table shows a list of NumPy / SciPy APIs and their corresponding CuPy implementations.

I will buy a modest CUDA GPU and start experimenting with how to do this.

I'm not a developer, and I see (at least) two potential problems that could cause me to flounder and ultimately fall flat on my face and fail:

1. I buy the wrong CUDA GPU and the speed up is minimal or nonexistent.
2. I configure my arrays wrong (e.g. fast vs slow axis) or rely heavily on indexed arrays, and so much time is spent moving data between CPU, memory and GPU that the speedup is minimal or nonexistent.

Question: I'm going to try to move some of my scipy/numpy calculation to a new GPU and would like some specific suggestions to help me avoid disappointing results due to the factors above.

While this is similar to a "best practices" question, I don't have any experience yet, and so it's easier for me to ask and support a question about what not to do wrong than one about how to do everything right.

There will be a separate but related question in the near future where I will include a specific example and some alternative implementations in NumPy, and hopefully at that time I will have benefitted from some help here and have a modest GPU up and running.

---

Background and related posts:

• https://pypi.org/project/pycu/
• https://pypi.org/project/cupy/

From this answer to CUDA & Python for numerical integration and solving differential equations:

• CuPy; and
• scikit-cuda.

Looking around, I found CudaPyInt and it uses PyCuda

From this answer to Writing code on the CPU while developing, running it on the GPU when live - which approach?:

ArrayFire has a C++ API as well as a Python API. You can switch between several backends including CPU, CUDA, and OpenCL. It will also handle memory movement and kernel fusion for you.

also

• GPU vs CPU calculation

Answer 1

I buy the wrong CUDA GPU and the speed up is minimal or nonexistent.

It is highly unlikely that your choice of GPU will have a significant impact on your speed-up unless your model is very big.

To a first approximation, all GPUs are created equal except for two points of variation:

1. The number of streaming multiprocessors.
2. The amount of RAM they have.

RAM is the easiest to understand. If all your data can fit in a GPU's RAM then you don't have to move it back and forth between your computer. Since moving data is slow, avoiding movement gives you speed gains.

However, the situation is not straight-forward. A GPU can crunch numbers while moving data, so if you're doing enough math to saturate the GPU the cost of data movement is effectively zero.

More streaming multiprocessors means you can crunch more numbers at a time. However, this too is not straight-forward. The GPU can interleave calculations while it waits for, eg, data to load from the GPU's RAM.

My Thinkpad P1 has an Nvidia Quadro P2000, which retails for \$500 at the moment. There are instances, like games, CAD, and deep learning where this is transformative versus not having the GPU. In contrast, a ~$2,000 Nvidia GeForce RTX 2080 Ti I have mostly gathers dust because it's rare for me to need that much power and, when I do, I typically have access to supercomputers and run my workloads there.

The incremental boost you get from having any GPU will have a greater impact on your work than the boost you will get from having a specific GPU.

My suggestion for you is that you find a cheap GPU you can use to experiment with and build the algorithms you need. I recommend this because having local access to a GPU and a comfortable development environment can really accelerate your development cycle versus working wholly in the cloud. Though, you could, per the below, save money by experimenting in the cloud first.

Once you have things working, if you're still hurting for performance you can buy time on Google Cloud's GPUs for ~\$0.20/hr. At the moment I think they have K80's and P100's. This might be enough for you to do the work you want or, at least, will be a cheap way to determine whether you should spend \$2,000+ on a bigger home GPU.

If your machine doesn't have a slot for a GPU internally, a Thunderbolt-enabled enclosure offers high data transfer rates and the flexibility of peripheral device.

tl;dr Don't invest thousands in a GPU until you know you need one that expensive. Either use the cloud or find a cheap GPU you can experiment with.

---

I configure my arrays wrong (e.g. fast vs slow axis) or rely heavily on indexed arrays, and so much time is spent moving data between CPU, memory and GPU that the speedup is minimal or nonexistent.

A challenging aspect of programming GPUs is maximizing your parallelism. It's difficult to answer this question, given all the ways you can do things incorrectly, so I think your best bet is to get access to a GPU (see above), experiment, and come back with more specific questions if you hit roadblocks.

That said, your goal in programming a GPU is to get high-level libraries to do as much of the work for you as possible. These libraries often have strong opinions about how your data should be laid out and use those assumptions to build fast operations. Unless you're doing relatively novel things, you should assume that what you're doing can be achieved with a library and find that library.

After you've got things running with your library, you can use tools like PyTorch Profiler or Nvidia Nsight to profile the code and identify bottlenecks to performance.