# Why am I not seeing faster neural network training after upgrading to a vastly better GPU?

I was previously running my neural networks using the Lasagne library to build and train neural networks in Theano on an NVIDIA GTX 750 Ti. I'm using a genetic algorithm to tune the hyperparameters of my neural network, so I bought a GTX 780 Ti to speed up training each neural network. The 780 Ti should have over 3 times the single precision floating point performance of the 750 Ti, but I don't see any speed-up at all.

• I checked GPU utilization in GPU-Z and it is the same for both GPUs (mid-high 60's%). Why is the beefier GPU using the same percentage of resources, but not running any faster?
• I uninstalled all NVIDIA software including the driver (restarting twice in the process), then re-installed everything using the CUDA 7.5 Toolkit Installer and then I installed the CUDNN 7.5 (5?) libraries/headers/binaries I restarted my neural network and it was still performing as poorly as before.

Some probably important information:

• This is on Windows 7
• For nvcc I'm using cl-version=2013 because that's what I have and it works fine
• CPU is an Intel Core i5-2500K @ 3.3GHz
• 8 GB system memory
• Data is loaded off of a Samsung SSD 850 EVO 500GB drive
• 3 GB of GPU memory (other specs can be found at the link above)
• Card is plugged into a PCI-E Gen2 (1x16) slot
• The networks vary a lot, but they're all multilayer perceptrons with between 1-5 hidden layers
• 15-150 neurons per layer, all DenseLayers
• 75 inputs, 10 outputs
• Currently no drop-out or Gaussian noise before any layers
• Batch size is 1024
• Data is shuffled before each epoch
• Hidden layer activation functions are all elu
• Output activation function can be sigmoid, tanh, or elu
• Loss function is mean squared error
• Training data size is 274MB with 934918 samples
• Validation data size is 30.5MB with 103880 samples

My first thought was that my networks were too small to make any difference on a small vs. big GPU, but the bigger GPU is stressed just as much as the littler one when running, plus these networks were running faster on the GRID K520 GPUs on Amazon EC2 (on Ubuntu) (both the K520 and 780 Ti use the Kepler GPU architecture while the 750 Ti was Nvidia's first Maxwell card).

What could be causing my new card to be so slow? How can I get it to run faster than the older crappier card?

Edit: Running the nn on Ubuntu I see a significant speed-up, not as big as I hoped, but definitely noticeable. I'm probably not making full use of the GPU with my small networks. One possible reason for the slowness in Windows is WDDM mode vs. TCC mode: https:// devtalk.nvidia.com /default/topic/895331/cuda-setup-and-installation/tesla-k20-vs-titan-x-performance-for-the-same-code/post/4724884/#4724884. I'm going to try installing my old card in a PCI-E 1x slot and make that the display driver and use my new card for computation only

Edit the second: Here's what one of my neural networks would look like:

input_var = T.matrix('inputs')
input = lasagne.layers.InputLayer(shape=(1024, 75), input_var=input_var)
network = lasagne.layers.DenseLayer(input, 40,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
network = lasagne.layers.DenseLayer(network, 113,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
network = lasagne.layers.DenseLayer(network, 76,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
output = lasagne.layers.DenseLayer(
network, num_units=10,
nonlinearity=lasagne.nonlinearities.tanh)
prediction = lasagne.layers.get_output(output)
loss = lasagne.objectives.squared_error(prediction, target_var)
loss = loss.mean()

test_prediction = lasagne.layers.get_output(output, deterministic=True)
test_loss = lasagne.objectives.squared_error(test_prediction, target_var)
test_loss = test_loss.mean()

val_fn = theano.function([input_var, target_var], test_loss)

• I am not an expert with neural network, but I have some experience with GPU computing. Everything depends on algorithms (GPU/memory occupancy, data alignment, number of iterations etc.). But you have mentioned that on Amazon it was working faster, so first I would run general benchmarks based on CUDA samples and compare the results. It will show you that something is generally wrong with your PC (drivers, OS, system settings etc). Second thing is to profile parts of applications - data loading/fetching, calculations, etc separately - you will see some similarities or not between that two GPUs. – Krzysztof Bzowski May 6 '16 at 6:12
• So I ran the neural network on Ubuntu and there I see a significant speed-up. It's not the 3-3.5x speed-up I was hoping for, but you probably only get that kind of speed up for large convolutional or recurrent neural networks. @KrzysztofBzowski I've already profiled a single training session and it all looks good, most time is spent in GPU operations instead of CPU ops or loading/fetching. I'll definitely try some general benchmarks based on CUDA samples like you suggest. – Joels Elf May 6 '16 at 15:51
• Are you using that new GPU also as a video card? – Krzysztof Bzowski May 6 '16 at 16:22
• Many factors can affect performance besides just the hardware. Without knowing more about your algorithm and its implementation, it's hard to give any kind of meaningful answer or advice. – Paul May 10 '16 at 2:47
• I'm using the Lasagne library to build and train neural networks in Theano. Since I'm tuning the parameters of the nn with a genetic algorithm the network is built dynamically, but I'll put a static example up in the main question. – Joels Elf May 12 '16 at 13:59

I suppose, you right and your network is not that big to 100%-utilize the GPU. The bottle-neck here seems to be not the GPU itself, but the transfer rate between RAM and VRAM and here the difference between 750 Ti and 780 Ti is not that significant. You can try to improve the training speed by hiding the latency of memory transfer - you have to assure you transfer new data set to the card, while it's calculating the previous one. You may also need to split you task in to smaller portions.

It's also important to notice, that 750 Ti uses a newer (Maxwell) GPU-architecture, while 780 Ti uses the predecessor (Kepler) GPU-generation. See here for more details.