I was previously running my neural networks using the Lasagne library to build and train neural networks in Theano on an NVIDIA GTX 750 Ti. I'm using a genetic algorithm to tune the hyperparameters of my neural network, so I bought a GTX 780 Ti to speed up training each neural network. The 780 Ti should have over 3 times the single precision floating point performance of the 750 Ti, but I don't see any speed-up at all.
- I checked GPU utilization in GPU-Z and it is the same for both GPUs (mid-high 60's%). Why is the beefier GPU using the same percentage of resources, but not running any faster?
- I uninstalled all NVIDIA software including the driver (restarting twice in the process), then re-installed everything using the CUDA 7.5 Toolkit Installer and then I installed the CUDNN 7.5 (5?) libraries/headers/binaries I restarted my neural network and it was still performing as poorly as before.
Some probably important information:
- This is on Windows 7
- For nvcc I'm using
cl-version=2013because that's what I have and it works fine - CPU is an Intel Core i5-2500K @ 3.3GHz
- 8 GB system memory
- Data is loaded off of a Samsung SSD 850 EVO 500GB drive
- 3 GB of GPU memory (other specs can be found at the link above)
- Card is plugged into a PCI-E Gen2 (1x16) slot
- The networks vary a lot, but they're all multilayer perceptrons with between 1-5 hidden layers
- 15-150 neurons per layer, all DenseLayers
- 75 inputs, 10 outputs
- Currently no drop-out or Gaussian noise before any layers
- Batch size is 1024
- Data is shuffled before each epoch
- Hidden layer activation functions are all elu
- Output activation function can be sigmoid, tanh, or elu
- Loss function is mean squared error
- Updates use nesterov momentum
- Training data size is 274MB with 934918 samples
- Validation data size is 30.5MB with 103880 samples
My first thought was that my networks were too small to make any difference on a small vs. big GPU, but the bigger GPU is stressed just as much as the littler one when running, plus these networks were running faster on the GRID K520 GPUs on Amazon EC2 (on Ubuntu) (both the K520 and 780 Ti use the Kepler GPU architecture while the 750 Ti was Nvidia's first Maxwell card).
What could be causing my new card to be so slow? How can I get it to run faster than the older crappier card?
Edit: Running the nn on Ubuntu I see a significant speed-up, not as big as I hoped, but definitely noticeable. I'm probably not making full use of the GPU with my small networks. One possible reason for the slowness in Windows is WDDM mode vs. TCC mode: https:// devtalk.nvidia.com /default/topic/895331/cuda-setup-and-installation/tesla-k20-vs-titan-x-performance-for-the-same-code/post/4724884/#4724884. I'm going to try installing my old card in a PCI-E 1x slot and make that the display driver and use my new card for computation only
Edit the second: Here's what one of my neural networks would look like:
input_var = T.matrix('inputs')
input = lasagne.layers.InputLayer(shape=(1024, 75), input_var=input_var)
network = lasagne.layers.DenseLayer(input, 40,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
network = lasagne.layers.DenseLayer(network, 113,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
network = lasagne.layers.DenseLayer(network, 76,
nonlinearity=lasagne.nonlinearities.elu,
W=lasagne.init.GlorotUniform())
output = lasagne.layers.DenseLayer(
network, num_units=10,
nonlinearity=lasagne.nonlinearities.tanh)
prediction = lasagne.layers.get_output(output)
loss = lasagne.objectives.squared_error(prediction, target_var)
loss = loss.mean()
updates = lasagne.updates.nesterov_momentum(loss, params, learning_rate=0.5, momentum=0.5)
test_prediction = lasagne.layers.get_output(output, deterministic=True)
test_loss = lasagne.objectives.squared_error(test_prediction, target_var)
test_loss = test_loss.mean()
train_fn = theano.function([input_var, target_var], loss, updates=updates)
val_fn = theano.function([input_var, target_var], test_loss)
