4
$\begingroup$

This question is based more on the theory of neural networks than my particular implementation. Therefore I will leave out my code unless requested.

I'm working on a project in C# which can create and train feed forward neural networks. It works well and can train networks to perform some simple functions such as squaring the input or other basic math functions.

My next task was to use this to train a network to predict an outcome of positive or negative based on 39 continuous input variables. I have a dataset comprised of around 7000 training examples.

I did a ROC curve analysis on each input variable and found that some are very good predictors and some are almost useless.

I sorted the variables by AUC, the best was around 78%. I created a network with only this variable as it's input and one hidden layer with one neuron. After training, the network performed better than random but not amazingly.

I next added the second best variable by AUC (75%) as another input, keeping everything else the same. After training the network's fitness score was about 1.5 times better than with one input.

Next I added the third best variable and trained. The networks fitness score was only around 85% of that when it had 2 inputs. I have tried adding more neurons to the hidden layer, running the training for more iterations and many different runs of initial random weights and biases and it seems that no matter what I do the network will perform worse with 3 inputs than 2.

I find this confusing as neural networks are theoretically able to replicate any function. So even if the 3rd variable I introduced was a rubbish predictor then the training should just be able to set the weight of that input to zero and it will perform the same as it did with two inputs.

Questions:

  1. Is my assumption correct that adding inputs to a neural network (and given enough training iterations) it should never perform worse than with less inputs?

  2. Do I require more neurons in the hidden layer when adding more inputs for assumption 1 to hold? Would a good guideline be one neuron in the hidden layer for every input?

  3. Is it a fair assumption that, given everything I have tried, my training code has not been written well enough to train networks to their full capability? (Or in other words, it should be easy for the networks to just ignore the third input, so the problem is more likely to be that my training code is not good enough rather than the networks are not able to easily ignore the third input)?

  4. Is it a good method to create networks with successively more inputs to see which variables improve the fitness the most, and then not use those that don't improve the network? Or should I just create a network with all 39 input variables, train it for a long time and then perform some kind of analysis on the resulting weights to see which inputs are being weighted close to zero and remove them? (as they are essentially being ignored by the network).

Extra details: Each neuron in the hidden layer is connected to every input and the output neuron is connected to every neuron in the hidden layer. All neurons use a sigmoid function with weights and a bias. If the output of the output neuron is >0.8 then this is taken as the network predicting positive, otherwise negative.

$\endgroup$
  • $\begingroup$ After a long time spent adjusting the properties of my neural network, and the training parameters. I have found that I needed to increase the number of training iterations to achieve the same level of fitness with more inputs. I have not used back propagation but implemented my own hill climbing algorithm which works well but may not be perfect. I will post on the main stack overflow to ask for improvements to my hill climbing algorithm. $\endgroup$ – Karl Apr 29 '15 at 9:01
  • 1
    $\begingroup$ I planned to answer but see that you already found the issue. I would however point out that there are already many open source NN toolboxes available which have all kinds of optimizers already built in, as well as other nice features like GPU acceleration. Reinventing the wheel is really unnecessary here and only creates a high chance of introducing subtle bugs. $\endgroup$ – cfh Apr 29 '15 at 13:03
  • $\begingroup$ @Karl, you really describe a situation I'm in perfectly. Was hoping there would be some thoughtful answers here. Did you find anything helpful? Will explore longer training for more inputs. $\endgroup$ – Joshua R. Feb 1 '18 at 23:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.