This question is based more on the theory of neural networks than my particular implementation. Therefore I will leave out my code unless requested.
I'm working on a project in C# which can create and train feed forward neural networks. It works well and can train networks to perform some simple functions such as squaring the input or other basic math functions.
My next task was to use this to train a network to predict an outcome of positive or negative based on 39 continuous input variables. I have a dataset comprised of around 7000 training examples.
I did a ROC curve analysis on each input variable and found that some are very good predictors and some are almost useless.
I sorted the variables by AUC, the best was around 78%. I created a network with only this variable as it's input and one hidden layer with one neuron. After training, the network performed better than random but not amazingly.
I next added the second best variable by AUC (75%) as another input, keeping everything else the same. After training the network's fitness score was about 1.5 times better than with one input.
Next I added the third best variable and trained. The networks fitness score was only around 85% of that when it had 2 inputs. I have tried adding more neurons to the hidden layer, running the training for more iterations and many different runs of initial random weights and biases and it seems that no matter what I do the network will perform worse with 3 inputs than 2.
I find this confusing as neural networks are theoretically able to replicate any function. So even if the 3rd variable I introduced was a rubbish predictor then the training should just be able to set the weight of that input to zero and it will perform the same as it did with two inputs.
Questions:
Is my assumption correct that adding inputs to a neural network (and given enough training iterations) it should never perform worse than with less inputs?
Do I require more neurons in the hidden layer when adding more inputs for assumption 1 to hold? Would a good guideline be one neuron in the hidden layer for every input?
Is it a fair assumption that, given everything I have tried, my training code has not been written well enough to train networks to their full capability? (Or in other words, it should be easy for the networks to just ignore the third input, so the problem is more likely to be that my training code is not good enough rather than the networks are not able to easily ignore the third input)?
Is it a good method to create networks with successively more inputs to see which variables improve the fitness the most, and then not use those that don't improve the network? Or should I just create a network with all 39 input variables, train it for a long time and then perform some kind of analysis on the resulting weights to see which inputs are being weighted close to zero and remove them? (as they are essentially being ignored by the network).
Extra details: Each neuron in the hidden layer is connected to every input and the output neuron is connected to every neuron in the hidden layer. All neurons use a sigmoid function with weights and a bias. If the output of the output neuron is >0.8 then this is taken as the network predicting positive, otherwise negative.
