Training accuracy improves but test set accuracy remains the same

Question

I have built an ANN model with 5 hidden layers and 100 nodes in each layer to solve a multilabel classification problem. After the first run, I get a training accuracy of ~66% and a test set accuracy of ~55%. When I ran the code with 1000 nodes in each hidden layer,the training accuracy improves to ~98% but the test set accuracy is around ~52%. I tried dropout but it only lowered the training accuracy a bit while the wide difference between the training and test accuracy still exists. Can anyone please help?

Answer 1

I think you experience the implications of overtraining. From this question and this paper:

Since a neural network with a sufficient number of neurons in the hidden layer can exactly implement an arbitrary training set, it can learn both investigated dependencies and a noise that will lower the predictive ability of the network.

So, in your case, you increased the number of nodes in the hidden layer - and now, your training set resulted in overtraining. Your neural-network now is much more directed on replicating the data in the training dataset, as opposed to "prediction of the reality". This results in the stagnation (and even reduction in quality) for the test dataset.

It is hard to give more advice without details. But I would suggest at least to:

• play with different partitioning of the sample into training and test datasets
• play with the model itself in terms of features
• play with a training algorithm