# Improving function approximation with neural network

I am building a neural network to approximate a data set which takes 3 inputs and gives 1 output. After testing the network using a few different iterations of hidden layers and adjusting optimizers and activation functions, there seems to be no significant improvement to the solution. This suggests to me there is something inherently wrong with my approach. Notably, as the amount of input variables increase the problem with accuracy of the solution arises (i.e. with one idenpendent variable I can achieve very good accuracy). I believe applying the basic machine learning techiniques do not translate well to higher dimensional inputs. That being said, I am new to machine learning so there could be something I am missing. Here is an example of the output of the network: The three input parameters are altitude, mach, and fault parameter. This plot is an altitude "slice". The trend of underprediction I believe is a result of the network trying to satisfy all the different altitudes, at lower altitudes there is a noticable underprediction while higher altitudes tend to overpredict. The neural network used to generate this approximation had a basic structure. An input layer to a hidden layer of 100 nodes (celu activation) and an output layer. Different iterations of this structure seem to have no effect, they converge to the same solution. I want to know if I am doing something wrong or need to take a different approach to solving this problem. The issue seems to be trying to use a simple network to capture a multidimensional solution, but I cant find anything on proper setups for multidimensional inputs. Also, if you have any recomendations for resources on machine learning (for function approximation specifically), I would appreciate them.

EDIT: While I have made significant improvements to the accuracy of the solution using techniques described below. I found the issue causing skewness in my data was a coding error, I was redefining a normalization value for the test data when plotting (normalizing values by a different set of testing data).

• You can read about my experience with a similar problem here: bbanerjee.github.io/ParSim/assets/tech_reports/… – Biswajit Banerjee May 7 at 21:37
• By using one hidden layer with CELU, you are implicitly assuming that the positive part of the approximation can be represented as a linear combination of linear polynomials of different variables. It is not surprising that you are overestimating sometimes and underestimating at other times. Adding another would add more "basis" functions to your approximation space and could prove to be useful. – Abdullah Ali Sivas May 7 at 23:31
• I get that one celu layer might be too simple of a system, however I have tried multiple layers, with different activation functions, with little improvement to the solution. Outside this mixing and matching of layers I really have no direction to steer my systems structure. – Frosty May 8 at 1:24
• @Frosty, look at the Universal Approximation Theorem (en.wikipedia.org/wiki/Universal_approximation_theorem) particularly arbitrary depth case; it looks like if you are using RELU (I would guess something similar would apply to CELU) there will be a neural network with sufficiently many hidden layers of minimal width which would approximate your function to the desired tolerance. The number of layers could be arbitrarily large. – Abdullah Ali Sivas May 8 at 1:46

## 1 Answer

Most likely, you have the problem set up correctly and just need to adjust various things.

• What is the scale for altitude? You probably want to normalize it if you haven't already, especially since it seems like the fault parameter and mach are both $$\approx$$[0,1]
• 100 units is quite a lot. Note that more units =/= lower error, especially for something the problem you describe which seems to have relatively few examples to train on. Try something closer around 16
• On the other hand, you probably want to have 2 or maybe 3 hidden layers
• Use relu or tanh activation instead of celu.
• You need to add some regularization to the weights: l2 regularization will work
• The learning rate and regularization parameter are both very important. You need to find suitable values for them, possibly with hyperparameter optimization. The learning rate especially can be a bit tricky if tuning by hand
• Also important is how many epochs you train for

Note that a lot of these questions can be answered by hyperparameter optimization. Nni is a great package for this if you're using python. On the other hand, maybe you'll adjust a few things and find that your performance is already acceptable without it.

• Hi, thanks for the advice. I've tried implementing a few of you suggestions into my code, however I am still not getting the desired accuracy. I am currently trying to implement weight regularization however I am having some issues. With my current implementation, the weights all become very small (on the order of 10^-8), which makes sense. However, I believe this is not working as intended because the prediction of the network is completely off. Are the magnitude of these values on the expected order? If so, is there a way to "punish" the weights less? – Frosty May 15 at 21:30
• I was looking at max values in the weights and this is what I got julia> maximum(para[5]) 0.00023283294f0 julia> maximum(para[4]) 2.240105f-5 The rest of the for the network are these or smaller. – Frosty May 15 at 21:38
• im going to count this as an answer because it helped improve my model. – Frosty May 15 at 23:19
• Yes, the l2 regularization needs to have it's own hyperparameter attached to it. So the regularization term looks like \beta || w || where \beta is the strength of regularization and || w || is the 2 norm of the weights. Or if you want to think in terms of the gradient, the gradient contribution would look like 2\beta w – Taw May 16 at 0:57