I am building a neural network to approximate a data set which takes 3 inputs and gives 1 output. After testing the network using a few different iterations of hidden layers and adjusting optimizers and activation functions, there seems to be no significant improvement to the solution. This suggests to me there is something inherently wrong with my approach. Notably, as the amount of input variables increase the problem with accuracy of the solution arises (i.e. with one idenpendent variable I can achieve very good accuracy). I believe applying the basic machine learning techiniques do not translate well to higher dimensional inputs. That being said, I am new to machine learning so there could be something I am missing. Here is an example of the output of the network: The three input parameters are altitude, mach, and fault parameter. This plot is an altitude "slice". The trend of underprediction I believe is a result of the network trying to satisfy all the different altitudes, at lower altitudes there is a noticable underprediction while higher altitudes tend to overpredict. The neural network used to generate this approximation had a basic structure. An input layer to a hidden layer of 100 nodes (celu activation) and an output layer. Different iterations of this structure seem to have no effect, they converge to the same solution. I want to know if I am doing something wrong or need to take a different approach to solving this problem. The issue seems to be trying to use a simple network to capture a multidimensional solution, but I cant find anything on proper setups for multidimensional inputs. Also, if you have any recomendations for resources on machine learning (for function approximation specifically), I would appreciate them.
EDIT: While I have made significant improvements to the accuracy of the solution using techniques described below. I found the issue causing skewness in my data was a coding error, I was redefining a normalization value for the test data when plotting (normalizing values by a different set of testing data).