I only have a basic understanding of deep learning, but looking through it I had an idea on how to approximate global minima of the NN.
However, for it's activation function I am only able to use:
- point-wise multiplication
- scalar products
- multiplication with fixed matrices
on the vectors. So for example any polynomial works, as it is just a combination of multiplication and addition. Therefore, the usual activation functions like sigmoid and ReLU cannot be used as sigmoid needs division and exponentials and ReLU needs to be able to discern between cases.
For all I can see the functions I can use are all unbounded, so the usual universal approximation theorem cannot be used. However, ReLU also appears to be useful and is unbounded.
Therefore my question is: Do you know of a useful activation function that fulfils my criteria? Or is this impossible?