I only have a basic understanding of deep learning, but looking through it I had an idea on how to approximate global minima of the NN.

However, for it's activation function I am only able to use:

  • addition
  • point-wise multiplication
  • scalar products
  • multiplication with fixed matrices

on the vectors. So for example any polynomial works, as it is just a combination of multiplication and addition. Therefore, the usual activation functions like sigmoid and ReLU cannot be used as sigmoid needs division and exponentials and ReLU needs to be able to discern between cases.

For all I can see the functions I can use are all unbounded, so the usual universal approximation theorem cannot be used. However, ReLU also appears to be useful and is unbounded.

Therefore my question is: Do you know of a useful activation function that fulfils my criteria? Or is this impossible?

  • 2
    $\begingroup$ You should write out your criteria more explicitly, probably in list form using mathematical notation. $\endgroup$
    – spektr
    Jul 19 '19 at 6:26
  • 1
    $\begingroup$ If you only have these operations available, isn’t it the case that any function the network represents must be a polynomial? If so, you can just use least squares / convex programming in general to find the global minimum for almost all loss functions of interest. $\endgroup$
    – cdipaolo
    Aug 24 '19 at 23:18

@olukatorzu I think we need to look at this problem with a different perspective. It might give you some inspiration. Let's ask the question is a neural-network a universal approximator?

I will explain this with Play-Doh. hehe!

Just vectors alone is Play-Doh that has been flattened on the table (linear function).

Activation_functions creates 3d curves in the Play-Doh, so now you can create complicated 3d shapes (Non-linear function). Remember activation_functions are tools that can shape things differently (biscuit-cutter that only make circles in the Play-Doh!! LOL)

Gradient descent is the magic that turns that Play-Doh to your desired shape but only if you give that function the right instructions and tools.

Remember if you can't get the right shape with one set of Play-Doh combine it with another shape of Play-Doh (join two neural-networks together).

So, the real question is, can you shape Play-Doh to any shape you want? If yes, your criteria can be fulfilled.

  • $\begingroup$ I don't see how this analogy is any help to answer the question. $\endgroup$
    – Dirk
    Dec 25 '19 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.