3
$\begingroup$

I just for the first time saw the function softmax() in this SO answer to How do I use a minimization function in scipy with constraints and was intrigued.

Another way of weighting variables where the sum of the weights is constrained to equal 1, is to use minimize with no constraints, initialize with near-zero values but use a softmax in the scoring function.

SciPy's scipy.special.softmax links to the IMA Journal of Numerical Analysis' Accurately computing the log-sum-exp and softmax functions The abstract begins:

Evaluating the log-sum-exp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification.

and the introduction cites examples, but the paper focuses on issues of computational evaluation. Wikipedia's Softmax function seems long, thorough and descriptive, but seems written for audiences who already have some knowledge in machine learning or are fluent "speakers of math".

I do a lot of fitting and optimization but so far scipy.optimize.minimize on straightforward python functions has offered everything I need, and when I hit the linked Stack Overflow question all I needed to know is that I could add bounds to the parameters.

But I find this softmax() function intriguing so I can't let it go. In order to at least get me started, could someone address:

Question: What problems does softmax() solve and when should I think of using it?

and do so in relatively simple terms?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ J. Denker, O. Matan, "Handwritten Character Recognition Using Neural Network Architectures." In 4th USPS Advanced Technology Conference, Washington, D.C., Nov. 1990, pp. 1003-1011: " John Bridle recently proposed an normalization scheme for classifiers with $N$ mutually-exclusive outcomes (Bridle, 1989). This scheme, which he has named Softmax, is applied as a post-processor to the "raw outputs" of the neural network. Softmax's appeal is that its outputs are positive and sum to 1, thus satisfying the axioms of probability theory. " $\endgroup$
    – njuffa
    Nov 13, 2022 at 2:07
  • 1
    $\begingroup$ To read about the claimed advantages of using Softmax from the horse's mouth, see John S. Bridle, "Probabilistic Interpretation of Feedforward Classification Network Outputs, with Relationships to Statistical Pattern Recognition." In F.F. Soulié and J. Hérault (eds.), Neurocomputing, Springer 1990, pp. 227-236. See first page of Bridle's paper here. $\endgroup$
    – njuffa
    Nov 13, 2022 at 2:12
  • $\begingroup$ @njuffa Probability axioms and horses mouths aside, do you think you'd consider posting an "...in simple terms" class answer? $\endgroup$
    – uhoh
    Nov 14, 2022 at 8:27

1 Answer 1

Reset to default
3
$\begingroup$

Suppose you are training a neural network to predict the probability that a given picture is a picture of a cat, dog or tiger. This is an example of a problem called multi-class classification. The picture can belong to exactly one of these 3 classes. A typical neural network for this would have three outputs. First output predicts p(cat), second output predicts p(dog) and third output predicts p(tiger). Note: p denotes probability, so p(cat) means probability of being a cat.

Now you would like the probabilities to obey the laws of probability.

i.e.

0<=p(cat)<=1, 0<=p(dog)<=1, 0<=p(tiger)<1

and

p(cat) + p(dog) + p(tiger) = 1.0

Without doing something special, your neural network will not obey these constraints. You will have to do something special. This "something special" is the softmax layer or softmax activation function.

So, you would design a neural network to have 3 outputs, add a softmax layer on top of that, and what comes out of the softmax layer will be guaranteed to obey the laws of probability. Which you can use with cross-entropy loss functions. You can look-up the math of the softmax function on Wikipedia. I find Aurelien Geron's book Hands-on machine learning with Scikit-Learn, Keras and Tensorflow instructive.

Edit: As requested by @uhoh: explanation which was found useful

Improve this answer
$\endgroup$
5
  • $\begingroup$ I can just rescale the outputs with a linear transform to get the same thing, why the logs and exponentials? $\endgroup$
    – uhoh
    Nov 16, 2022 at 12:17
  • $\begingroup$ If you could write down your linear transform, I'd be happy to comment $\endgroup$
    – NNN
    Nov 16, 2022 at 12:34
  • $\begingroup$ Maybe stackoverflow.com/a/47763299/13560598 is what you're looking for. $\endgroup$
    – NNN
    Nov 16, 2022 at 13:51
  • 1
    $\begingroup$ To implement $0 \le p_c, p_d, p_t \le 1$ and $p_c + p_d + p_e = 1$ with a linear transform you first add a constant to each so that their minimum is now zero, then divide each by the sum of all three. So there must be something more to this than just making some output values "obey the laws of probability." Okay I"ll take a look at that link in the morning. Thanks! $\endgroup$
    – uhoh
    Nov 16, 2022 at 14:13
  • 1
    $\begingroup$ Yes, that's it! I think that a link to the SO discussion (or similar) will complete your answer. $\endgroup$
    – uhoh
    Jan 16 at 3:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.