5
$\begingroup$

Suppose I have values of log(P(x_i)), i.e. log-probabilities to events x_i. The probabilities are very small, so that these log-values are of the order of -1e3.

I want to compute an expectation value. In order to evaluate the corresponding sum, I need the probabilities P(x_i) themselves so I tried to call numpy.exp. This however, returns only 0 as the precision of basic floats in Python is not high enough to resolve a number like exp(-1000).

What is the typical way to circumvent this issue?

$\endgroup$

1 Answer 1

8
$\begingroup$

If your final result is of the order of magnitude of exp(-1000) $\approx 5 \cdot 10^{-435}$, then you are out of luck; no matter how you compute it, it will always underflow. There is simply no representable floating-point binary64 number that small. The slower Bigfloats are the only way out.

Some of the issues with large/small intermediate values, however, can be solved with a few manipulations:

$\endgroup$
2
  • $\begingroup$ Thank you for the hints! I will try to set up Bigfloat then. $\endgroup$
    – reloh100
    Feb 8, 2022 at 22:57
  • 1
    $\begingroup$ Have you tried the log-sum-exp trick before abandoning the idea? It seems tailored to what you need. $\endgroup$ Feb 9, 2022 at 7:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.