To use newer accelerators like this, I need to perform matmul in 4-bit precision. How do I tell whether this operation is stable? Wondering if there well common heuristics in terms of properties of matrices $\{A_i\}$.

For classification tasks, matrix multiplication is stable if $\langle y, y_{\text{4-bit}}\rangle>0$ with high probability for random $x$ and $y=A_d\ldots A_2 A_1 x$.

Orthogonal matrices seem more likely to work in 4-bits, but that's too hard of a restriction.

  • 1
    $\begingroup$ What is "4-bit precision"? Are these integers? I'm not aware of 4-bit floating point numbers. $\endgroup$ Feb 21 at 0:57
  • 1
    $\begingroup$ That's wild.... $\endgroup$ Feb 21 at 13:18
  • 1
    $\begingroup$ @WolfgangBangerth This is what majority of worlds compute will soon be used for (if not already) - super low precision matmuls for AI models $\endgroup$ Feb 21 at 18:45
  • 1
    $\begingroup$ Using 4-bit numbers you can either represent 16 non-negative numbers, or 8 positive and 8 non-positive numbers. The moment you multiply two number above 4, it will overflow. Matrix-matrix multiplication would be very unstable with these numbers $\endgroup$ Feb 25 at 23:17
  • 1
    $\begingroup$ That is a fairer question. If the input is normalized and convolution kernel is relatively small (or if you are working with finite fields -think of modular arithmetic-) then I would expect it to be reasonably stable. For example, if the input is the output of a logit layer and the convolution is 3x3 (as it is a classic scenario in image classifiers), this quantization would be sufficient. It is a very operator and domain dependent question. $\endgroup$ Feb 26 at 4:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.