Stability of 4-bit matrix multiplication

To use newer accelerators like this, I need to perform matmul in 4-bit precision. How do I tell whether this operation is stable? Wondering if there well common heuristics in terms of properties of matrices $$\{A_i\}$$.

For classification tasks, matrix multiplication is stable if $$\langle y, y_{\text{4-bit}}\rangle>0$$ with high probability for random $$x$$ and $$y=A_d\ldots A_2 A_1 x$$.

Orthogonal matrices seem more likely to work in 4-bits, but that's too hard of a restriction.

• What is "4-bit precision"? Are these integers? I'm not aware of 4-bit floating point numbers. Feb 21 at 0:57
• That's wild.... Feb 21 at 13:18
• @WolfgangBangerth This is what majority of worlds compute will soon be used for (if not already) - super low precision matmuls for AI models Feb 21 at 18:45
• Using 4-bit numbers you can either represent 16 non-negative numbers, or 8 positive and 8 non-positive numbers. The moment you multiply two number above 4, it will overflow. Matrix-matrix multiplication would be very unstable with these numbers Feb 25 at 23:17
• That is a fairer question. If the input is normalized and convolution kernel is relatively small (or if you are working with finite fields -think of modular arithmetic-) then I would expect it to be reasonably stable. For example, if the input is the output of a logit layer and the convolution is 3x3 (as it is a classic scenario in image classifiers), this quantization would be sufficient. It is a very operator and domain dependent question. Feb 26 at 4:40