When are two vectors considered “close”?

Question

I want to check numerically if a certain vector relation like $$ \alpha_1v_1+...+\alpha_kv_k=c \ (1)$$ holds (where $v_i,c$ are vectors of $100$ or more components). For this, I use least squares approximation, and to see the error, I evaluate the norm of the difference $$ \|\alpha_1v_1+...+\alpha_kv_k-c\|.$$ All $v_i$ and $c$ are computed numerically, so they are subject to errors. The size of the vectors is $100$ components.

When can I say that the error is small enough, i.e. the relation $(1)$ holds numerically?

I ask the question because I obtain errors between $0.2$ and $1$, which seem rather large, but if we look component wise, the average error is small.

Answer 1

You want to compare two scalars $a=\sum_k \alpha_k v_k$ and $c$ for "equality". This, of course, does not work in floating point arithmetic, but what one often does is to see whether the difference is small compared to the sizes of the two individual components. In other words, $a$ and $c$ would be close if the relationship $$ |a-c| \le \varepsilon \frac{|a|+|c|}{2} $$ is true for a suitably chosen $\varepsilon$. Here, you compare the difference against the average size of the objects you take the difference of. A typical size for $\varepsilon$ would be $10^{-12}$ if you deal with floating point numbers: small enough that it really counts as small, but at the same large enough compared to the floating point roundoff error of $10^{-16}$ that you can afford a reasonable number of operations in the computations of $a$ and $c$ (where every one of these operations would of course accumulate to the numerical error in these quantities).

Answer 2

In scipy.numpy there is the function allclose to compare two arrays. Have a look at the default values there.

However, it is problem dependent what you can call numerically close... Apart from the rounding error you may need to consider approximation errors and the like.

Furthermore, there is a lot of norms you can take. Often the problem comes with a norm. In finite element computations, one typically takes the euclidean norm weighted with a mass matrix M. Then, M weights the components of the error vector properly.