7
$\begingroup$

Why do people use the classical least squares approach so often ? If I use the absolute value instead of the power, I immediately know how far away the solution is:

$$ res = \frac{abs(x - x_{model})}{x}$$

What is the benefit of squaring ?

$\endgroup$
  • 3
    $\begingroup$ I faintly remember reading a tongue-in-cheek comment along the lines of "The statisticians think there is a physical reason, and the scientists think there is a statistical reason". $\endgroup$ – fred Oct 23 '15 at 13:04
  • 1
    $\begingroup$ See also why-havent-robust-and-resistant-statistics-replaced-classical-techniques on stats.stackexchange . $\endgroup$ – denis Dec 15 '15 at 13:19
16
$\begingroup$

Why do people use the classical least squares approach so often?

Primarily, squaring makes the problem twice-differentiable, thus many different solution methods apply (quasi-Newton methods, Levenberg-Marquardt, Gauss-Newton), and there is still some flexibility with respect to what is being squared (e.g., I can replace $x$ with $f(x)$ for a wide class of functions $f$).

From these methods, there are some nice linear algebra benefits, too: we can use Cholesky/sparse Cholesky.

If I use the absolute value instead of the power, I immediately know how far away the solution is...

The absolute value approach (more generally, L1-minimization approaches) can sometimes be tractable, if an LP reformulation is possible, but it can also really slow things down if an LP formulation is not possible (e.g., taking the L1 norm of a difference of nonlinear functions).

$\endgroup$
  • 4
    $\begingroup$ The computational advantages of least-squares approaches are in fact so great that it is common to solve problems with more "robust" cost-functions with iteratively reweighted least squares (which converts the optimization into a sequence of least squares problems). $\endgroup$ – GeoMatt22 Oct 23 '15 at 2:47
12
$\begingroup$

Athough Geoff Oxberry's answer addresses computational reasons why it's easier to minimize the sum of squared residuals than the sum of absolute values, it doesn't discuss statistical reasons for preferring the least squares solution.

For problems in which measurement errors are independent and normally distributed, the (appropriately weighted) least squares solution provide a maximum likelihood estimator of the parameters. Even if the measurement errors aren't independent, it's possible to extend ordinary least squares to correct for correlation of the residuals. In the linear case there are a number of other important statistical properties that the least squares solution has.

On the other hand, there are times when the parameter estimate the minimizes the sum of absolute values of the residuals is better from a statistical point of view. In particular, if there are rare but extremely bad "outlier" measurements that aren't normally distributed, minimizing the sum of the absolute values of the residuals will typically provide a better estimate than least squares. In "robust regression" the sum of absolute values of the residuals and other related objective functions are often used.

$\endgroup$
2
$\begingroup$

If your question is "do people sometimes use the $2$-norm to measure distances without full justification?," then the answer is probably yes. Choosing the correct norm to measure differences between high-dimensional objects is often not simple to decide.

One motivation for using the $2$-norm to measure distances between vectors in $\mathbb{R}^n$ is that it is invariant under Euclidean/rigid transformations. If you have some reason to want your algorithm to also be invariant under those transformations, then it is a natural choice.

There are also reasons to use the $1$-norm (often as a convexification of the $0$-"norm" which counts the nonzero entries, if you care about sparsity). See Geoff Oxbury's answer for more on this.

$\endgroup$
1
$\begingroup$

For one-dimensional data, the point that minimizes the sum of squared distances is the arithmetic mean and the point(s) that minimize the sum of absolute distances is/are the median. A median-like solution may sometimes be prefered over a mean-like solution, but the non-uniqueness and "weird" dependencies make it at elast less tractable theoretically.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.