7
$\begingroup$

For a typical Ridge Regression method for solving an inverse problem $$ \min_x ||A~x - b||^2 + \lambda^2||\Gamma~x||^2 $$ Which has an analytical solution of $$ \hat{x}_{est}=(A^TA+\lambda^2 \Gamma^T\Gamma)^{-1} A^T b $$ How should $A, \lambda, x, b$ be 'normalized'? I have a model where $b_{data}$ is a discrete measurement (counts of particles). Ideally the sum of the forward projection of the estimate, $\sum A~\hat{x}_{est}$, would be equal to the sum of the observed data, $\sum A ~ \hat{x}_{est} = \sum b_{data}$

Currently I find that when as I increase the tikhonov parameter $\lambda$ from 0 to a non-zero number then the sum $\sum A ~ \hat{x}_{est}$ rapidly decreases. I need the "absolute" scale preserved.

Should my modelling matrix $A$ be normalized in some fashion? Should the data vector $b_{data}$? Please be as specific as possible.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ The question of normalization is the same as the choice of the Tikhonov parameter, which is basically tied to the statistics of the noise on the measurement. If you don't have (good) information about that, you should use a so-called heuristic parameter choice rule such as the L-curve to find a good value. (Roughly speaking, you want to balance the tracking term and the regularization term, such that $\|Ax_\lambda - b\| \sim \|\Gamma x_\lambda\|$.) Your observation about the growth is a fundamental property of Tikhonov regularization: You always have $x_\lambda \to 0$ as $\lambda\to\infty$. $\endgroup$
    – Christian Clason
    Aug 24, 2016 at 6:10
  • $\begingroup$ Ah, yes I had read about the L-curves. I suppose it's not possible to do it another way. If I want to keep the sum normalization I can force it by multiplying $\hat{x}_{est}$ by $\sum b_{data} / \sum A \hat{x}_{est}$ $\endgroup$
    – abnowack
    Aug 24, 2016 at 6:40
  • 3
    $\begingroup$ There's a lot of possible parameter choice rules, so it doesn't have to be the L-curve. If you know that the right solution satisfies the sum normalization, you should build this into the regression: Minimize $\|Ax-b\|^2$ subject to $\sum Ax = \sum b$ (e.g., using a Lagrange multiplier). This would in itself act as regularization, so you don't need Tikhonov (i.e., you can set $\lambda=0$, if $A$ is injective). $\endgroup$
    – Christian Clason
    Aug 24, 2016 at 9:55

1 Answer 1

Reset to default
6
$\begingroup$

It's a question of how you choose $\lambda$ and $\Gamma$ (of course). Think for a moment about what happens if you choose $\Gamma=I$ and make $\lambda$ large: in that case you say that it is more important to you to minimize the regularization term $\|Ix\|^2=\|x\|^2$ than to minimize the misfit $\|Ax-b\|^2$. Obviously, making the term $\|x\|^2$ small means making $x$ small, and so choosing $\lambda$ large is equivalent to saying: it is more important to me that the vector $x$ be small, than the vector $x$ fitting the data.

But you can also choose a different matrix $\Gamma$. If, for example, $\Gamma$ was a matrix that computes the differences between successive vector elements, i.e., $$ \Gamma = \begin{pmatrix} 1 & -1 & 0 & \cdots \\ 0 & 1 & -1 & \cdots \\ \vdots & \vdots & \vdots & \ddots \end{pmatrix} $$ then even if you make $\lambda$ large, what you are saying now is that you care about successive elements of $x$ being similar, but you are not trying to minimize the size of $x$ any more.

In other words, if you want to preserve certain features, then they should be in the null-space of $\Gamma$, whereas the features you want to suppress should not be in the null space.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Good way of describing this, thank you. Should that first equation be $\lambda^2 ||I x||^2 = ||x||^2$? $\endgroup$
    – abnowack
    Aug 24, 2016 at 19:48
  • $\begingroup$ No, I really meant $\|Ix\|^2=\|x\|^2$. $\lambda^2$ is really just a weight factor between the two terms that describes how much you favor minimizing the second norm over the first norm. $\endgroup$
    – Wolfgang Bangerth
    Aug 24, 2016 at 20:26
  • $\begingroup$ Ah, sorry I misread that portion. I saw it as a constraint equation not an identity. $\endgroup$
    – abnowack
    Aug 24, 2016 at 20:40
  • $\begingroup$ Yes, that was how it was intended. I just wanted to point out that if you choose $\Gamma=I$, then you're just minimizing the $\ell_2$ norm of the vector. $\endgroup$
    – Wolfgang Bangerth
    Aug 24, 2016 at 22:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.