How to compute the optimal ridge regression model

Question

I found R function ridge.cv very useful.

I would like to implement the equivalent function in MATLAB. As a starting point, I used MATLAB function b0 = ridge(y,X,k,scale), however it gives completely different results. Why does this might happen? What value should I set for variable "scale" (1 or 0 and what's their difference?)? And how could I implement it from the scratch in MATLAB?

Answer 1

You can get the source and documentation for the parcor package which includes the ridge.cv function on CRAN.

You can implement the function in matlab by looking at the source. Alternatively, I think it should be possible to call R from Matlab.

Answer 2

One form of ridge regression is the problem:
     minimize |Ax - b|, and keep x near a given point x0 (often 0).
If we cast this as:
     minimize |Ax - b|^2 + w^2 |x - x0|^2 ,
we can solve it by running least squares on these extended inputs:

[ A ]    [ b ]
[ w I ]  [ w x0 ]

Here w is a weight factor that balances minimizing |Ax - b|, and keeping x near x0.
How should one choose it if one has no idea ? A rule of thumb is to make |Ax - b| and |x - x0| roughly equal -- print those values.

Added: Weighted least squares, with different weights w$_i$, is a powerful method.
For example, w$_i \sim \frac{1}{\sqrt |\text{x}_i|}$ pushes small x$_i$ towards 0 (think iPhone fingers), "sparsifies" —
a poor man's approach to L1-norm regularization; see LASSO.

Not Matlab, but hope this helps.