So I'm working on a fitting algorithm using the levenberg-marquardt algorithm and I'm a bit stumped as to how to handle fixed parameters. Looking around at other code, like the minpack version of the l-m algorithm, it looks like they are just setting the columns of the jacobian for fixed parameters to be 0.0, which makes sense as they are not changing. $J$ is being computed numerically and is working fine if there are not any fixed parameters.

The problem is that I always get a singular uninvertible matrix for $J^TJ$ when I do this, for example something like

[[       1005,          0,    -110500],
 [          0,          0,          0],
 [    -110500,          0,    3.013e7]]  

(The 2nd parameter was held fixed in this case...)

Does anyone have some experience with the L-M algo? Any ideas as to how handle the fixed parameters in the jacobian? Should I just create a jacobian that only has the free parameters and patch in the fixed parameters somewhere down the line later?


There are a couple of options here:

  1. Don't compute elements of the Jacobian corresponding to the fixed parameters and don't include these parameters in your vector of parameters to be fitted. In your code this would require a different version of the function and Jacobian for each collection of parameters that might be fixed in a particular call to the LM routine.

  2. Compute all of the elements of $F$ and $J$ but then remove those elements of $J$ corresponding to the fixed parameters. This allows you to write one routine for computing the Jacobian but then requires you to modify your LM routine so that you can specify which parameters are fixed. Once you've deleted columns from $J$ corresponding to the fixed parameters, you can compute $J^{T}J$ without the rows and columns of 0's that you've mentioned. However, the vector $\Delta p$ that you get back as a solution to the linear system will need to have 0's inserted in positions corresponding to the fixed parameters.

  3. You could also modify the linear system of equations to force the solution to do what you want. If parameter $k$ is fixed, then zero out column $k$ and row $k$ of $J^{T}J$, then set the diagonal entry in row $k$, column $k$ to 1. Finally, make sure that the $k$th entry in the right hand side vector is 0. This will ensure that $\Delta p_{k}=0$.

  • $\begingroup$ Thanks for the info. I'm trying option #2 at the moment. $\endgroup$
    – reptilicus
    Jul 20 '14 at 16:41

I am not much into the thing but when we use l-m for training neural nets there is a trick we use:

The original equation is $$\Delta w = - (J^T J)^{-1} J^T e$$ by subtracting the term $uI$ in order to make the matrix invertable where u is a constant and I is the identity matrix so the equation will become $$\Delta w = - (J^T J+uI)^{-1} J^T e$$ so when $u$ is big enough it will insure that the matrix is invertible.

As I said I am talking from the NNs point of view there the $u$ parameter is calculated in an iterative fashion and often modified as the algorithm progress. So you may need to modify it according to your minimization objective and I sorry if I did not make this clear enough.

  • $\begingroup$ Welcome to Scicomp.SE! Could you add more details (preferably with formulas in LaTeX)? As it stands, your answer is really hard to understand. (StackExchange is about detailed, canonical answers that are useful as reference for later visitors.) $\endgroup$ Nov 27 '15 at 17:41
  • $\begingroup$ Thanks; it's clearer now. I have edited your equation to use LaTeX markup so they're easier to read. But I believe what you're describing is actually the standard Levenberg-Marquardt method (in fact, introducing the shift -- which you call $u$ -- is exactly Levenberg's contribution). However, that will not leave the parameters $e_i$ you want to fix unchanged. $\endgroup$ Nov 28 '15 at 10:09
  • $\begingroup$ First of all i wish to thank you alot it was really sweet of you I wish not to leave the parameter e unchanged all I am saying is that the identity matrix is enough to make the update matrix invertable . However I am afraid that I messinterpreted the question so if you think that the answer is redundent or unhelpfull I shall be more than pleased to remove it $\endgroup$ Nov 28 '15 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.