Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm trying to re-implement Neural Networks in Python. I implemented the cost function and the backpropagation algorithm correctly. I have checked them by executing its Octave equivalent code.

But when I'm trying to use the scipy.optimize.fmin_cg function, the iterations take a very long time to execute. It exits with a warning and gives me an error saying that the "desired error rate wasn't achieved".

The same program in Octave executes fine. However, it has its own fmincg function defined.

What am I doing wrong?

share|improve this question

As a future hint, use the double question mark ?? to pull the source of the function:

Looking at the scipy sources this error message comes up if the internal parameter alpha_k is zero or None.

This value in turn is tied to the internal Wolfe Line search algorithm. In particular it is called when the search doesn't find a better value along the search trajectory. Your function likely has a linear contour somewhere along it that the optimizer falls into and gets confused.

Perhaps try adding a callback, and see where the failing search is generated?

share|improve this answer
As a follow up to this, scipy picks its max iteration count based on the size of the vector to be optimized. This is a better than average guess, but ought to be documented more obviously. – meawoppl Jan 15 '14 at 0:12

I assume you didn't specify the fprime parameter. If you don't provide this param fmin_cg has to figure out its own solution what usually is much slower than which a provided optimal solution. Your code might look like this:

theta = fmin_cg(compute_cost_reg, fprime=compute_gradient_reg,
                x0=theta, args=(X, y, lambd), maxiter=50)
share|improve this answer

I know this is an old question, but I've just been struggling with a similar issue and thought I'd post my solution incase anyone else comes across this.

I found the problem was that I hadn't properly initialised the $ \Theta $ vector to break the symmetry.

After doing this, I ran fmin_cg with f and fprime, and although it still ran rather slowly, it did so without any of the errors or warnings I had been seeing up to that point.

In fact, it actually did a better job of minimising the cost function than the matlab implementation, despite both having max_iters = 100.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.