8
$\begingroup$

I'm trying to re-implement Neural Networks in Python. I implemented the cost function and the backpropagation algorithm correctly. I have checked them by executing its Octave equivalent code.

But when I'm trying to use the scipy.optimize.fmin_cg function, the iterations take a very long time to execute. It exits with a warning and gives me an error saying that the "desired error rate wasn't achieved".

The same program in Octave executes fine. However, it has its own fmincg function defined.

What am I doing wrong?

Improve this question
$\endgroup$
2
  • $\begingroup$ everybody, I got an error'It seems the kernel died unexpectedly" where did I do wrong, is there something else I need to do in order to use the fmin_cg for vectorized calculation? $\endgroup$ – Wei Gao Sep 21 '16 at 1:39
  • $\begingroup$ To avoid symmetry breaking you should initialize params as: params = numpy.random.randn(dims)*0.01 $\endgroup$ – Shivam Singh Oct 1 '18 at 19:35
6
$\begingroup$

As a future hint, use the double question mark ?? to pull the source of the function:

Looking at the scipy sources this error message comes up if the internal parameter alpha_k is zero or None.

This value in turn is tied to the internal Wolfe Line search algorithm. In particular it is called when the search doesn't find a better value along the search trajectory. Your function likely has a linear contour somewhere along it that the optimizer falls into and gets confused.

Perhaps try adding a callback, and see where the failing search is generated?

Improve this answer
$\endgroup$
1
  • $\begingroup$ As a follow up to this, scipy picks its max iteration count based on the size of the vector to be optimized. This is a better than average guess, but ought to be documented more obviously. $\endgroup$ – meawoppl Jan 15 '14 at 0:12
2
$\begingroup$

I assume you didn't specify the fprime parameter. If you don't provide this param fmin_cg has to figure out its own solution what usually is much slower than which a provided optimal solution. Your code might look like this:

theta = fmin_cg(compute_cost_reg, fprime=compute_gradient_reg,
                x0=theta, args=(X, y, lambd), maxiter=50)
Improve this answer
$\endgroup$
0
$\begingroup$

I know this is an old question, but I've just been struggling with a similar issue and thought I'd post my solution incase anyone else comes across this.

I found the problem was that I hadn't properly initialised the $ \Theta $ vector to break the symmetry.

After doing this, I ran fmin_cg with f and fprime, and although it still ran rather slowly, it did so without any of the errors or warnings I had been seeing up to that point.

In fact, it actually did a better job of minimising the cost function than the matlab implementation, despite both having max_iters = 100.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.