4
$\begingroup$

My problem is something similar to this.

I'm trying to implement a (Neural Network) Cost function, Back propogation algorithm in Python.

The Neural Network has 3 layers. Hence 2 parameters to optimize (Theta1 and Theta2; depicted as T1 and T2 in the program).

I tried the solution offered in above link, but doesn't work for me.

my Cost Function is defined with following parameters.

def CostFunction(Thetas, X, y, Lambda = None):

    T1 = Thetas[0]
    T2 = Thetas[1]

    if(Lambda == None):
        Lambda = 0
    
    m = X.shape[0]
    # so on.......
    return J, [Theta1_grad, Theta2_grad]

How would I optimize the above function using Scipy.optimize.fmin_bfgs?

I have the following code, doesn't seem to work:

initial_values = np.zeros([0]))
myargs = (Thetas ,X, y)
def decorated_cost(Thetas):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=1)
print scipy.optimize.fmin_bfgs(decorated_cost, Thetas , maxiter=400)

I need to optimize both Theta1 and Theta2 but the API only takes 1 parameter as input.

Also, for some reason the above code (after a couple of iterations) the order of parameters of input to Cost function seem to change when operated by optimize function.

Improve this question
$\endgroup$
4
  • $\begingroup$ yes. I'm trying to minimize Theta1, Theta2 so that entire cost returned (depicted as "J" in the program) is reduced. $\endgroup$
    – raul_w
    Oct 7, 2012 at 18:32
  • $\begingroup$ I tried fixing the parameters, but it keep getting the following eror TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' $\endgroup$
    – raul_w
    Oct 7, 2012 at 19:18
  • $\begingroup$ initial values should not be zeros in an Neural Network training algorithm. My initial values set for the program are 2 values of Thetas (randomly initialized). Scipy optimize function runs for a couple of arguments and stops. Giving some error in function "approx_fprime" defined inside scipy.optimize module $\endgroup$
    – raul_w
    Oct 7, 2012 at 19:38
  • $\begingroup$ this question has some answers on reddit. We'd be happy to take it on scicomp.se. $\endgroup$
    – Aron Ahmadia
    Oct 8, 2012 at 8:33

1 Answer 1

Reset to default
3
$\begingroup$

The objective function should return only the cost value, not the gradient. Something like this:

def decorated_cost(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1))

Note that I added [0] after the call to CostFunction in decorated_cost.

You could also do something like this:

def decorated_cost(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

def decorated_gradient(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[1]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1), fprime=decorated_gradient)

but that is inefficient. Instead CostFunction should be split into two functions, one for the objective function and one for the gradient.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.