Take the 2-minute tour ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.

My problem is something similar to this.

I'm trying to implement a (Neural Network) Cost function, Back propogation algorithm in Python.

The Neural Network has 3 layers. Hence 2 parameters to optimize (Theta1 and Theta2; depicted as T1 and T2 in the program).

I tried the solution offered in above link, but doesn't work for me.

my Cost Function is defined with following parameters.

def CostFunction(Thetas, X, y, Lambda = None):

    T1 = Thetas[0]
    T2 = Thetas[1]

    if(Lambda == None):
        Lambda = 0

    m = X.shape[0]
    # so on.......
    return J, [Theta1_grad, Theta2_grad]

How would I optimize the above function using Scipy.optimize.fmin_bfgs?

I have the following code, doesn't seem to work:

initial_values = np.zeros([0]))
myargs = (Thetas ,X, y)
def decorated_cost(Thetas):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=1)
print scipy.optimize.fmin_bfgs(decorated_cost, Thetas , maxiter=400)

I need to optimize both Theta1 and Theta2 but the API only takes 1 parameter as input.

Also, for some reason the above code (after a couple of iterations) the order of parameters of input to Cost function seem to change when operated by optimize function.

share|improve this question
    
yes. I'm trying to minimize Theta1, Theta2 so that entire cost returned (depicted as "J" in the program) is reduced. –  raul_w Oct 7 '12 at 18:32
    
I tried fixing the parameters, but it keep getting the following eror TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' –  raul_w Oct 7 '12 at 19:18
    
initial values should not be zeros in an Neural Network training algorithm. My initial values set for the program are 2 values of Thetas (randomly initialized). Scipy optimize function runs for a couple of arguments and stops. Giving some error in function "approx_fprime" defined inside scipy.optimize module –  raul_w Oct 7 '12 at 19:38
    
this question has some answers on reddit. We'd be happy to take it on scicomp.se. –  Aron Ahmadia Oct 8 '12 at 8:33
add comment

migrated from stackoverflow.com Oct 8 '12 at 20:57

This question came from our site for professional and enthusiast programmers.

1 Answer

up vote 2 down vote accepted

The objective function should return only the cost value, not the gradient. Something like this:

def decorated_cost(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1))

Note that I added [0] after the call to CostFunction in decorated_cost.

You could also do something like this:

def decorated_cost(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

def decorated_gradient(Thetas, X, y, Lambda):
    return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[1]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1), fprime=decorated_gradient)

but that is inefficient. Instead CostFunction should be split into two functions, one for the objective function and one for the gradient.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.