# Using scipy.optimize to implement a neural network with back propagation

My problem is something similar to this.

I'm trying to implement a (Neural Network) Cost function, Back propogation algorithm in Python.

The Neural Network has 3 layers. Hence 2 parameters to optimize (Theta1 and Theta2; depicted as T1 and T2 in the program).

I tried the solution offered in above link, but doesn't work for me.

my Cost Function is defined with following parameters.

def CostFunction(Thetas, X, y, Lambda = None):

T1 = Thetas[0]
T2 = Thetas[1]

if(Lambda == None):
Lambda = 0

m = X.shape[0]
# so on.......


How would I optimize the above function using Scipy.optimize.fmin_bfgs?

I have the following code, doesn't seem to work:

initial_values = np.zeros([0]))
myargs = (Thetas ,X, y)
def decorated_cost(Thetas):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=1)
print scipy.optimize.fmin_bfgs(decorated_cost, Thetas , maxiter=400)


I need to optimize both Theta1 and Theta2 but the API only takes 1 parameter as input.

Also, for some reason the above code (after a couple of iterations) the order of parameters of input to Cost function seem to change when operated by optimize function.

-

## migrated from stackoverflow.comOct 8 '12 at 20:57

This question came from our site for professional and enthusiast programmers.

yes. I'm trying to minimize Theta1, Theta2 so that entire cost returned (depicted as "J" in the program) is reduced. –  raul_w Oct 7 '12 at 18:32
I tried fixing the parameters, but it keep getting the following eror TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' –  raul_w Oct 7 '12 at 19:18
initial values should not be zeros in an Neural Network training algorithm. My initial values set for the program are 2 values of Thetas (randomly initialized). Scipy optimize function runs for a couple of arguments and stops. Giving some error in function "approx_fprime" defined inside scipy.optimize module –  raul_w Oct 7 '12 at 19:38
this question has some answers on reddit. We'd be happy to take it on scicomp.se. –  Aron Ahmadia Oct 8 '12 at 8:33

The objective function should return only the cost value, not the gradient. Something like this:

def decorated_cost(Thetas, X, y, Lambda):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1))


Note that I added [0] after the call to CostFunction in decorated_cost.

You could also do something like this:

def decorated_cost(Thetas, X, y, Lambda):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[1]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1), fprime=decorated_gradient)


but that is inefficient. Instead CostFunction should be split into two functions, one for the objective function and one for the gradient.

-