# Using scipy.optimize to implement a neural network with back propagation

My problem is something similar to this.

I'm trying to implement a (Neural Network) Cost function, Back propogation algorithm in Python.

The Neural Network has 3 layers. Hence 2 parameters to optimize (Theta1 and Theta2; depicted as T1 and T2 in the program).

I tried the solution offered in above link, but doesn't work for me.

my Cost Function is defined with following parameters.

def CostFunction(Thetas, X, y, Lambda = None):

T1 = Thetas[0]
T2 = Thetas[1]

if(Lambda == None):
Lambda = 0

m = X.shape[0]
# so on.......


How would I optimize the above function using Scipy.optimize.fmin_bfgs?

I have the following code, doesn't seem to work:

initial_values = np.zeros([0]))
myargs = (Thetas ,X, y)
def decorated_cost(Thetas):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=1)
print scipy.optimize.fmin_bfgs(decorated_cost, Thetas , maxiter=400)


I need to optimize both Theta1 and Theta2 but the API only takes 1 parameter as input.

Also, for some reason the above code (after a couple of iterations) the order of parameters of input to Cost function seem to change when operated by optimize function.

• yes. I'm trying to minimize Theta1, Theta2 so that entire cost returned (depicted as "J" in the program) is reduced.
– raul_w
Oct 7 '12 at 18:32
• I tried fixing the parameters, but it keep getting the following eror TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple' TypeError: unsupported operand type(s) for -: 'tuple' and 'tuple'
– raul_w
Oct 7 '12 at 19:18
• initial values should not be zeros in an Neural Network training algorithm. My initial values set for the program are 2 values of Thetas (randomly initialized). Scipy optimize function runs for a couple of arguments and stops. Giving some error in function "approx_fprime" defined inside scipy.optimize module
– raul_w
Oct 7 '12 at 19:38
• this question has some answers on reddit. We'd be happy to take it on scicomp.se. Oct 8 '12 at 8:33

The objective function should return only the cost value, not the gradient. Something like this:

def decorated_cost(Thetas, X, y, Lambda):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1))


Note that I added [0] after the call to CostFunction in decorated_cost.

You could also do something like this:

def decorated_cost(Thetas, X, y, Lambda):
return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[0]

return NeuralNetwork.CostFunction(Thetas, X, y, Lambda=Lambda)[1]

initial_values = np.array([0.1, 0.2])
print scipy.optimize.fmin_bfgs(decorated_cost, initial_values, maxiter=400, args=(X, y, 1), fprime=decorated_gradient)


but that is inefficient. Instead CostFunction should be split into two functions, one for the objective function and one for the gradient.