I am trying out the scipy Minimizer from the scipy.optimize package. I was just playing around to see how the optimization package works. But I cannot seem to get the Optimizer to work when I supply my own Gradient (Option jac). I Always get value Errors. Here is my example:
import numpy as np from scipy import optimize # Simulate own data x = np.arange(-5,10,1) theta = np.array([5,2]) noise = np.random.normal(0,1,len(x)) y = theta + theta * x + noise # the cost function def cost_function(theta, x, y): cost = 0 theta0, theta1 = theta def cost_sum(x_i, y_i): cost_component = (y_i - theta0 - theta1*x_i)**2 return cost_component for i in range(len(y)): cost += cost_sum(x[i], y[i]) return cost # the gradient def cost_grad(theta, x, y): grad = np.zeros((1,2)) theta_diff_1 = np.array([0.01,0]) theta_diff_2 = np.array([0,0.01]) grad[0,0] = ( cost_function(theta + theta_diff_1, x, y) - cost_function(theta - theta_diff_1, x, y) ) / 0.02 grad[0,1] = ( cost_function(theta + theta_diff_2, x, y) - cost_function(theta - theta_diff_2, x, y) ) / 0.02 return grad # call Optimizer from scipy res = optimize.minimize(cost_function, theta, args=(x,y), method = 'BFGS', jac=cost_grad) print(res)
This Code works without supplying the gradient. As far as I know, the jacobian is defined to be the Vector of first derivatives of the function (typically as a row vector). So this should have Dimension 1x2 in my example (which it does have). The description says it should have the same Dimension as Theta. However, if I try this, it still does not work.
I would be happy to hear hear your thoughts on this. Many thanks in Advance!