Numpy.polyfit with regularization

Question

I am trying to use the numpy polyfit method to add regularization to my solution. My non-regularized solution is

coefficients = np.polynomial.polynomial.polyfit(x,y,5)
ypred = np.polynomial.polynomial.polyval(x,coefficients)

How would I modify this to add L2-regularization?

Answer 1

Here is a piece of code I developed in order to fit polinomials using regularization:

/* x_tr and y_tr is trainig data, x_te and y_te test data */
degree = 4
identity_size = degree + 1
x_tilda = np.array([x_tr, np.ones(len(x_tr))]).transpose()


for degree in range(2, degree + 1):
    x_tilda = np.append(np.array(np.power(x_tr, degree))[np.newaxis].transpose(), x_tilda, axis=1)

y_reg = y_tr

identidad = np.identity(identity_size)

def get_theta(rho):
    mul1 = np.linalg.inv(
        np.add(
            np.matmul(x_tilda.transpose(), x_tilda), 
            np.dot(identidad, rho)))
    mul2 = np.matmul(x_tilda.transpose(), y_reg)
    return np.matmul(mul1, mul2)

def poly(x, rho): 
    theta = get_theta(rho)
    print("Theta polyfit: ", np.polyfit(x_tr,y_reg, degree))
    print("Theta formula: ", get_theta(rho))

    p = np.poly1d(theta)
    return p(x)

rho = 0  // this is the regularization param
plt.figure()
plt.scatter(x_te,poly(x_te, rho), c ='r')
plt.scatter(x_te,y_te, c ='b')<code>

Above solution based on the following formula for polynomials (instead of X, it is used poly(X) for a general analytical solution with a squared-error loss.

$$ \hat{\beta} = (X^TX)^{-1}X^TY $$