0
$\begingroup$

I have been trying for 2-3 days now to get L2 regularized logistric regression to work in Matlab (CVX) and Python(CVXPY) but no success. I am fairly new to convex optimization so I am quite frustrated. Following is the equation that I am trying to solve using CVX/CVXPY. I have taken this equation from the paper https://intentmedia.github.io/assets/2013-10-09-presenting-at-ieee-big-data/pld_js_ieee_bigdata_2013_admm.pdf

In the case of L2 regularized logistic regression the problem becomes: $$ \text{minimize} \frac{1}{m}\sum_{i=1}^{m}\log[1 + \exp(-b_i\mathbf{A}_i^Tx)] + \lambda\Vert x\Vert_2^2$$ where $\lambda$ is the regularization factor.

My Matlab (CVX) code is

function L2
m = 800; N = 5;
lambda =0.000001;

A = load('/path/to/training/file'); 
b= A(:,6); //Label Matrix (800x1)
A = A(:,1:5); //Feature matrix (800x5)

cvx_begin
    variable x(N)
    minimize( (1/m * sum( log(1+ exp(-1* A' * (b * x')) ) ) ) + lambda*(norm(x,2)))

cvx_end

CVX returns an error saying which makes sense but the paper mentions the above equation. How can I solve it ?

Your objective function is not a scalar.

After trying on Matlab, I tried on CVXPY. Here is the python code

from cvxopt import solvers, matrix,log, exp,mul
from cvxopt.modeling import op,variable
import numpy as np

n = 5
m=800
data = np.ndarray(shape=(m,n), dtype=float,)
bArray = []

file = open('/path/to/training/file')

i = 0;
j=0;
for line in file:
    for num in line.split():
        if(j==5):
            bArray.append(float(num))
        else:
            data[i][j] = num
            j = j + 1

    j=0
    i = i + 1

A = matrix(data)
b_mat= matrix(bArray)
m, n = A.size


lamb_default = 0.000001

x=variable(n)

b = -1*b_mat
w = exp(A.T*b*x)
f = (1/m) + sum(log(1+w)) + lamb_default*mul(x,x)

lp1 = op(f)
lp1.solve()
lp1.status
print(lp1.objective.value())

I get the error

TypeError: incompatible dimensions

So, my question is: What am I doing wrong in the code for calculation of L2 problem in CVX/CVXPY ?

$\endgroup$
  • $\begingroup$ In your MATLAB /CVX code, A'(bx') is a matrix rather than a scalar. You'll need to rewrite this to get the objective function that you want. $\endgroup$ – Brian Borchers May 16 '15 at 17:03
  • $\begingroup$ Yes, after searching on google I realized that I need to only out a scalar value. I am very new to convex and don't know how to rewrite :( $\endgroup$ – Behroz Sikander May 16 '15 at 17:07
  • 4
    $\begingroup$ It seems that you have misunderstood the linear algebra in the objective function you have written down. This has nothing to do with programming. The first thing to do is write down the dimensions of each matrix and vector involved. That will help you see what you've misunderstood. $\endgroup$ – David Ketcheson May 16 '15 at 17:32
  • $\begingroup$ @DavidKetcheson I actually did the dimension calculation on paper but whatever I did I was not getting a scaler value. The code that I posted was after the frustration :D that something might work :D. It is pretty bad on my part but still :D $\endgroup$ – Behroz Sikander May 17 '15 at 19:50
3
$\begingroup$

David Ketcheson has already indicated the problem in his comment. I will flesh it out here.

Note the form of the argument of the log in the logistic regression objective:

$log[1+exp(−b_iA^T_ix)]$

where $b_i$ and $x$ are vectors and $A_i$ is a matrix. Only in the above order of matrix-vector multiplications will you get a scalar as an exponent.

The Matlab code should be (thanks for the correction OP):

log(1 + exp(-1*b.*(A * x)))      

This will be a vector which will work with the following sum:

sum(log(1 + exp(-1*b.*(A * x))))

The Python code could be (updated after OP's comment):

log_vec = [log(1+ exp(b[i]*A[i,:].T*x)) for i in range(n)] 
f = (1/m) + sum(log_vec) + lamb_default*mul(x,x)

This list comprehension will work for numpy matrix type. Make sure it works for matrix type of cvxopt.

$\endgroup$
  • $\begingroup$ thanks for the answer. Python code didn't work and returned "TypeError: incompatible dimensions". Matlab code has a little problem and following fixed it "(1/m * sum( log(1+ exp(-b.* (A * x)) ) ) )" . See the ".*" in the equation. You can update your answer and i will mark it as correct. $\endgroup$ – Behroz Sikander May 17 '15 at 19:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.