How do I correctly multiply vectors and matrices in Python and MATLAB?

Question

I have been trying for 2-3 days now to get L2 regularized logistric regression to work in Matlab (CVX) and Python(CVXPY) but no success. I am fairly new to convex optimization so I am quite frustrated. Following is the equation that I am trying to solve using CVX/CVXPY. I have taken this equation from the paper https://intentmedia.github.io/assets/2013-10-09-presenting-at-ieee-big-data/pld_js_ieee_bigdata_2013_admm.pdf

In the case of L2 regularized logistic regression the problem becomes: $$ \text{minimize} \frac{1}{m}\sum_{i=1}^{m}\log[1 + \exp(-b_i\mathbf{A}_i^Tx)] + \lambda\Vert x\Vert_2^2$$ where $\lambda$ is the regularization factor.

My Matlab (CVX) code is

function L2
m = 800; N = 5;
lambda =0.000001;

A = load('/path/to/training/file'); 
b= A(:,6); //Label Matrix (800x1)
A = A(:,1:5); //Feature matrix (800x5)

cvx_begin
    variable x(N)
    minimize( (1/m * sum( log(1+ exp(-1* A' * (b * x')) ) ) ) + lambda*(norm(x,2)))

cvx_end

CVX returns an error saying which makes sense but the paper mentions the above equation. How can I solve it ?

Your objective function is not a scalar.

After trying on Matlab, I tried on CVXPY. Here is the python code

from cvxopt import solvers, matrix,log, exp,mul
from cvxopt.modeling import op,variable
import numpy as np

n = 5
m=800
data = np.ndarray(shape=(m,n), dtype=float,)
bArray = []

file = open('/path/to/training/file')

i = 0;
j=0;
for line in file:
    for num in line.split():
        if(j==5):
            bArray.append(float(num))
        else:
            data[i][j] = num
            j = j + 1

    j=0
    i = i + 1

A = matrix(data)
b_mat= matrix(bArray)
m, n = A.size


lamb_default = 0.000001

x=variable(n)

b = -1*b_mat
w = exp(A.T*b*x)
f = (1/m) + sum(log(1+w)) + lamb_default*mul(x,x)

lp1 = op(f)
lp1.solve()
lp1.status
print(lp1.objective.value())

I get the error

TypeError: incompatible dimensions

So, my question is: What am I doing wrong in the code for calculation of L2 problem in CVX/CVXPY ?

Answer 1

David Ketcheson has already indicated the problem in his comment. I will flesh it out here.

Note the form of the argument of the log in the logistic regression objective:

$log[1+exp(−b_iA^T_ix)]$

where $b_i$ and $x$ are vectors and $A_i$ is a matrix. Only in the above order of matrix-vector multiplications will you get a scalar as an exponent.

The Matlab code should be (thanks for the correction OP):

log(1 + exp(-1*b.*(A * x)))      

This will be a vector which will work with the following sum:

sum(log(1 + exp(-1*b.*(A * x))))

The Python code could be (updated after OP's comment):

log_vec = [log(1+ exp(b[i]*A[i,:].T*x)) for i in range(n)] 
f = (1/m) + sum(log_vec) + lamb_default*mul(x,x)

This list comprehension will work for numpy matrix type. Make sure it works for matrix type of cvxopt.