# How can i solve this non-convex multi-variable optimization problem?

I want to solve the following optimization problem: $$\min_{A,B,X} \|Y-AX\|_F^2 + \lambda_1 \|Z-BX\|_F^2+ \lambda_2 \|B\|_F^2$$ $$s.t ~~x_{ij}~ \geq 0$$

in which, $Y$ and $Z$ are data matrices and are $d \times n$ and $l \times n$ (the corresponding columns of $Y$ and $Z$ are data samples), and $A,B,X$ are the parameter matrices and are $d \times k$, $l \times k$ and $k \times n$ respectively. $n$ is around 100 and $d,k$ are around 20.

I think this is called a multivariate optimization problem. I tried to solve it using a general solver like Matlab's "fmincon" and it found a good minimum point. I think it uses the sequential programming algorithm.

My question is how can i solve this optimization problem in a systematic way to find a good optimum point comparable to what the Matlab solver found?

In your case, the blocks would be $X$ and $(A,B)$. So in each iteration of the algorithm, you fix one block and update the other block by directly minimizing the objective with respect to the other block. If this is computationally intractable you can always linearize the objective. This is equivalent to just taking the gradient of the objective with respect to the non-fixed block and project it back to the feasible set.
In case of your problem, in the $X$ iterations, you fix $(A,B)$ and then update $X$ by a gradient step and then project it back to non-negative orthant to keep it feasible. Then, alternatively, you fix $X$ and update $(A,B)$ by a gradient step (or optimizing with $X$ fixed). You perform this update until convergence. Note that you need to be careful with the step-size in each of those steps, if you are using the gradient, in order to make sure you always reduce the objective by each update.