The matrix math in this paper is terribly hard to follow, but here's what equation (19) should look like
$\left(
\begin{array}{c|c|c|c}
c_{11} \mathbf{I} - \mathbf{A}^T & c_{12} \mathbf{I} & c_{12} \mathbf{I} & \mathbf{0}^{3\times 3} \\
\hline
c_{21} \mathbf{I} & c_{22} \mathbf{I} - \mathbf{A}^T & c_{23} \mathbf{I} & \mathbf{0}^{3\times 3} \\
\hline
c_{31} \mathbf{I} & c_{32} \mathbf{I} & c_{33} \mathbf{I} - \mathbf{A}^T & \mathbf{0}^{3\times 3} \\
\hline
d_{11} \mathbf{I} - \mathbf{B}^T & d_{12} \mathbf{I} & d_{12} \mathbf{I} & \mathbf{0}^{3\times 3} \\
\hline
d_{21} \mathbf{I} & d_{22} \mathbf{I} - \mathbf{B}^T & d_{23} \mathbf{I} & \mathbf{0}^{3\times 3} \\
\hline
d_{31} \mathbf{I} & d_{32} \mathbf{I} & d_{33} \mathbf{I} - \mathbf{B}^T & \mathbf{0}^{3\times 3} \\
\hline
\mathbf{t}_A^T & \mathbf{0}^{1\times 3} & \mathbf{0}^{1\times 3} & \\
\mathbf{0}^{1\times 3} & \mathbf{t}_A^T & \mathbf{0}^{1\times 3} & \mathbf{I}-\mathbf{C}\\
\mathbf{0}^{1\times 3} & \mathbf{0}^{1\times 3} & \mathbf{t}_A^T & \\
\hline
\mathbf{t}_B^T & \mathbf{0}^{1\times 3} & \mathbf{0}^{1\times 3} & \\
\mathbf{0}^{1\times 3} & \mathbf{t}_B^T & \mathbf{0}^{1\times 3} & \mathbf{I}-\mathbf{D}\\
\mathbf{0}^{1\times 3} & \mathbf{0}^{1\times 3} & \mathbf{t}_B^T & \\
\end{array}
\right)
\left(
\begin{array}{c}
\mathbf{x}_1 \\
\hline
\mathbf{x}_2 \\
\hline
\mathbf{x}_3 \\
\hline
\mathbf{t}_x \\
\end{array}
\right)
=
\left(
\begin{array}{c}
\mathbf{0}^{3\times 1} \\
\hline
\mathbf{0}^{3\times 1} \\
\hline
\mathbf{0}^{3\times 1} \\
\hline
\mathbf{0}^{3\times 1} \\
\hline
\mathbf{0}^{3\times 1} \\
\hline
\mathbf{0}^{3\times 1} \\
\hline
\\
\mathbf{t}_C \\
\\
\hline
\\
\mathbf{t}_D \\
\\
\end{array}
\right)$
I've used dividing lines to help with alignment. Each block within lines in the matrix is $3 \times 3$. I've marked zero blocks with superscripts to indicate their size.
The following matrices are dimension $3 \times 3$: $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\mathbf{D}$, and $\mathbf{I}$.
The following vectors are dimension $3 \times 1$: $\mathbf{x}_1$, $\mathbf{x}_2$, $\mathbf{x}_3$, $\mathbf{t}_x$, $\mathbf{t}_A$, $\mathbf{t}_B$, $\mathbf{t}_C$, and $\mathbf{t}_D$.
The following are scalars: $c_{ij}$, $d_{ij}$.
This makes the whole system
$\mathbf{N}^{24 \times 12} \mathbf{x}^{12 \times 1} = \mathbf{b}^{24 \times 1}$
The most obvious way to solve this in the least squares sense.
$\mathbf{N}\mathbf{x} = \mathbf{b}$
$\mathbf{N}^T\mathbf{N}\mathbf{x} = \mathbf{N}^T\mathbf{b}$
$\mathbf{x} = \left( \mathbf{N}^T\mathbf{N}\right)^{-1}\mathbf{N}^T\mathbf{b}$
It's not explicitly said in the paper that this is what they did, but it's possible.