tl;dr: Use MATLAB primarily for numerical computation, not symbolic computation.
MATLAB was primarily designed as a numerical computation package, and its symbolic capabilities were added later. Also, as a general rule, it's easier to compute quantities with concrete values than it is to calculate the same quantities in the abstract. For instance, no closed form solutions exist for general polynomial equations of a single variable of degree 5 or greater. Given both of these trends, the expectation is that, for any given symbolic expression and MATLAB operation, MATLAB is unlikely to return a symbolic solution.
For the specific case of overdetermined linear systems, MATLAB backslash, as noted by DaveP, is doing least-squares, so it's doing a QR factorization, followed by a linear solve, probably by LU decomposition.
rref, on the other hand, is putting the system into reduced row echelon form.
In principle, the
rref algorithm is essentially Gaussian elimination, probably augmented by good choices of pivots for some numerical stability. Divisions are done by pivots, which probably makes symbolic computation relatively straightforward (even if the resulting expressions are beyond heinous).
QR decompositions can be computed with many algorithms (modified Gram-Schmidt, Givens rotations, Householder transformations). Modified Gram-Schmidt is probably the simplest, and involves norms and divisions.
rref is likely to yield simpler expressions, that is probably why
rref works, but backslash fails. In either case, trying to use these algorithms for symbolic manipulation is generally not a good strategy because these algorithms were designed primarily for numerical computation, not to grind through symbolic expressions. Could you use them for symbolic manipulation? Sure. Is it a good idea? Probably not.