Like with any other thing you want to test, you need to come up with a list of situations where you know that something is true, and then check this. In many cases, knowing that something is true does not require being able to actually compute the numerically correct answer, which simplifies things.
For example:
- As @BlaisB suggestes, if you rotate the reference cell, the determinant of the Jacobian needs to be one. You don't need to know the exact Jacobian matrix.
- The same is true if you translate the cell.
- If you scale the cell by a factor, the Jacobian needs to be a multiple of the identity matrix. The multiplication factor is simply the scaling factor.
- If you randomly generate cells that are (i) convex, and (ii) whose vertices are oriented in the correct way, the determinant needs to have a positive determinant.
- If you evaluate the determinant at multiple points of an affine triangle, it needs to be the same everywhere.
- If you compute $\int_T 1 \; dx$ via quadrature, then the result of the quadrature needs to be equal to the area/volume of the cell. Because the area is something that's easy to compute, it's easy to assess that the answer you get is correct up to round-off.
- ...
All of these make for good, individual testcases that are relatively simple to write. Within an hour, I could probably come up with a dozen or more such testcases.