replace $A$ by a factored form from which the determinant can be computed stably and cheaply. E.g., add new triangular variables $L$ and $R$ and the constraint $A=PLR$ for some fixed permutation matrix $P$. Then determinants and their gradients are trivially to compute.
In case the solution has no such factorization, you need to change $P$ after a preliminary round of minimization with $P=I$.