A rather straight forward method would be to use the Inverse Power Method with $\mu = 0$.
This method will converge to the eigenvalue of A with the smallest magnitude. One thing you want to be careful of is if the smallest eigenvalue isn't very close to 0 in magnitude, or you have several with the same magnitude, all of which are the minimum, you could end up with slow convergence or oscillations. Usually these shortcomings could be dealt with by changing $\mu$ every few iterations, however this would require refactoring the matrix. If its just slow convergence, and you don't want to refactor the matrix, you can just throw more iterations at it. If there are multiple eigenvalues with the same magnitude, you will have to change $\mu$ and refactor atleast once, which may or may not be worth it.
Luckily, trying this, even with your own implementation is a matter of a small handful of lines of code, so you can try it, see how it converges, and see if it will work in your case.