Updating matrix diagonal with Woodbury matrix identity and maintaining numerical accuracy

Question

I have a dense matrix A and its corresponding inverse $A^{-1}$. The Woodbury matrix identity states:

$$ (A + UCV)^{-1} = A^{-1} - A^{-1}U(C^{-1} + VA^{-1}U)^{-1}VA^{-1} $$

I wish to perform small updates to the diagonal of $A$ and compute corresponding $A^{-1}$. The updates can actually be described as a constant $k$ multiplying an identity matrix: $kI$ and in my case $U=V^T$ where $V$ is a tall matrix with just ones and zeros that selects the desired diagonal positions of $A$ to modify.

Everything is fine so far, except $k$ needs to be quite large (e.g., $10^6$). I am noticing that for large $k$, changes made are not "reversible" without significant error. What I mean by reversible is performing two update steps, one with $kI$ on the original $A$ and another with $-kI$ on the updated matrix to recover the original $A^{-1}$. If $k$ is small enough, it works fine, but otherwise I run into serious accuracy issues and the resulting matrix does not match the initial one closely. For my target application, I need to be able to apply $k$ to diagonal elements or $-k$ at will, so my question is: Is there a clever way to do this which preserves numerical accuracy even for large values of $k$?

Note: I am using double precision for all values stored and for computation. Furthermore, $k$ is a real scalar, but the matrix $A$ is complex, although that should not really matter in this case.