You don't need the inverse even with the goal of finding $K^{-1} h h^{T} K^{-1} - K^{-1}$. If you are interested to have this expression, I would explain how you can convert it to a matrix equation and then solve it more efficiently:
Let's define the $X$ as:
$$X = K^{-1} h h^{T} K^{-1} - K^{-1}$$
Your objective is to calculate $X$ in this equation by assuming that $K$, $h$, and $h^{T}$ are all known:
$$KXK = hh^{T}-K$$
I define:
$$B = hh^{T} - K$$
By using $ \mathrm{vec}$ operator this equation would be transformed to a standard linear equation as:
$$(K^{T} \otimes K) \mathrm{vec}(X) = \mathrm{vec}(B)$$
Define: $K^{T} \otimes K = A$
Finally:
$$A \mathrm{vec}(X) = \mathrm{vec}(B)$$
There are numerous efficient linear solver and as you mentioned in the comment that the dimension of $K$ is not that high, it should be fairly efficient.
Update:
This Python code compare the proposed method with direct way to calculate $X$:
import numpy as np
d = 10
K = np.zeros((d,d))
alpha = 1
gamma = 10
t_vec = np.linspace(-3,3,d)
for i in range(d):
for j in range(d):
K[i][j] = np.exp(-alpha*(t_vec[i]**2)-alpha*(t_vec[j]**2)-gamma*((t_vec[i]-t_vec[j])**2))
A = np.kron(K.T,K)
h = np.random.rand(d,d)
B = np.matmul(h,h.T) - K
vecB = B.flatten(order='F')
vecX = np.linalg.solve(A,vecB)
X = vecX.reshape((d,d))
print X
Kinv = np.linalg.inv(K)
X_direct = np.matmul(np.matmul(Kinv,np.matmul(h,h.T)),Kinv) - Kinv
print X_direct
print np.abs(X-X_direct)
The outputs are:
X
:
[[1.59909954e+16 1.82624715e+12 4.24902134e+10 4.53728217e+08
2.05584992e+08 2.44120203e+08 5.98264287e+08 3.43515584e+10
3.12705250e+12 1.31655477e+16]
[1.82624715e+12 5.09588697e+09 4.78071076e+06 2.16534890e+05
9.59540287e+04 2.92820805e+04 2.33641869e+05 1.22037867e+07
2.91201539e+09 3.74298362e+12]
[4.24902134e+10 4.78071076e+06 1.65187655e+05 2.15188820e+03
8.22687923e+02 8.22528897e+02 2.14605722e+03 1.18784215e+05
1.27880035e+07 3.05868993e+10]
[4.53728217e+08 2.16534890e+05 2.15188820e+03 3.48340497e+01
1.33228000e+01 1.04249016e+01 3.45272712e+01 2.08247428e+03
2.62962350e+05 3.37679580e+08]
[2.05584992e+08 9.59540287e+04 8.22687923e+02 1.33228000e+01
4.33655487e+00 4.31336472e+00 1.42180232e+01 7.12777365e+02
9.01300211e+04 2.01206353e+08]
[2.44120203e+08 2.92820805e+04 8.22528897e+02 1.04249016e+01
4.31336472e+00 4.81918386e+00 1.18107131e+01 7.69453357e+02
7.36892858e+04 2.10911516e+08]
[5.98264287e+08 2.33641869e+05 2.14605722e+03 3.45272712e+01
1.42180232e+01 1.18107131e+01 4.00277702e+01 1.87646704e+03
1.95000703e+05 5.95359066e+08]
[3.43515584e+10 1.22037867e+07 1.18784215e+05 2.08247428e+03
7.12777365e+02 7.69453357e+02 1.87646704e+03 1.41229229e+05
1.68594796e+07 2.65035020e+10]
[3.12705250e+12 2.91201539e+09 1.27880035e+07 2.62962350e+05
9.01300211e+04 7.36892858e+04 1.95000703e+05 1.68594796e+07
3.77226961e+09 2.08638514e+12]
[1.31655477e+16 3.74298362e+12 3.05868993e+10 3.37679580e+08
2.01206353e+08 2.10911516e+08 5.95359066e+08 2.65035020e+10
2.08638514e+12 1.53843211e+16]]
X_direct
:
[[1.59909954e+16 1.82624715e+12 4.24902134e+10 4.53728217e+08
2.05584992e+08 2.44120203e+08 5.98264287e+08 3.43515584e+10
3.12705250e+12 1.31655477e+16]
[1.82624715e+12 5.09588697e+09 4.78071076e+06 2.16534890e+05
9.59540287e+04 2.92820805e+04 2.33641869e+05 1.22037867e+07
2.91201539e+09 3.74298362e+12]
[4.24902134e+10 4.78071076e+06 1.65187655e+05 2.15188820e+03
8.22687923e+02 8.22528897e+02 2.14605722e+03 1.18784215e+05
1.27880035e+07 3.05868993e+10]
[4.53728217e+08 2.16534890e+05 2.15188820e+03 3.48340497e+01
1.33228000e+01 1.04249016e+01 3.45272712e+01 2.08247428e+03
2.62962350e+05 3.37679580e+08]
[2.05584992e+08 9.59540287e+04 8.22687923e+02 1.33228000e+01
4.33655487e+00 4.31336472e+00 1.42180232e+01 7.12777365e+02
9.01300211e+04 2.01206353e+08]
[2.44120203e+08 2.92820805e+04 8.22528897e+02 1.04249016e+01
4.31336472e+00 4.81918386e+00 1.18107131e+01 7.69453357e+02
7.36892858e+04 2.10911516e+08]
[5.98264287e+08 2.33641869e+05 2.14605722e+03 3.45272712e+01
1.42180232e+01 1.18107131e+01 4.00277702e+01 1.87646704e+03
1.95000703e+05 5.95359066e+08]
[3.43515584e+10 1.22037867e+07 1.18784215e+05 2.08247428e+03
7.12777365e+02 7.69453357e+02 1.87646704e+03 1.41229229e+05
1.68594796e+07 2.65035020e+10]
[3.12705250e+12 2.91201539e+09 1.27880035e+07 2.62962350e+05
9.01300211e+04 7.36892858e+04 1.95000703e+05 1.68594796e+07
3.77226961e+09 2.08638514e+12]
[1.31655477e+16 3.74298362e+12 3.05868993e+10 3.37679580e+08
2.01206353e+08 2.10911516e+08 5.95359066e+08 2.65035020e+10
2.08638514e+12 1.53843211e+16]]
np.abs(X - X_direct)
:
[[4.00000000e+00 4.88281250e-04 1.52587891e-05 0.00000000e+00
2.98023224e-08 0.00000000e+00 0.00000000e+00 1.14440918e-05
4.88281250e-04 6.00000000e+00]
[1.46484375e-03 3.81469727e-06 1.02445483e-08 8.73114914e-11
2.91038305e-11 2.54658516e-11 2.91038305e-11 5.58793545e-09
9.53674316e-07 9.76562500e-04]
[0.00000000e+00 3.72529030e-09 5.82076609e-11 0.00000000e+00
3.41060513e-13 3.41060513e-13 9.09494702e-13 5.82076609e-11
1.86264515e-09 3.81469727e-06]
[5.96046448e-08 1.45519152e-10 4.54747351e-13 2.13162821e-14
7.10542736e-15 1.77635684e-15 1.42108547e-14 0.00000000e+00
1.16415322e-10 5.96046448e-08]
[2.98023224e-08 2.91038305e-11 1.13686838e-13 7.10542736e-15
1.77635684e-15 3.55271368e-15 3.55271368e-15 2.27373675e-13
4.36557457e-11 2.98023224e-08]
[1.19209290e-07 2.91038305e-11 2.27373675e-13 1.77635684e-15
4.44089210e-15 1.77635684e-15 1.77635684e-15 1.13686838e-13
4.36557457e-11 2.98023224e-08]
[0.00000000e+00 2.91038305e-11 4.54747351e-13 0.00000000e+00
3.55271368e-15 1.77635684e-15 7.10542736e-15 0.00000000e+00
5.82076609e-11 2.38418579e-07]
[7.62939453e-06 5.58793545e-09 1.45519152e-11 9.09494702e-13
1.13686838e-13 2.27373675e-13 4.54747351e-13 5.82076609e-11
3.72529030e-09 0.00000000e+00]
[0.00000000e+00 9.53674316e-07 3.72529030e-09 1.16415322e-10
1.45519152e-11 4.36557457e-11 8.73114914e-11 3.72529030e-09
1.90734863e-06 2.44140625e-04]
[2.00000000e+00 4.88281250e-04 7.62939453e-06 1.19209290e-07
5.96046448e-08 2.98023224e-08 2.38418579e-07 3.81469727e-06
1.70898438e-03 2.00000000e+00]]
Which you can see the difference is pretty small and shows that the proposed method actually works.