It seems that both $x$ and $y$ have an offset, since a Lissajous curve should oscillate around the origin, so: $x=A\sin{\left(at+\delta\right)}+x_0$ and $y=B\sin{\left(bt\right)}+y_0$.
Besides that, your test equations do not seem to be correct. Because the ratio $\frac{a}{b}$ determines the number of "lobes" (points where the curve crosses itself) of the cyclic curve. When assuming that the data contains noise/uncertainty and that at least one complete oscillation has been complete. It seems that there are no lobes, which means that $a=b$ and that there would be only be a phase shift between $x$ and $y$. However this should produce an elliptical curve, which does not seems to be the case. This curve does look similar to a hysteresis curve.
Edit:
But to answer how you might fit data to a Lissajous curve we first have to look at why you can't calculate the vector $t$. The inverse of the sine function, $\sin^{-1}$, has actually another solution beside the one you mentioned: $\sin{x}=a\rightarrow x=\sin^{-1}{a}+2\pi n\bigvee x=-\sin^{-1}{a}+2\pi+\pi$. These two slopes can be seen by plotting $\sin^{-1}\left(\sin{(x)}\right)$, which gives a triangle wave. However the absolute value of the slopes of both data sets should be the same, so for most data points, using index $i$, the following should be true:
$$
\left|\frac{d}{di}t(i)\right|=\left|\frac{d}{di}\frac{1}{a}\left(\sin^{-1}\left(\frac{x(i)-x_0}{A}\right)-\delta\right)\right|
$$
Instead of taking the absolute value, you could also square both sides.
However $t$ is still unknown, but this should also hold for $y$, so:
$$
\left|\frac{d}{di}\frac{b}{a}\left(\sin^{-1}\left(\frac{x(i)-x_0}{A}\right)-\delta\right)\right|-\left|\frac{d}{di}\sin^{-1}\left(\frac{y(i)-y_0}{B}\right)\right|=0
$$
However I could only think of one solution to be able to use $\frac{d}{di}$ while fitting in MATLAB. And that is taking the equivalent of diff(x)
, namely splitting $x$ up into two vectors: x1 = x(1:end-1); x2 = x(2:end);
. So the custom equation would look like this:
abs(b / a * (asin(x1 / A) - asin(x2 / A))) - abs(asin(y1 / B) - asin(y2 / B)) = 0
This method loses information about $\delta$, but once you know all the other parameters it would be much easier to find $\delta$ with another method. Another downside of this approach is that you will end up with a total of 5 data sets, x1
, x2
, y1
, y2
and a vector of zeros. However MATLAB cftool can only handle 3 data inputs, so I am not sure if you could perform a fit using cftool.
However according to this report it is possible to fit a Lissajous curve using a least squares algorithm.
This report might also be interesting, since it is about least square fitting of ellipses (so $a=b$). I have only taken a quick look at it, so I am not 100% sure how useful it would be to you.