My hunch is that you may be prescribing boundary data at both ends of the domain (which leads to oscillations) or incorrectly specifying the boundary data. For a hyperbolic problem, you need to respect the flow of information in your numerical method. For example, consider the linear hyperbolic equation
$$ \mathbf u_t = \mathbf A \mathbf u_x $$
with Dirichlet boundary conditions at the endpoints $x = x_L$ and $x = x_R$. For simplicity, we assume that $\mathbf A$ is a constant coefficient matrix with non-degenerate eigenvalues and linearly independent eigenvectors. By the spectral theorem we can write
$$ \mathbf A = \mathbf P \mathbf D \mathbf P^{-1} $$
where $\mathbf D$ is a diagonal matrix of eigenvalues and $\mathbf P$ is the respective eigenvector matrix. Using the decomposition, we can rewrite our PDE as
$$ \mathbf v = \mathbf P^{-1} \mathbf u $$
$$ \mathbf v_t = \mathbf D \mathbf v_x $$
The system is now decoupled and the sign of the eigenvalue will tell you the direction that the characteristics $\xi = x + \lambda t$ are propagating. Characteristics are lines where the solution is constant. For example, for $\lambda_k > 0$, characteristics propagate from left to right. Consider the $(x, t)$-plane of the characteristics. At the initial time, $t = 0$, the initial condition supplies the starting point for the emanating characteristics. For $t > 0$, the characteristics travel from left to right. If we trace any characteristic from the initial time, we can see that the characteristic will eventually exit the domain at $x_R$. If we try to trace any characteristic emanating from $x_L$, we run into the issue that there is no information provided at this boundary point, thus we must prescribe the boundary condition for $v_k$ at $x_L$. If we were to prescribe a boundary condition at $x_R$ as well, this will lead to numerical instability for we are over constraining our system by telling it how the characteristics should behave at the outflow boundary.
If you are implementing the uncoupled system, then the boundary data that you should impose should still be in terms of the characteristic variables. For example, when solving
$$ u_t = v_x $$
$$ v_t = u_x $$
for $-1 \leq x \leq 1$, and Dirichlet boundary conditions $u(-1,t) = f_1(t)$, $v(-1,t) = f_2(t)$, $u(1,t) = f_3(t)$, $u(1,t) = f_4(t)$, it is easy to see that the characteristic variables are $(u + v)$ and $(u - v)$:
$$ (u+v)_t = (u+v)_x $$
$$ (u-v)_t = -(u-v)_x $$
and that we need to impose an incoming characteristic for $(u+v)$ at $x=-1$:
$$u(-1,t) + v(-1,t) = f_1(t) + f_2(t) $$
and an incoming characteristic for $(u-v)$ at $x=1$:
$$u(1,t) - v(1,t) = f_3(t) + f_4(t) $$
