Consider an ODE of the form:

$$d_t u(t) = \alpha  u(t) + b(t), \quad b(t) = \exp(t\beta ).$$

By using an integrating factor of $\exp(-t\alpha)$ you can get the solution:

\begin{align}
u(t) &= \exp(t\alpha)u(0) + \int_{0}^t\exp((t-s)\alpha)b(s)\,ds \\
&= \exp(t\alpha)u(0) + \exp(t\alpha)\int_{0}^t\exp((\beta-\alpha)s)\,ds.
\end{align}

Set $\gamma = \beta-\alpha$, then if $\gamma = 0$ the solution is:

\begin{equation}
u(t) = \exp(t\alpha)u(0) + \exp(t\alpha)\int_{0}^t\,ds = \exp(t\alpha)(u(0) + |t|)
\end{equation}

Now let $\gamma\ne 0$, then:

\begin{align}
\int_0^t\exp(\gamma s)\,ds &= \int_0^t\sum_{k=0}^{\infty}\frac{ s^k\gamma^k}{k!}\,ds = \operatorname{sign}(t)\sum_{k=0}^{\infty}\frac{t^{k+1}\gamma^k}{(k+1)!} \\
&=\operatorname{sign}(t)\gamma^{-1}\gamma\sum_{k=0}^{\infty}\frac{t^{k+1}\gamma^k}{(k+1)!}
= \operatorname{sign}(t)\gamma^{-1}\sum_{k=1}^{\infty}\frac{t^k\gamma^k}{k!} \\
&= \operatorname{sign}(t)\gamma^{-1}(\exp(t\gamma)-1).
\end{align}

Then if I didn't mess up something, the solution is:
$$u(t) = \exp(t\alpha)(u(0) - \operatorname{sign}(t)\gamma^{-1}) + \exp(t\beta)\operatorname{sign}(t)\gamma^{-1}.$$
You don't need to do any time-stepping now, and you can analyze the numerical errors you can get in this expression.