Consider an ODE of the form:
$$d_t u(t) = \alpha u(t) + b(t), \quad b(t) = \exp(t\beta ).$$
By using an integrating factor of $\exp(-t\alpha)$ you can get the solution:
\begin{align} u(t) &= \exp(t\alpha)u(0) + \int_{0}^t\exp((t-s)\alpha)b(s)\,ds \\ &= \exp(t\alpha)u(0) + \exp(t\alpha)\int_{0}^t\exp((\beta-\alpha)s)\,ds. \end{align}
Set $\gamma = \beta-\alpha$, then if $\gamma = 0$ the solution is:
\begin{equation} u(t) = \exp(t\alpha)u(0) + \exp(t\alpha)\int_{0}^t\,ds = \exp(t\alpha)(u(0) + |t|) \end{equation}
Now let $\gamma\ne 0$, then:
\begin{align} \int_0^t\exp(\gamma s)\,ds &= \int_0^t\sum_{k=0}^{\infty}\frac{ s^k\gamma^k}{k!}\,ds = \operatorname{sign}(t)\sum_{k=0}^{\infty}\frac{t^{k+1}\gamma^k}{(k+1)!} \\ &=\operatorname{sign}(t)\gamma^{-1}\gamma\sum_{k=0}^{\infty}\frac{t^{k+1}\gamma^k}{(k+1)!} = \operatorname{sign}(t)\gamma^{-1}\sum_{k=1}^{\infty}\frac{t^k\gamma^k}{k!} \\ &= \operatorname{sign}(t)\gamma^{-1}(\exp(t\gamma)-1). \end{align}
Then if I didn't mess up something, the solution is: $$u(t) = \exp(t\alpha)(u(0) - \operatorname{sign}(t)\gamma^{-1}) + \exp(t\beta)\operatorname{sign}(t)\gamma^{-1}.$$ You don't need to do any time-stepping now, and you can analyze the numerical errors you can get in this expression.