The gist of this answer is that this problem fits into a general framework that is part of machine learning.
I think this can be done using the standard machinery of clustering / filtering problems in statistics, which are pretty standard. There's a lot of literature on this, so I'm going to present just the basic outline of how this can be done: it's easy to write down a basic model and then improve on it.
Let the set of observations be $O=\{o_n\}_n$, with each observation $o = (t, x)$ consisting of time and location. Let $\Lambda$ be the set of all objects' labels (fixed for now). We can write down the likelihood function for all the observations given a specific assignment of labels, and then try to find the assignment of object labels that maximizes log-likelihood.
This translates to finding that assignment of labels to observations that is the most likely under some statistical model of objects' behaviour.
The likelihood function is, given an assignment of labels $\{\lambda_n\}_n$, the p.d.f. of those observations given this assignment:
$$ L(\{\lambda_n\}) = \mathbb{P}(o_1,\ldots,o_n\mid \{\lambda_n\}). $$
We can try to factor this likelihood function as
$$ L(\{\lambda_n\}) = \prod_\lambda \mathbb{P}(o^\lambda_1) \mathbb{P}(o^\lambda_2\mid o^\lambda_1)\cdots\mathbb{P}(o^\lambda_k \mid o^\lambda_{k-1},o^\lambda_{k-2},\ldots,o^\lambda_1), $$
where the product goes over all assigned object labels $\lambda$, and each term is the likelihood of observing the same object $\lambda$ at different observations $o^\lambda_1,\ldots,o^\lambda_k$.
To write down the probabilities we can try to model their motion as constant-velocity motion, so that
$$ \mathbb{P}(o^\lambda_j \mid o^\lambda_{j-1},o^\lambda_{j-2},\ldots) $$
can be written in terms of $(t_j,x_j), (t_{j-1},x_{j-1}), (t_{j-2},x_{j-2})$ using (say) the normal distribution
$$ N\left(x_j \mid x_{j-1} + (t_j - t_{j-1})\frac{x_{j-1}-x_{j-2}}{t_{j-1}-t_{j-2}}, \xi \right). $$
This says that after observing object $\lambda$ at $o^\lambda_{j-2}$, then $o^\lambda_{j-1}$ the expected next location is at $x_{j-1}+\delta t\cdot v$, with some variance $\xi$. For the initial observations one might write down
$$ \mathbb{P}(o^\lambda_1) = \mathrm{const}, \qquad \mathbb{P}(o^\lambda_2|o^\lambda_1) = N(x_2 \mid x_1, (t_2-t_1)\eta), $$
where $\mathrm{const}$ is a p.d.f. (up to a multiplicative constant that will drop out later) that is like a penalty for introducing a new object label, and where $\eta$ represents uncertainty in how much an object might move in one time step.
Once this is all done, $-\log L$ becomes a sum of squared errors (although a bit laborious to write out in full) in the observations given an assignment of labels. This sum of squared errors (negative log-likelihood) can be minimized by a simple iterative process: walk through observations in chronological order, and for each next observation pick a label (existing or new) that results in the smallest sum of squared errors over observations so far.
This might already be good enough. If not, you might consider equipping each observation with a hidden state, such as $(t, x, v, a)$ where only $t$ and $x$ are directly observed, and each object also has velocity $v$ and acceleration $a$ that need to be inferred. You might also have to consider more carefully what the optimal assignment of labels is (the above strategy is a greedy assignment; a common method is the EM algorithm), and how to best determine the total number of objects (e.g., introduce a penalty into $-\log L$ for the total number of objects).
What I described here is a kind of Bayesian filtering approach (this is too long already). But this is a pretty standard problem: you have a hidden state (assignment of labels, velocities, etc.), and a statistical model of how that hidden state produces the observations that you have (the likelihood function), and you are trying to infer the hidden state from the observations (by maximization of log-likelihood). Textbooks that discuss machine learning, Bayesian inference, graphical models would be helpful here.
