If we had, for example, labeled data, where for each entry (label) we have several data distributions associated to it, how can I get something meaningful from them?
Is this a solvable problem? Is there a better way to solve it than just taking some parameters which describe the data (mean, standard deviation, etc.)? Do you have some example of this kind of problems, if they exist?
Your question is ill-posed.
Take just the data points corresponding to one label. I think that when you say "for each label, we have several data distributions associated to it" that what you really mean is that the data does not simply correspond to one simple probability distribution like a Gaussian, but actually to a superpositions of simple distributions -- e.g., a superposition of Gaussians. But a "superposition of Gaussian distributions" is just another "probability distribution": The points still correspond to one distribution, it's just not as simple any more.
So your question then is essentially this: If I have a set of points corresponding to some, possibly complex, probability distribution, how can I characterize this probability distribution?
If this is a one-dimensional variable, the way this is typically done is through a "kernel density estimate". This concept can be generalized to higher dimensions.
