I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) A PCA will take a 64-dimensional data set (say) and might return 4 dimensions which explain 95% of the variance. From a 4-point vector in the new space and the covariance matrix you could recover the original record with pretty good fidelity in this example.
So what about the case where x has 64 dimensions, and the data have the form (x, y) where y is a measurement? Is there something similar, that will let me predict y from a lower dimensional set of inputs? I.e., y $\approx$ g(x'), where dim(x') < dim(x), and there is a map m: x' $\mapsto$ x. (y' = g(x') and another map p: y' $\mapsto$ y is fine too.)
Often what people do in this situation is choose a model such as a GLM and do a partial regression, eliminating those dimensions where the regression coefficients are close to 0. However I'm looking for a method which does not assume any kind of model at all. Do these exist?
The image below illustrates an extreme case of this. Nominally the data is 2-dimensional, but the y-dimension is just a bit of 0-mean noise--z can be estimated well enough on the basis of x alone. So I'm wondering if there's a technique which will derive this, here but also cases where the redundancy is less obvious.