Dimensionality reduction of the domain of f(x)

Question

I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) A PCA will take a 64-dimensional data set (say) and might return 4 dimensions which explain 95% of the variance. From a 4-point vector in the new space and the covariance matrix you could recover the original record with pretty good fidelity in this example.

So what about the case where x has 64 dimensions, and the data have the form (x, y) where y is a measurement? Is there something similar, that will let me predict y from a lower dimensional set of inputs? I.e., y $\approx$ g(x'), where dim(x') < dim(x), and there is a map m: x' $\mapsto$ x. (y' = g(x') and another map p: y' $\mapsto$ y is fine too.)

Often what people do in this situation is choose a model such as a GLM and do a partial regression, eliminating those dimensions where the regression coefficients are close to 0. However I'm looking for a method which does not assume any kind of model at all. Do these exist?

The image below illustrates an extreme case of this. Nominally the data is 2-dimensional, but the y-dimension is just a bit of 0-mean noise--z can be estimated well enough on the basis of x alone. So I'm wondering if there's a technique which will derive this, here but also cases where the redundancy is less obvious.

Answer 1

Take a look at active subspaces, e.g., Active Subspace Methods in Theory and Practice: http://epubs.siam.org/doi/abs/10.1137/130916138

And a PDF here: http://inside.mines.edu/~pconstan/docs/constantine-asm.pdf

I have a SIAM book (Active Subspaces: Emerging Ideas for Dimension Reduction in Parameter Studies) coming out in March.

Suppose $f$ maps $\mathbb{R}^m$ to $\mathbb{R}$. One model that addresses what you're looking for is

$$ f(x) \approx g(W^T x), $$

where $W$ is $m\times n$, and $g$ is a function that maps $\mathbb{R}^n$ to $\mathbb{R}$. In active subspaces, $W$ is the first $n$ eigenvectors of the symmetric, positive semidefinite matrix,

$$ C = \int (\nabla f)(\nabla f)^T\rho\,dx, $$

where $\rho=\rho(x)$ is a given density function on the domain. The paper I linked to discusses how to estimate $W$ and how to construct $g$---along with error estimates.

Here's a video of the process of exploiting active subspaces for constructing response surfaces (i.e., surrogate models or metamodels): https://www.youtube.com/watch?v=mJvKzjT6lmY

However, in the figure you show, the function is noisy, so the exact gradient might not be the best choice; those eigenvectors will pick up directions associated with the high frequency noise. In the statistics literature, your problem goes by the name sufficient dimension reduction, which is set in the context of regression (i.e., supervised learning) with one set of variables designated as predictors (the independent variables) and the other set designated responses (the dependent variable). Take a look at Dennis Cook's book Regression Graphics, which is my favorite overview of the field. And yes, sufficient dimension reduction is an entire subfield of statistics.

EDIT: I updated the video link, the book is now published by SIAM, and we have a new active subspaces website.

Answer 2

I think it might be helpful to perform a tensor product spline interpolation or rather, approximation by least squares as the data is noisy, e.g., by using Matlab's Curve fitting toolbox, and see whether the coefficients along certain dimensions tend to zero, meaning it's a low (hopefully 0-) order polynomial along that dimension.