Applying same feature selection to multiple data sets with Weka

Question

I am using the Weka workbench to train a protein fold classifier. I imported my training data into Weka and performed PCA-based feature selection. This seems to have worked fine, but now I cannot evaluate my trained classifier on the test data because the test data contains all the original attributes. Of course, if I try to run the feature selection on the test data, I will come up with a different set of features.

In Weka, after you have applied feature selection to a training set, how do you pull those same features out of a test set?

Answer 1

With any such modelling thing, you are going to have to recalculate the model using the new training set (ie. the original minus its test set).

The usual approach is to randomly extract a subset for testing. Then train using all of the remaining data points.

Of course there will be some random variability according to which are extracted, so you can repeat the process to get some statistical significance.

Your final model will not be trained on all of your data. You either have to live with that (what I usually see in the Natural Language Processing field), or once you have determined your best parameters compute the final model using all of your data - with the understand you won't be able to test it.

Answer 2

It appears that the PCA-based feature selection generates pseudo-features that are linear combinations of the original features. After performing feature selection on the training set, I have not found a way to pull out the same pseudo-features from the test set.

However, if you're not tied to a particular feature selection method (PCA in this case), you can alternatively use a feature selection method that is more straightforward. For example, I ended up using a method that sorts the features according to information gain. Using this method, it is easy to identify the top n features that give the best information gain and extract only those features from the training and test data.

As suggested in the comments, there may be additional alternatives implemented in different languages (which I would be willing to consider, given a more detailed response), but this is the best I could find in the Weka environment.