How to find the number of principal components that lead to the smallest generalization error?

Question

I am working on a paper part of which is the application of validation rules to find how many principal components give us the least generalization error.

The concept goes more or less like this: "Given that the dimension of the model is reduced, we reset our window size to 60 days to avoid overfitting problem. After running multivariate linear regression within the first 20 components using 60 days training set, we find that the first 12 components give the smallest generalization error on the 30 testing days. Finally, we compute the in-sample and out-of-sample residuals."

Note that, the dimension of the full principal components matrix is 483 (days) X 482 (values).

The implementation is trivial indeed, yet I am strangling on how to "choose" the correct sub-matrices for any of these cases.

All suggestions (including matlab code) are welcome.

EDIT: Using the hint of cross validation as Arnold Neumaier mentioned below, and provided that I have already implemented the function that splits the initial dataset into parts, does the following solution solve partially the problem? What should I do next?

for i=1:1:10
  training_set = ex1_data_txt(find(split_assignments(:,i)==0),:);
  test_set = ex1_data_txt(find(split_assignments(:,i)==1),:);

  % determine weights from the training set
  phi_train=[training_set(:,1).^(0) training_set(:,1).^(1)];
  w=pinv(phi_train)*training_set(:,2);
  phi_test=[test_set(:,1).^(0) test_set(:,1).^(1)];
  % apply learned weights to the test set and compute MSE
  MSE(i)=sum((test_set(:,2)-phi_test*w).^2)/size(test_set,1);
end

Answer 1

By cross validation. Divide your data set into 5 random batches of about equal size. Apply the estimation procedure 5 times with always one batch left out, and work out for each choice of effective dimension the generalization error by testing it on the remainder. Average these generalization errors .(If you want to be cautios, consider tin place of the averages the averages plus $k$ times the standard deviations, for some $k$ that you feel addresses your level of risk aversion.)

This gives you a curve from which you can read off the answer.

If the curve has a pronounced minimum, it gives you the effective dimension.

If, instead, the curve has an essentially flat bottom region, it gives you an indifference interval of effective dimensions, and by Ockahm's razor, you'd choose the smallest value in the indifference interval as the effective dimension.

Afterwards, you can do with this effective dimension the PCA on the full data set, and can expect a similar generalization errro on unseen data. But of course only when the original data were representative of the unseed data.

Window sizes, etc. can be chosen along similar lines.