I am working on a paper part of which is the application of validation rules to find how many principal components give us the least generalization error.
The concept goes more or less like this: "Given that the dimension of the model is reduced, we reset our window size to 60 days to avoid overfitting problem. After running multivariate linear regression within the first 20 components using 60 days training set, we find that the first 12 components give the smallest generalization error on the 30 testing days. Finally, we compute the in-sample and out-of-sample residuals."
Note that, the dimension of the full principal components matrix is 483 (days) X 482 (values).
The implementation is trivial indeed, yet I am strangling on how to "choose" the correct sub-matrices for any of these cases.
All suggestions (including matlab code) are welcome.
EDIT: Using the hint of cross validation as Arnold Neumaier mentioned below, and provided that I have already implemented the function that splits the initial dataset into parts, does the following solution solve partially the problem? What should I do next?
for i=1:1:10 training_set = ex1_data_txt(find(split_assignments(:,i)==0),:); test_set = ex1_data_txt(find(split_assignments(:,i)==1),:); % determine weights from the training set phi_train=[training_set(:,1).^(0) training_set(:,1).^(1)]; w=pinv(phi_train)*training_set(:,2); phi_test=[test_set(:,1).^(0) test_set(:,1).^(1)]; % apply learned weights to the test set and compute MSE MSE(i)=sum((test_set(:,2)-phi_test*w).^2)/size(test_set,1); end