In your example,
[US,TS]=ordschur(U, T, [1 2 3]); will work.
The documentation is quite cryptic.
Given a vector
CLUSTERS of cluster indices, commensurate with E = EIG(T), and such
that all eigenvalues with the same CLUSTERS value form one cluster,
[US,TS] = ordschur(U,T,CLUSTERS) will sort the specified clusters in
descending order along the diagonal of TS, the cluster with highest
index appearing in the upper left corner.
Did you understand that? Neither did I, at first. :) Essentially
select can be a vector of indices:
select(i) tells you in which cluster the
ith eigenvalue needs to go. These clusters are sorted in decreasing order: if e.g.
select = [1 2 3 3 2 1], then the 3rd and 4th eigenvalue (in the order specified by the diagonal of
T) will come first (in an unspecified order), then the 2nd and 5th (in an unspecified order), then the 1st and last (in an unspecified order).
To sort the eigenvalues in decreasing order, you can use the fact that the second output of
sort is a vector of indices that specifies a permutation.
[U, T] = schur(H);
[~, select] = sort(ordeig(T));
[US, TS] = ordschur(U, T, select);
(the second output of
sort is a vector of indices that we can use for this purpose.
sort sorts increasingly by default, but then
ordschur sorts cluster numbers in descending order, so it magically works.)
[EDIT: this still does not work on all matrices; see @Jake's answer below for a corrected version.]