Long time back I had a chance to work on dimensionality reduction which deals with processing data that falls in large sets. The basic idea behind it is that it processes the data through a few steps to orient it in the way that most information can be calculated from it.
It works pretty fine for matrices as well and is used largely. The best part is that you need not even program it as there are standard libraries available for it already.Major mathematical tools like Matlab and Mathematica also support this functionality straight forwardly.
There are two main algorithms that achieve this - Principal Component Analysis and Singular Value Decomposition.
What these algorithms actually achieve is finding that data that actually affects your reading by significant margin.Internet is full with information on these algorithms.The next link will show you how Apache is doing it. Apache Link

Long time back I had a chance to work on dimensionality reduction which deals with processing data that falls in large sets. The basic idea behind it is that it processes the data through a few steps to orient it in the way that most information can be calculated from it.

It works pretty fine for matrices as well and is used largely. The best part is that you need not even program it as there are standard libraries available for it already. Major mathematical tools like Matlab and Mathematica also support this functionality straightforwardly.

There are two main algorithms that achieve this - Principal Component Analysis and Singular Value Decomposition.

What these algorithms actually achieve is finding that data that actually affects your reading by a significant margin. The internet is full of information on these algorithms. This will show you how Apache is doing it.