# fastest and most efficent way to count all combinations in many sets and sum them together

I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in the software so I am turning to math possibly improve my algorithm to my problem. I have data, lots of data! I am collecting weather data for 30 days and each day contains just over 100,000 variables. The variables can reoccur on the other days or not. In total there are 12 billion different variables which can occur. So on each day only 100,000 of these 12 billion occur, but again these can reoccur on any other day or not. I am trying to see if say varA occurs varH occured 20 times in a 30 day period, or when varB occurs varZ occured 12 times in those 30 days. So currently my approach is running a nested loop through each day and counting each entry in a master count table.

Ex Master count

varA-varA : count
varA-varB : count
varA-varC : count
.
.
.
varZ-VarX : count
varZ-VarY : count
varZ-VarZ : count


I have been thinking of different ways to solve this. One approach I thought of is to break each day into a set, then extract the similarities between them somehow. Is there a method to count combinations in sets quickly?

Sorry, I use the word 'variable' not in the right context, weather event might be more accurate. The events do not have any values associated with them, they are descriptive. For example an event can be "HotWeatherAtLong22Lat48" or "LowPressureAtLong65Lat78". I have 12 billion events and only 10^5 of these events can on a single day. Some of these events might not occur at all in the 30 day period, some might occur on all 30 days, some might occur only on some of days, it is what seems quite random, but I would like to find the number of times a pair of events occur together within the 30 days –

• I wonder if a sparse matrix storage format would be useful. Sep 29, 2014 at 14:51
• That's definitely not my field since it's more computer-sciencey, but at a quick glance this looks like a job for Bloom filters. Have you heard about them? Oct 2, 2014 at 6:56
• No I have not, but am reading about them right now. This seems VERY interesting. I will learn about it and try to implement it into code withing the next few day. Thanks for the reply! Oct 2, 2014 at 16:21