I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in the software so I am turning to math possibly improve my algorithm to my problem. I have data, lots of data! I am collecting weather data for 30 days and each day contains just over 100,000 variables. The variables can reoccur on the other days or not. In total there are 12 billion different variables which can occur. So on each day only 100,000 of these 12 billion occur, but again these can reoccur on any other day or not. I am trying to see if say varA occurs varH occured 20 times in a 30 day period, or when varB occurs varZ occured 12 times in those 30 days. So currently my approach is running a nested loop through each day and counting each entry in a master count table.
Ex Master count
varA-varA : count varA-varB : count varA-varC : count . . . varZ-VarX : count varZ-VarY : count varZ-VarZ : count
I have been thinking of different ways to solve this. One approach I thought of is to break each day into a set, then extract the similarities between them somehow. Is there a method to count combinations in sets quickly?
Sorry, I use the word 'variable' not in the right context, weather event might be more accurate. The events do not have any values associated with them, they are descriptive. For example an event can be "HotWeatherAtLong22Lat48" or "LowPressureAtLong65Lat78". I have 12 billion events and only 10^5 of these events can on a single day. Some of these events might not occur at all in the 30 day period, some might occur on all 30 days, some might occur only on some of days, it is what seems quite random, but I would like to find the number of times a pair of events occur together within the 30 days –