2
$\begingroup$

I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in the software so I am turning to math possibly improve my algorithm to my problem. I have data, lots of data! I am collecting weather data for 30 days and each day contains just over 100,000 variables. The variables can reoccur on the other days or not. In total there are 12 billion different variables which can occur. So on each day only 100,000 of these 12 billion occur, but again these can reoccur on any other day or not. I am trying to see if say varA occurs varH occured 20 times in a 30 day period, or when varB occurs varZ occured 12 times in those 30 days. So currently my approach is running a nested loop through each day and counting each entry in a master count table.

Ex Master count

varA-varA : count
varA-varB : count
varA-varC : count
.
.
.
varZ-VarX : count
varZ-VarY : count
varZ-VarZ : count 

I have been thinking of different ways to solve this. One approach I thought of is to break each day into a set, then extract the similarities between them somehow. Is there a method to count combinations in sets quickly?

Sorry, I use the word 'variable' not in the right context, weather event might be more accurate. The events do not have any values associated with them, they are descriptive. For example an event can be "HotWeatherAtLong22Lat48" or "LowPressureAtLong65Lat78". I have 12 billion events and only 10^5 of these events can on a single day. Some of these events might not occur at all in the 30 day period, some might occur on all 30 days, some might occur only on some of days, it is what seems quite random, but I would like to find the number of times a pair of events occur together within the 30 days –

$\endgroup$
  • 1
    $\begingroup$ I wonder if a sparse matrix storage format would be useful. $\endgroup$ – James Sep 29 '14 at 14:51
  • $\begingroup$ That's definitely not my field since it's more computer-sciencey, but at a quick glance this looks like a job for Bloom filters. Have you heard about them? $\endgroup$ – Federico Poloni Oct 2 '14 at 6:56
  • $\begingroup$ No I have not, but am reading about them right now. This seems VERY interesting. I will learn about it and try to implement it into code withing the next few day. Thanks for the reply! $\endgroup$ – user2924127 Oct 2 '14 at 16:21
1
$\begingroup$

If I understand your description correctly, then your problem falls into a class where there are a very large number of queries you could make, each of which is relatively cheap, but taken together they are too expensive.

A sensible strategy in that case would be not to precompute the answers to all possible queries and store them, but to only run a query when you actually need it. The question is what you are going to learn if you know the answers to all of the billions of queries you could possibly do. Presumably, there are questions you want to ask, and these would be associated with specific queries that you could run when you need to.

$\endgroup$
0
$\begingroup$

Just a few thoughts/suggestions:

  1. Could you aggregate groups/sets of similar data points (by day, weather type, location, etc.) and store those in separate/temporary database tables? Even if it takes a few days to do this massive conversion, you then should be able to query those resulting summary tables very quickly in a variety of different ways.

  2. Are the tables in your relational database indexed in any way? Could you create more indexes? Perhaps this could improve performance of your queries.

  3. You mention that the weather event data isn't numeric/scalar (for example, a weather event may be "HotWeatherAtLong22Lat48"). Is there any you for you to convert this into numeric/scalar data? For example, create new database columns for "Longitude", "Latitude", "Temperature", etc. This also might take a few days of crunching to convert all the data. But once it's converted, you should be able to run all kinds of blazingly-fast queries.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.