# Algorithm design to filter on 5,000 stocks each of which has 4 months worth of data points

I want to filter on 5000 stocks, each of which has 4 month or more worth of data (>= 500 data points each). my filtering criteria will be based on 8 calculated values from the data points. for example, moving average, standard deviation and values alike.

my tools of choice are python pandas, numpy and CUDA C++; please take this as the default setting which can be expanded on, especially on the C++ side, I believe I can expand on Linear Algebra packs but I am not very familiar with them.

my current design is to:

1. read the data into pandas dataframe.
2. pass one stock at a time to C++ to compute all filtering criteria values. here I plan to use CUDA to speed up the calculations for each criteria value. e.g. running moving average calculation parallel on CUDA cores.
3. pass a list containing filter criteria values of each stock back to python.
4. sorting the stocks in terms of the filter criteria values; each criterion will weigh differently when I do the sorting.

I am still exploring, I know there will be better design/strategy with different tool set, but with these tools as given, is there a better strategy? Can I use Linear Algebra to speed this even further (potentially less codes)?

• This is frankly way too little data to deal with the complexity of CUDA or C++, especially if you're already using python. Pandas ought to be more than sufficient for the task, especially since it has much of the functionality you want built in. You're pretty vague on the definition of "filter on", but if you mean "create a binary value (or whatever) indicating whether an instrument is a candidate during portfolio selection", then pandas is the right tool for you. I suggest you look around for some basic pandas tutorials, as they will likely address precisely these sorts of operations. Oct 24 '17 at 13:17
• @TylerOlsen thanks Tyler. I guess when you said too little data to deal with the complexity, you meant it's probably not worth the complexity with such little data? I agree with your suggestion; I think for this little algorithm I am taking it more as practising CUDA rather than serious project. In terms of filtering, it's still very crude I only intended to calculate the 10 day moving average based on past data and sorting it in order. this of course doesn't need CUDA or C++ at all, I was thinking to use CUDA C++ to only do the actual calculation. Anything you could suggest in this sense? Oct 24 '17 at 13:39
• pandas.pydata.org/pandas-docs/stable/generated/… Oct 24 '17 at 13:57
• @TylerOlsen ok.. it makes perfect sense to use just python given these time series analysis feature that pandas has. I'll do both I guess with CUDA as a big stretch. thanks a lot. Oct 24 '17 at 14:43