I want to filter on 5000 stocks, each of which has 4 month or more worth of data (>= 500 data points each). my filtering criteria will be based on 8 calculated values from the data points. for example, moving average, standard deviation and values alike.
my tools of choice are python pandas, numpy and CUDA C++; please take this as the default setting which can be expanded on, especially on the C++ side, I believe I can expand on Linear Algebra packs but I am not very familiar with them.
my current design is to:
- read the data into pandas dataframe.
- pass one stock at a time to C++ to compute all filtering criteria values. here I plan to use CUDA to speed up the calculations for each criteria value. e.g. running moving average calculation parallel on CUDA cores.
- pass a list containing filter criteria values of each stock back to python.
- sorting the stocks in terms of the filter criteria values; each criterion will weigh differently when I do the sorting.
I am still exploring, I know there will be better design/strategy with different tool set, but with these tools as given, is there a better strategy? Can I use Linear Algebra to speed this even further (potentially less codes)?