This question is mainly about hardware, but also about software.
In my current work I have approximately 68 millions of combinations that I am iterating through, in parallel. For each of those combinations - there is processing involved on the same data set. The idea is very similar to how neural networks train, I guess.
My current problem is, that it takes 10 hours to process all combinations. I am using the C#
and the code can be simplified to this:
long combinationCount = 68,211,200; // (68m)
List<Data> data = new List<Data>();
long dataCount = data.Count(); // 65,800 (65k)
// itterate every combination
Parallel.For(0, combinationCount, new ParallelOptions { MaxDegreeOfParallelism = 4}, (index, loopState) => {
for(int i = 0; i < dataCount; i++) {
// proccess each data record
}
});
My current hardware is an HP laptop with processor i7-6500U
(2 cores, 4 threads). Quite old perhaps not the best choice for these kinds of tasks.
So my question is - how do I optimize and improve the performance of my applications? To be more precise - I would like to hear out ideas on following topics:
CPU
. Since I am planning on purchasing a stationary PC - would it be best to make a choice in flavor of:AMD Ryzen 1700/1700x/1800x
- 8 cores, 16 threadsIntel's i7 8700/8700k
- 6 core, 12 threads, higher frequency per core
Please note! The question is not about AMD
or Intel
, but more about whether it is better to have 8 physical cores at a lower clock speed than, a CPU with 6 cores, but a higher clock speed. Specifically for programming.
GPU
. It has come to my attention, that you can leverage calculations toGPU
cores instead ofCPU
. This in theory will increase the performance X-fold. The issue is that you need to use a lower-level programming language such asC++
to make advantage of that. SinceC#
is the only toolset in my possession - I was wondering whether it is worth investing into a betterVideo card
instead on anCPU
?
I have googled a bit and found a project altimesh Hybridizer
which seems to allow to code on C#
and compile it in such a way that a GPU
will run it. The idea is neat. If the above is true - then would that make More sense to purchase a higher-end GPU instead? Overall, how much of a performance increase can you expect from calculations being executed on GPU (CUDA cores?) as opposed to 6/8 core CPU? If it matters - I am looking at a GTX 1060 3GB/6GB
price range video card.
And again. I am sorry if this is not the best place to ask such question. My main concern is building a system that would sustain nowadays requirements for computations, be it AI/neural networks/ or huge data processing, as in my case
Computer science
toComputational Science
. I have updated a title a bit so it would be more descriptive. $\endgroup$