1
$\begingroup$

I've may have reached a point in my neural network research that I cannot continue without significant financial investment.

I am using neuroevolution to evolve a network on the EMNIST data set. It is embarrassingly parallel (100 individuals * 116,323 records). The program is impossible to accelerate with a GPU. I have tried. This leaves OpenMP, which to my surprise is granting a speedup near-equal to the number of cores. However, even a 72-core Virtual Machine is projected to take 34 days (200 iterations, minimum), and set me back >$2,400.

MPI is on my list of changes to make to the program, but (assuming seamless parallelization) the cores and time required will not change. Several combinations of virtual machines and core-counts (Amazon, Azure, Google) have revealed a cost estimate of $1,500-$2,500. For 1 experiment. Assuming that ALL settings are correct, and I do not need to run further experiments.

I am a sophomore-level undergraduate. My university has NO computing clusters, nor access to any. The best they have are 8-core workstations (that would take a year to run the job). I don't have $1,500 to spare. Are there any resources (sponsors, grants, laboratories, etc) that would help me get the cores I need (cloud, or "home supercomputer")?

$\endgroup$

migrated from academia.stackexchange.com Feb 27 '18 at 23:32

This question came from our site for academics and those enrolled in higher education.

  • $\begingroup$ Do you have an advisor, are you part of a research group, or is this your own independent study? $\endgroup$ – nengel Feb 26 '18 at 4:54
  • $\begingroup$ I have an advisor, but this is ~99% me. $\endgroup$ – Derek Smith Feb 26 '18 at 4:55
  • 7
    $\begingroup$ Have you talked to him about this? If he has contacts or even a second appointment at another university it may be possible to access some of their resources less formally, but otherwise you probably have to write up some kind of proposal. $\endgroup$ – nengel Feb 26 '18 at 5:00
  • 1
    $\begingroup$ Do you need to SHOUT at us? $\endgroup$ – Najib Idrissi Feb 26 '18 at 10:19
  • 1
    $\begingroup$ The big problem I see here is that each experiment is expected to consume almost 60k CPU-hours. On Stampede2, 10 repeats of this experiment would consume roughly 10,000 SUs, or "a pretty good starter allocation." In other words, unless you only need to run one or two runs of this program, you're very unlikely to find anyone with those kinds of resources lying around for free. I was going to point to the Jetstream Cloud trial allocation, but those are pitifully small, unfortunately. $\endgroup$ – chipbuster Feb 27 '18 at 23:52
5
$\begingroup$

You need funding, you need a better algorithm, a better implementation, or you need to change your problem. To get funding, you need to work with your adviser. This could be about using grant money or getting a grant for an XSEDE allocation. If your adviser has neither of those, it may not be a good choice to work with them even if you like their research: science requires funding.

If you want to make it cheaper, then think about a different algorithm. Even if it's doing a lot of branching and stuff like that, there must be some way to utilize linear algebra operations in some parts to make use of GPU (or Xeon Phi) acceleration. Maybe you need to precompute some values, or need a wholly different approach. Only you (and your adviser) would know. In fact, this algorithm development is many times the entire purpose of the research. Your adviser may have given you this project because no simple computer can solve this problem. Your goal isn't to find a bigger computer, it's to find out how to make your computer solve this problem (or at least, find out how to solve it better). No online tutorial or anything can help you with this: it's real research.

You also need to take a look at your implementation. If you're looping and branching in a high level language like Python, this could be the problem. You either need to re-write the code to be vectorized to do a bit better, or to get "production-quality" speeds you need to re-write it in a non-allocating format in a performance language (one which generate strictly typed and interprocedurally optimized compiled code) like C++ or Julia. It's not hard to find non-vectorized Python codes which can be made 1000x faster just by doing a better implementation and being careful with memory use (even vectorization is bad with memory use so that still leaves a lot to optimize).

And lastly, if this is a highly researched area where it's well-known it cannot do any better and you cannot get the funds/allocations to run the algorithms, pick a different project. Sometimes problems currently aren't possible. I highly doubt that your adviser would give you a problem that is in this domain though given that you're an undergraduate.

However, given you're a younger undergraduate, I would double/triple/quadruple check that you are actually correct that this kind of computing power is even required before going to blame anyone else. Embarrassingly parallel and (100 individuals * 116,323 records) is quite small, even to train on a desktop computer without a GPU. You should really make sure everything is as fast as it can be (rewrite it a few times in a few different languages, benchmark and profile each part, get rid of all memory allocations in the inner loops, etc.), make sure there's no better algorithm choices, make sure there is no cluster allocation you can get on, and only then should you consider moving to a new project that is more suitable to your current access to computing power.

$\endgroup$
  • 4
    $\begingroup$ I have often thought I needed to do compute on a huge dataset, got frustrated with how long it was taking, then found out ~10 seconds of compute on a subset gave essentially the same results as 10 hours. I suspect my experience is not unique. $\endgroup$ – user14717 Feb 28 '18 at 6:02
  • $\begingroup$ There are two things I need to point out. [1] I proposed this research project, and my adviser has (at times) trouble following it. [2] The program is written in C++; the problem is that the program is founded on multi-dimensional, variable-size STL vectors that cannot be addressed for use on a GPU. However, I am working on ways to serialize the network to an array at run-time. Work-in-progress, but that is another concern for another site. $\endgroup$ – user27075 Mar 6 '18 at 3:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy