1
$\begingroup$

I know for large amounts of data there are online computation services or supercomputers or Beowulf cluster. For a small quantity of data, a good computer is enough.

But sometimes, for medium qty of data it would be very nice to be able to use resources from the computers of my colleagues in displacement for example, or during the night.

Do you know if it is possible? I use python and I was expecting to use Ray. Any feedback about this type of practice?

$\endgroup$
  • $\begingroup$ If you're using Linux, take a look at Slurm $\endgroup$ – match May 2 at 15:57
  • $\begingroup$ How about HTCondor en.m.wikipedia.org/wiki/HTCondor $\endgroup$ – dr.blochwave May 4 at 14:54
  • $\begingroup$ thx for your help, I didn't know it. Slurm looks like a standard. I'll hve a look at it. $\endgroup$ – lelorrain7 May 6 at 13:40
1
$\begingroup$

You may be looking to run a job queue or automation software such as Gearman, Minion or Jenkins These work by having clients, say running politely on your colleagues' machines, contact your automation server for work which they complete and respond to. The Apache Spark project was aimed at doing this for pure computation. A lot of these have libraries for python or will be able to run stand alone python scripts for you. The trick will be moving the data around and I think Spark has that at its core.

Another possibility in the longer term is to consider languages with concurrency baked in such as Go or Raku. CPUs aren't getting any faster, they are just adding more cores, so performance gains will be coming from languages that take advantage of that. An added feature is that Raku can call routines from other language's libraries with its NativeCall interface, allowing you to re-use existing code.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thx I'll have a look at this! this is a good starting point! $\endgroup$ – lelorrain7 May 6 at 13:41
2
$\begingroup$

This is effective likely to involve setting up a small private cluster for yourself, accessing these other machines. Whether this will is theoretically possible will depend on your network and the permissions that whoever administrates your colleagues machines is willing to give you. If you have a series of linux machines which can see and SSH to each other across your local network, then you are well set, and will find multiple walkthroughs online, depending on chosen tool stack. If these are all heavily firewalled Windows machines you don't have administrator rights on and can't install new software on, then you may be out of luck.

I've not used Ray personally, but it seems to come with believable installation instructions (see the private cluster section here). I have experimented with Dask in similar circumstances, and it worked out ok. Overall, though there are social difficulties taking compute time on multiple machines 'owned' a local user and networks tend not to be as fast as on dedicated clusters. In many cases the lo-tech solution of SSHing into a friend's machine and running one copy of an embarrassingly parallel task is an easier solution.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.