Desktop software with HPC resources for back end number crunching

Question

Our workgroup produces a desktop application that simulates building energy performance. It is a .NET application and when the user is running a lot of simulations, they can be quite time consuming. The simulations are totally parallelizable, and we have some very significant HPC computing resources at "the office".

One idea we've had is to allow the users to offload simulations that we know will be very long running (while each individual simulation runs about 30-120 seconds, running large numbers of simulations could take several days). Has anyone done this before? If so, did you use any libraries to make the job easier? Was it worth the effort?

edited to add:

Individual tasks for offloading the simulation would be

• packaging a file (about 5Mb),
• uploading it to our servers,
• decomposing the package into individual simulations (each takes about 30-120 seconds and is totally parallelizable), the number of simulations is a function of the number of options selected by the user (insulation, building orientation, etc) and the worst case of selecting every possible option would result in about 1E50 simulations. Running 100 to ~1E5 simulations isn't unknown, but the majority of users will run less than 10.
• reassembling the completed simulations and downloading the now much larger file.

We aren't sure what interface to use, as our group is new to this, and with budget cuts, it might get completed in time, but needs to be easy for the next folks (if any) to maintain.

This app already uses .NET 4 and can expand to use all the cores that the user has (our dev machines have 8 cores).

Answer 1

You might consider using Condor: http://research.cs.wisc.edu/condor/. It could be overkill, depending on what kind of HPC resources you are talking about and how much they will change. However, its fairly easy to get starting, and it takes care of launching independent jobs, fault tolerance, load balance, etc.

It would be more helpful if you were more specific about the individual tasks (how long, how asynchronous) and what interface you would use for interaction (launch process, library, etc)

Matt

Answer 2

If you are in the .NET world, have you looked at the multi-threading support in .NET 4; and the asynchronous features that are being added to .NET 4.5?

Multithreading does not help you offload processing to other machines, but it does let you better utilize the multicore PCs we're all using these days. 4.0 has some nice features in this regard.

4.5 is currently in technology preview - so it is worth a look to see if it will help. There was an article in MSDN introducing the new asynchronous features in the last month or so (Oct or Nov, and definitely no earlier than Sept). It looked like these features were being pushed as a way of parallelizing traditionally less parallel code, but IIRC they also included some multi-server support.

Answer 3

The answer depends on what you have as the HPC backend. The question refers to existing resources available to NREL users. There are Linux clusters and a Windows cluster based on the Windows Compute Cluster Server.

1. To use the linux cluster the author needs to make sure the simulation routine runs on Mono (the implementation of .NET platform for linux) and make sure Mono is installed on the nodes.

2. The next question is whether the users have accounts on the compute clusters and can get the data to the nodes. The simplest case is with a single filesystem accessible from local PCs and from the nodes.

   I am not sure if such a filesystem is available to NREL users, therefore some copying will be involved. This already makes things complicated.

3. Next the application should be able to compile a job description script, submit it to the respective cluster and query the queueing system for the state of the simulations until the job gets out of the queue.

4. Finally the application should be able to retrieve the results.

All This is doable though not trivial and would be highly specific to the setup of this particular lab.

Commercial applications from Ansys or MSC let the user save a simulation job description that can be later run in batch mode on a remote cluster.

A more general solutions would be to use the Grid infrastructure if available.

Next one could go for service oriented architecture where a Web-service is set up somewhere to wait for simulation requests and dispatch them onto the cluster thus decoupling the application from the knowledge of the specifics of the available resources.

This is widely used for instance in computational biology, where tools are used to compose simulation workflows consisting entirely of web-service requests for processing.

Just to note that offloading computations to remote computing resources is commonly used in applications like Google voice recognition for Android.