High Throughput Computing using Condor

Introduction - why use Condor ?

Many researchers will have encountered obstacles such as these in the course of their work:

Often the only "solution" which presents itself is to borrow machines from colleagues to carry out the computing needed overnight or at weekends.

Pretty soon it becomes apparent that even this can no longer cope with the rising tide of computation and the thought naturally comes to mind:

"if only there were more PCs I could use".

This is where the Advanced Research Computing (ARC) Condor service can help. It allows applications to run on over 500 campus PCs and can reduce the time needed to perform certain classes of computing problem by orders of magnitude.

Which computing problems can Condor help with ?

Condor is a powerful system, but unfortunately it is not powerful enough to miraculously speed up any given computing problem. Usually some effort will be needed to adapt problems to Condor-style execution and the degree of difficulty will vary with the type of problem. Broadly speaking, Condor is most successful with problems where the same processing is applied repeatedly to different data sets or applied repeatedly using different simulation parameters (e.g. random numbers in Monte Carlo methods).

Condor has helped many researchers at Liverpool in disciplines as varied as medical and biological sciences, engineering and mathematical sciences - so don't be put off by these points. The Introduction to High Throughput Computing and the Condor limitations page should help you to decide whether your application will work with Condor. If you are still unsure, please feel free to contact ARC consultant Dr Ian C. Smith (the Condor service administrator) to discuss things (email: i.c.smith@liverpool.ac.uk).

What support is available ?

Researchers who wish to run MATLAB-based software (i.e. M-files) can benefit from tools created to help make the transition to Condor easier (see section on MATLAB applications). The majority of applications which run on Condor make use of MATLAB so there is considerable accumulated experience in this area. There is also help available for statisticians who use the R programming language (see the section on R applications). Help is also available for users of other software.

Sounds good ... how do I access Condor ?

Condor is provided completely free of charge to staff (including associates) and postgraduate students but we do ask that all users register for the service using this form which should be returned to the CSD helpdesk in Brownlow Hill. Please include a brief description of your project and try to keep us informed of any published research which has benefited from using the Condor service (including conference proceedings and dissertations) - this will help secure future support for Condor at Liverpool.