# Running Fortran Applications under HTCondor

All of the files for this example can be found on
**condor.liv.ac.uk** in
**/opt1/condor/examples/fortran**.

## Contents

IntroductionCreating the Fortran 90 files

Compilation and initialisation

Creating the simplified submit description file

Running the HTCondor jobs

Combining the results

Discussion

## Introduction

HTCondor is well suited to running large numbers of Fortran jobs concurrently. If the application applies the same kind of analysis to large data sets (so-called "embarrassingly parallel" applications) or carries out similar calculations based on different random initial data (e.g. applications based on Monte Carlo methods), HTCondor can significantly reduce the time needed to generate the results by processing the data in parallel on different hosts. In some cases, simulations and analyses that would have taken years on a single PC can be completed in a matter of days.

The application will need to perform three main steps:

- Create the initial input data and store it into files which can be processed in parallel.
- Process the input data in parallel using the HTCondor pool and write the outputs to corresponding files.
- Combine the data in the output files to generate the overall results.

As a very simple example we are going to use Monte Carlo analysis to estimate the
value of *pi*. This is a slightly artificial example however, Monte Carlo methods are used in
many practical applications and this example has the advantage of being easy to understand
and fairly straightforward to program. You can find many articles on this example on the
internet (two useful ones are:
https://www.geeksforgeeks.org/estimating-value-pi-using-monte-carlo/ and
https://academo.org/demos/estimating-pi-monte-carlo/
) but a very brief description is given below.

Imagine a circle of radius *r* inscribed in a square of sides *2r* and
further imagine we choose points inside the square at random (a bit like throwing
darts extremely badly at a dartboard). If a large number of points (*N*) are used then
we would expect the ratio of those falling inside the circle (*Nc*) to those inside
the square (*Ns=N*) to be in the same proportion as the area of the circle to the area of
the square. For large values of *N* we can therefore calculate an approximate value of *pi*
as *4(Nc/Ns)*, independently of *r*.

In the Fortran 90 program used below we will centre the circle on the origin and set *r=1.0*
hence the square has sides of length *2r=2.0* and we need to generate points with
coordinates *(x,y)* from
uniform random distributions in the range *-1.0 ≤ x ≤ 1.0* and *-1.0 ≤ y ≤ 1.0*.

Instead of calculating the points serially in turn, the points will be calculated in batches using a number of HTCondor jobs so that the calculations are performed concurrently (i.e. at the same time). In more realistic examples, this is what will speed up the overall computation. Clearly the points will need to be chosen in a statistically independent manner for the Monte Carlo technique to work properly so we use a different random number seed for each job, these having been written to separate input files beforehand.

## Creating the Fortran files

The first step is to create a program to generate the *N* input seed files
with names of the form *seed0, seed1, ... seed<N-1>*.
A suitable Fortran 90 program is this
(initialise.f90 ) which generates 1000
input files:

This is rather more complicated than it needs to be because of Fortran's crude string handling capabilities and it may be easier to use a more powerful language such as Python.

program initialise implicit none double precision x integer seed, i character(len=10) :: filename character(len=10) :: format_string do i = 0, 999 if (i < 10) then format_string = "(A4,I1)" else if (i < 100) then format_string = "(A4,I2)" else format_string = "(A4,I3)" endif write (filename, format_string) "seed", i open(666,file=trim(filename),status='unknown') write(666,'(i4)') i close(666) enddo end

The next Fortran 90 progam is the one that will actually be run on the HTCondor pool to calculate how many points fall within the circle. An example implementation (pi.f90) is given below:

program pi implicit none double precision x, y integer :: incircle, samplesize integer, allocatable :: seed(:) integer n, i, seed_value parameter(samplesize=1000) open(1,file='seed',action='read') read(1,*) seed_value call random_seed(size = n) allocate(seed(n)) seed(1) = seed_value call random_seed(put=seed) incircle = 0 do i = 1, samplesize call random_number(x) call random_number(y) x = x*2.0d0 - 1.0d0 ! generate a random point y = y*2.0d0 - 1.0d0 ! generate a random point if (x*x + y*y .lt. 1.0d0) then incircle = incircle+1 ! point is in the circle endif end do open(2,file='incircle',status='unknown') write(2, *) incircle close(2) endThis reads in a seed value from a file and initialises the random number generator with it. It then generates 1000 points and determines whether they are inside the unit circle. If so, a counter is incremented and finally the total number of points inside the circle is written to an output file. Note that the

*random_number()*function generates random numbers in the range

*0.0 ≤ x < 1.0*.

The final step is to sum all of the points falling inside the circle to calculate
*pi* by reading the partial sums from the results files (*incircle**).
This can be achieved using a Fortran 90 code such as this
(combine.f90):

program combine implicit none double precision pi integer seed, i, incircle integer total_incircle character(len=20) :: filename character(len=10) :: format_string total_incircle = 0 do i = 0, 999 if (i < 10) then format_string = "(A8,I1)" else if (i < 100) then format_string = "(A8,I2)" else format_string = "(A8,I3)" endif write (filename, format_string) "incircle", i open(666,file=trim(filename),status='unknown') read(666,*) incircle close(666) total_incircle = total_incircle + incircle enddo pi = 4.0d0 * DBLE(total_incircle) / 1000000.0 print '(A,F8.6)','Monte-Carlo estimate of pi: ', pi endThis reads the partial sum stored in each file in turn and sums these to calculate the approximate value of

*pi*. Again this is uncessarily complicated because of Fortran's basic string manipulation capabilities and using a higher level language, e.g. Python, could make life easier.

## Compilation and initialisation

The GNU **gfortran** compiler can be used to create the executables
that are to be run on the HTCondor server i.e.:

$ gfortran initialise.f90 -o initialise

$ gfortran combine.f90 -o combineIt useful is to create a UNIX executable from the code to be run on the HTCondor pool as well for testing purposes:

$ gfortran pi.f90 -o pi

The GNU Fortran compiler can also generate executables to be run under Windows
for use on the HTCondor pool so that is not necessary to compile
the code on a PC. To compile the pi.f90 code into a Windows executable (**.exe**)
actually on the HTCondor server use:

$ gfortran-win pi.f90 -o pi.exe

The 1000 seed files can be created using the previously built initialisation executable:

$ ./initialiseThe HTCondor code can then be tested on the server with a one off seed file e.g.

$ cp seed123 seed $ ./pi $ cat incircleIt can be extremely difficult to track down errors when jobs are run on the HTCondor pool and so testing the application first on the server should be considered

**essential**.

## Creating the simplified job submission file

Each HTCondor job needs a submit description file to describe how the job should be run. These can appear rather arcane to new users and therefore to help simplify the process, tools have been created which will work with more user-friendly submit description files. These are automatically translated into files which HTCondor understands and which users need not worry about. For this example, a simplified submit description file such as the one below can be used example called run_pi:

executable = pi.exe indexed_input_files = seed indexed_output_files = incircle log = log.txt total_jobs = 1000

The Windows executable is specfied in the **executable** line. The lines
starting **indexed_input_files** and **indexed_output_files** specify
the input and output files which differ for each individual job.
The total number of jobs is given in the **total_jobs** line.
The underlying job submission processes will ensure
that each individual job receives the correct input file (**seed0.dat .. seed999.dat**)
and that the output files are indexed in a corresponding manner to the input files
(e.g. output file **incircle1** will correspond to **seed1**)

It is also possible to provide a list of non-indexed files (i.e. files that are common to all jobs), for example:

input_files = common1.dat,common2.dat,common3.datThis is useful if the common (non-indexed) files are relatively large.

**Aside:**

For testing, the

**indexed_output_files**line can be omitted so that all of the output files are returned (the default). It is useful to also capture the standard output and error (which would normally be printed to the screen) using the

**indexed_stdout**and

**indexed_stderr**attributes respectively. For production runs, the output files should always be specified just in case there is a run-time problem and they are not created. In this case, HTCondor will place the job in the held ('H') state. To release these jobs and run them elsewhere use:

$ condor_release -all.

To find out why jobs have been held use:

$ condor_q -held

The job submission file can be edited using the **nedit** or **nano** editors, however all of
the options in it can be overidden temporarily from the command line (the file itself
is not changed). For example to submit five jobs instead of 1000 use:

$ fortran_submit run_pi -total_jobs=5and to also change the executable to be used:

$ fortran_submit product -total_jobs=5 -executable=other.exe(use the

*--help*option to see all of the options available)

This is useful for making small changes without the need to use the UNIX system editors to change the job submission file.

## Running the HTCondor Jobs

The HTCondor jobs are submitted from the the HTCondor server using the command:

$ fortran_submit run_pi

It should return with something like:

$ fortran_submit run_pi Submitting job(s)..................................................... ...................................................................... ...................................................................... 1000 job(s) submitted to cluster 1394.You can monitor the progress of all of your jobs using:

$ condor_qInitially the HTCondor jobs will remain in the idle state until machines become available: e.g.

$ condor_q -- Schedd: Q1@condor1 : <10.102.32.11:45062?... @ 08/06/19 10:25:35 OWNER BATCH_NAME SUBMITTED DONE RUN IDLE TOTAL JOB_IDS smithic CMD: run_pi.bat 8/6 10:23 524 97 379 1000 1394.4-999 476 jobs; 0 completed, 0 removed, 379 idle, 97 running, 0 held, 0 suspendedThe overall state of the pool can be seen using the command:

$ condor_status

## Combining the results

Once the jobs have completed (as indicated by the lack of any jobs in the queue),
the directory should contain 1000 output files named
**incircle0, ... incircle999**
which can be processed using the **combine** executable to generate the final result.
e.g.:

$ ./combine Monte-Carlo estimate of pi: 3.156000

## Discussion

Despite generating one million points, our estimate for *pi* is only
accurate to a couple of significant figures so this is clearly not
a very efficient method. If we examine the the partial sum files, the
reason for this becomes apparent:

$ cat incircle* | more 789 789 789 789 789 789 789 ...All of the partial sums are exactly the same ! This means that we have essentially solved the same problem 1000 times and got the same solution so that the maximum number of significant digits we could expect is three (since there are 1000 points in each job). If we submitted one million jobs, with each generating just a single point, then we would then expect around 80 % of the jobs to have a point inside the circle. Therefore another way of looking at this would be to say the problem has only fine grained parallelism and is not suitable for HTCondor

*in this form*.

We can do better than this by generating the random points *beforehand* and
storing them into files. This program (randpoint.f90) does just that:

program rand_points implicit none double precision x, y integer seed, iout, i, j character(len=10) :: filename character(len=10) :: format_string do i = 0, 999 if (i < 10) then format_string = "(A6,I1)" else if (i < 100) then format_string = "(A6,I2)" else format_string = "(A6,I3)" endif write (filename, format_string) "points", i open(666,file=trim(filename),status='unknown') do j = 1,1000 call random_number(x) call random_number(y) !write(666,'2(F8.6)') x, y write(666,*) x, y enddo close(iout) enddo endThe HTCondor jobs now read 1000 points from each file and calculate the number of points lying inside the circle. This code could be used for example (newpi.f90):

program pi implicit none double precision x, y integer :: incircle, samplesize integer, allocatable :: seed(:) integer n, i, seed_value parameter(samplesize=1000) open(1,file='points',action='read') incircle = 0 do i = 1, samplesize read(1,*) x, y print *, x, y x = x*2.0d0 - 1.0d0 ! generate a random point y = y*2.0d0 - 1.0d0 ! generate a random point if (x*x + y*y .lt. 1.0d0) then incircle = incircle+1 ! point is in the circle endif end do open(2,file='incircle',status='unknown') write(2, *) incircle close(2) endThe partial sums can be combined as before:

$ ./combine Monte-Carlo estimate of pi: 3.140900This is an improvement on the previous attempt and gives the same result as a serial version (the code for this in in serial.f90).

Although this is something of a toy example it does illustrate an important point - it very important when dealing with Monte Carlo methods to pay attention to the statistical properties of the random number generator(s). The first method used here is widely cited as an example on the internet (see for example this MPI version from Oak Ridge National Lab) however it is extremely inefficient and possibly fatally flawed.