UK National HPC Service

Computer Services for Academic Research Logo
Home | Helpdesk | Machine Status | Search | Apply

Submitting Jobs with LSF


The batch scripts are given to LSF with the command bsub. This is achieved with "bsub < scriptname". Here we describe the various useful options to bsub. These options can be added to the command line or placed at the top of the script (the option needs to be prefixed with "#BSUB"). Options placed on the command line can be used to override those specified in the file.

The most useful two options are -W and -n. The -n options allows you request a certain number of processors. The -W option requests a certain amount of wall clock time, this means that your job will automatically finish after that amount of time is used up if it has not already finished. The -c option is similar to -W in that it is a way of restricting the amount of time your job runs for, however -c is the total amount of CPU time used. Both -W and -c are measured in minutes.

You will also need to specify where your job runs and this can be achieved with the -q and -m options. The -m option specifies which machine to run on such as green. The -q option tells it which queue to run on for that particular machine.

The output from the batch job (the stderr and stdout) can be controlled with the -e and -o options. The -e options specifies where the stderr should be put and the -o option specifies where the stdout should be put. If neither option is specified the output should be emailed to you, if only the -o option is specified the stdout and stderr are merged into the specified file. It is also useful to put "%J" at the end of the filename as this will create a unique file for each job output.

The -J option is used to give your job a name which is useful to identify which of your jobs are running when using some of the LSF monitoring commands.

There a few other useful options such us -w which creates a dependency expression for your batch jobs. Often you may wish to chain jobs so that the jobs run in turn. The -w option takes a jobid and possibly a condition such as specifying that the job has exited or done. See the examples further on for instances of this in operation.

There are a number of options to allow the job to email you when it reaches various stages.

You will also find more flags and more information on the flags described here in the bsub manpage.

Summary of BSUB Commands

BSUB Command Meaning Notes
-B Sends an email when the job is dispatched and begins execution. NB Do not specify an email address after this option as it will not work. The email address is taken from the .forward file.
-b begin_time Specifies the earliest date or time that the job can be dispatched. -
-c cpu_time Limits the total CPU time the job can use (hour:minute) This time is minutes by default.
-e file_name Writes the stderr to "file_name" In LSF this cannot be combined with the -o. See -o for more details.
-J job_name Assigns the specified name to the job This can also be used to set up job arrays. See later notes for details.
-m host_name Specifies the host to run the job on -
-n min_proc[,max_proc] Specifies the number of processors to run the job on. In LSF, if one number is specified the job will run on that number of processors. If two are specified it will run on any number of processors between the two numbers.
-o file_name Writes the stdout to file_name. In LSF, if -o is specified and -e isn't, then the stdout and stderr are automatically written to the same file.
-q queue_name Submits the job to the specified queues. gnormal is no longer required for submitting jobs to Green. Instead use the -m green option together with -q normal.
-W runtime Sets the run limit of the job. This is not a direct equivalence as the time is not cpu time. However, it is a per process run time that ignores time spent whilst the job is suspended.
-w expresion Sets a dependency expression. This is not a direct equivalence as the time is not cpu time. However, it is a per process run time that ignores time spent whilst the job is suspended.
-R string Sets a resource request. Sometimes you may wish to make a more specific request to LSF such as ensuring all your processors appear on one host (useful on Newton).

Example 1

The following script can be submitted which will run a.out on 8 processors asking for one hour of green and writing the output to a file called output. It should be submitted using bsub < scriptname

	#BSUB -n 8
        #BSUB -W 1:00
        #BSUB -m green
        #BSUB -q normal
        #BSUB -o output
        mpirun -np 8 ./a.out

Example 2

The following is an equivalent script that can be submitted to newton passing the options as command line arguments -

bsub -n 8 -W 1:00 -m newton -q normal -o output scriptname

Only the mpirun line then needs to appear in the file.

Example 3

The following batch script sets a name up for the job (with -J) and we can use the -w option to submit the file with a dependency expression that the previous job completed normally.

	  #BSUB -n 32
          #BSUB -W 20
          #BSUB -J myjob
          #BSUB -m green
          #BSUB -q normal
          #BSUB -o output.%J
          mpirun -np 8 ./a.out

We can then run bsub < scriptname which will submit the job and perhaps return a numeric job id, lets say 1234. We can use this to ensure a second submission does not start until job number 1234 has completed by using bsub -w 1234 < scriptname.

If you want your next job to start even if the previous one failed then you can use bsub -w 'ended(1234)' < scriptname.

Page maintained by This page last updated: Thursday, 01-Dec-2005 13:44:33 GMT