Tags:
create new tag
, view all tags

How to Submit Jobs

Introduction

  • Computations on the cluster are started by submitting a job to the queue manager (SGE),
  • The queue manager selects a compute node and starts the job on that node,
  • jobs run in batch mode, not interactive mode.

  • The job file holds the instructions needed to complete the computation, and may contain directives for the queue manager,
  • That job file is submitted with the qsub command, with optional directives for the queue manager.

  • This primer describes the basics on how to submit any type of job, although more is needed to submit parallel jobs (i.e., MPI, OpenMPI, or OpenMP), so refer to the corresponding primer.
  • There is an efficient way to submit a slew of similar jobs, known as a job array. Again refer to the appropriate primer.

To Submit a Job

one usually writes two files:

  1. one with the commands needed to run a computation, the job file,
  2. one with the qsub command and the required flags to submit that computation to the queue, the qsub file.

A Simple Example

Let us assume that you have a program (crunch.f) that you have successfully compiled as crunch and want to run.

The program crunch is in /pool/cluster/myname/test, and all the files needed to run it are in that same directory, and/or the files produced will go there. The corresponding job and qsub files would be qCrunch.job and qCrunch.qsub.

The Job File (qCrunch.job)

The job file is a script file, written using either csh or sh syntax, with the list of unix commands you need to execute to run a computation.

Note that unless you tell qsub otherwise, the job file must use csh syntax (not sh), by contrast to the script starting convention, a #! in the first line of a job file is ignored by qsub (and similarly a job file does not need to have the execution permission bit on).

The job file qCrunch.job would be something like:

hydra% cat qCrunch.job
cd /pool/cluster/myname/test
./crunch

The Qsub File (qCrunch.qsub)

Now write a qsub file qCrunch.qsub, i.e.:

hydra% cat qCrunch.qsub
qsub -j y -o qCrunch.log -N qCrunch qCrunch.job

Submit the Job

Submit the job as follows:

hydra% source qCrunch.qsub
Your job NNNNNNN ("qCrunch") has been submitted

  • This will submit a job named "qCrunch" (-N), with the output send to "qCrunch.log" (-o) and requesting to merge (join) the std err and std output in a single file (-j y).
  • The queue manager will start your job on a compute node, and the output of your job will go to the file qCrunch.log, in your home directory.
  • You can specify the error file name with -e filename if you choose not to use -j y.
  • The job output file(s), as specified with -o and/or -e, is/are appended to the file(s), if that/these file(s) already exist(s).
  • If you do not specify an output file nor an error file, look in your home directory for a files named name.oNNNNNNN file and name.eNNNNNNN file, where name is the name of your job file and NNNNNNN the job ID.

Job Status

  • You check the job status (or the cluster queue status) with the -qstat command:
    qstat -j NNNNNNN returns that job status, if still running or queued
    qstat -u yourusername returns the status of all your running or queued jobs
    qstat -g c returns the cluster queue overall status

  • You delete your jobs with the qdel command:
    qdel NNNNNNN kills or deletes from the queue that specific job
    qdel -u yourusername kills or deletes from the queue all your jobs

  • To check finished jobs, look at the log file, and/or use the qacct command:
    qacct -j NNNNNNN returns info on that job status, available only once it completed

Embedding Queue Directives in the Job File

You can also embed the flags you need to pass to qsub in the job file, so the example above is equivalent to this version of a qCrunch.job job file:

hydra% cat qCrunch.job
#
#$ -cwd -j y 
#$ -o qCrunch.log
#$ -N qCrunch 
#
./crunch
and the corresponding qsub file, is just
hydra% cat qCrunch.qsub
qsub qCrunch.job
so you could dispense of a qsub file and type the qsub command interactively.

The option -cwd allows you to avoid having to include the cd /pool/cluster/myname/test, assuming you submit the jobs from /pool/cluster/myname/test and it will direct the output file (qCrunch.log) to the current working directory, rather than your home directory.

The meaning of each flag is described in the qsub man page, in the example above they mean:

-cwd cd to the current working before staring this job (by default your job starts in your home directory)
-j y join (merge) stderr w/ stdout stream
-o qCrunch.log put merged stderr & stdout in a file called qCrunch.log (in the current working, if -cwd is set )
-N qCrunch name that job qCrunch
qCrunch.job the file name with the instructions to run the job (csh syntax, by default)

  • The available options (other flags) are explained in man qsub; read that man page 3 to 5 times, until you can recite it backwards wink

Note

  • All the embedded directive must be grouped near the top of the job file.

  • Since job files are in fact unix scripts, they can take arguments, and you can pass them after the job file name, similarly any flag (in fact anything) listed after the job file name is passed as an argument to the job, and not to the qsub command:
    • qsub -cwd -o test.log test.job is not equivalent to
    • qsub -o test.log test.job -cwd, the later pass the argument -cwd to the job, the former pass that argument to qsub.

  • If you write your job file using sh syntax, you must add the flag -S /bin/sh to qsub, or include the #$ -S /bin/sh directive in the job file.

  • As you queue more than one job, beware of name space, as the jobs will run concurrently (on different compute nodes, or not) and thus should not write to the same file(s).

  • If you job spawn a background task, make sure to add a wait at the end of the job script. This way the scheduler doesn't think that your job is finished while in fact it is still running (in the background).

  • Best practice rule is to not start/spawn background tasks.

  • Emacs users: make sure that your files are properly terminated with a NL (newline), otherwise the last line will not be executed.
Add the following lines in your ~/.emacs file to guarantee a newline at the end of every file edited with emacs:
;; always end a file with a newline
(setq require-final-newline t)

Advanced Details for Experienced Users

Requesting Resources

If your jobs need some specific resources, you can request them when submitting the job, as a -l arguments (or embedded directives).

You can specify as many -l as you need, each followed by the requested resource. for example:

request with to get
memory use limit -l s_data=2G 2GB of memory use
virtual memory use limit -l s_vmem=2G 2GB of virtual memory use
host with free memory -l mem_free=2G 2GB of free memory (obsoleted by -l mr=2G, see notes below)
reserve memory -l mr=2G inform the scheduler that you will use 2GB of physical memory (default assumption is 1GB)
cpu time limit -l s_cpu=2:00:00 two hour cpu limit
elapsed time limit -l s_rt=2:00:00 two hour real-time limit

NOTES

  • Refer to this page on memory use limits and reservation.
  • Always and use -l mr=XX if your need and will use more than 1GB/slot, where XX is 2.5G if your application will use 2.5GB of physical memory.
  • -l mem_free=2G is obsoleted by -l mr=2G (memory reservation), and is meaningless unless you specify -w n or -w w.
  • You can check your job status, including the requested resources, with qstat -j NNNNNNN where NNNNNN is the job ID.
  • You can look at what resources can be requested with the command qconf -sc. Look for the requestable (YES) variables and read the man pages on hydra for complex(5) and queue_conf(5) to learn more.

Environment Variables

When your job is started, the Grid Engine defines a slew of environment variables.

The variables QUEUE, JOB_NAME and JOB_ID hold, as expected, the queue name, the job name and the job ID. The first line of your job file could be:

echo + Job $JOB_NAME started `date` in queue $QUEUE with id=$JOB_ID on `hostname`

Look at ~hpc/show-env.log for a longer (complete?) list, and at hpc/show-env.job and hpc/show-env.qsub to see how that file was generated.

Catching Signals in Time Limited Queues

If the queue your job run under has a time limit (most do), you can catch the ran out of time signal if your job uses the sh syntax. (Unfortunately, the linux implementation of csh fails to properly catch these signals.)

Queues, under SGE, have real-time and cpu limits, each having a soft and a hard one (the soft being shorter than the hard). The real-time (or elapsed or wall clock) limit is the maximum allowed elapsed time, the cpu limit is the maximum allowed cpu use. The soft limit can be caught (like to save what has been computed so far), the hard limit causes termination of the job.

All the queues, but the uTz.q queue, have time limits.

For sh job, you can write something like:

#
#$ -S /bin/sh
#
warn()
{
 echo @ `date` warning, received $1 signal.
}
#
trap "warn xcpu" SIGXCPU
trap "warn usr1" SIGUSR1
trap "warn kill" SIGKILL
#
echo + `date +'%Y.%m.%d %T'` started on host `hostname` in queue $QUEUE with jobid=$JOB_ID
echo sleeping $1
sleep $1
echo = done `date`

For csh job files, you can't catch the signals, so I recommend that you end jor job file with:

echo = `date` all done.
and look for a line like = Thu Jan 26 17:41:17 EST 2012 all done. at the end of your job output file.

When the job is finished, you can check its exit status with the command qacct -j NNNNNNN, where NNNNNNN is the job ID.

More Examples

  • You can find more examples of job and qsub files, on hydra, in ~hpc/tests/queues, and in most cases corresponding *.log files.
    date.* trivial job that prints out the date
    example-ja.* job array example
    show-env.* prints job's environment variables
    sleep-sh.* trivial job that sleeps, [ba]sh syntax and catches termination signals
    sleep.* trivial job that sleeps, [t]csh syntax
    test-openmp-ifort.* test OpenMP job, Intel
    test-openmp-pgi.* test OpenMP job, PGI
    test-openmp.* test OpenMP job, generic
    worker.* run 'worker' example job

-- SylvainKorzennikHPCAnalyst - 26 Jan 2012

Topic revision: r7 - 2014-12-29 - SylvainKorzennikHPCAnalyst
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback