create new tag
, view all tags

How to submit a MPI job, using GNU, over IB

  • This primer describes how to submit parallel MPI jobs, using program compiled and linked with the GNU compilers, over IB (MVAPICH)
  • The executable(s) MUST be compiled and linked using GNU and MVAPICH, (not OpenMPI or MPICH)
  • The primer on compilers describe how to compile and link using GNU and MVAPICH.
  • An executable compiled w/ an other compiler (PGI, OpenMPI, Intel, etc) should not be submitted this way. It may run, but is likely to give you grief.

To Submit a Job

  • The basics on how to submit a job are described in the primer's introduction on job submission, so read that one first.
  • The job file include the command to launch your MPI program, using the mpirun command.
  • The number of processors and the machinefile (list of hosts to use) is not explicitly specified (hardwired) with the mpirun command.
  • You must invoke the corresponding mpirun (GNU MPI implementation, not ORTE), see example below.
  • The qsub file (or command)
    • will request a number of processors (CPUs, cores)
    • and specify the corresponding PE (parallel environment), via the qsub command
  • The job scheduler will grant the request and determine the hosts list for that specific job (i.e., the machinefile)
  • The jobs file can specify the PE and the number of processors via an embedded directive (i.e., #$ -pe mpich_ib 8)


A Minimal Job File

  • Let's assume that you want to run the MPI (MVAPICH) executable mycode
    [t]csh syntax [ba]sh syntax
    hydra% cat mycode-csh.job hydra% cat mycode-sh.job
    set MPICH = /usr/mpi/gcc/mvapich2-1.9 MPICH=/usr/mpi/gcc/mvapich2-1.9
    $MPICH/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines mycode $MPICH/bin/mpirun -np $NSLOTS -machinefile $TMPDIR/machines mycode
  • Note that the env var $NSLOTS and $TMPDIR are not defined in the job file,
  • These variables will be set by the Grid Engine at execution time and will hold the number of granted slots and the list of hosts (machines) to use.

A Minimal qsub File

  • The corresponding qsub file is
    [t]csh syntax [ba]sh syntax
    hydra% cat mycode-csh.qsub hydra% cat mycode-sh.qsub
    qsub -pe mpich_ib 8 \ qsub -pe mpich_ib 8 \
    -cwd -j y \ -cwd -j y \
    -N mycode \ -N mycode \
    -o mycode.log \ -o mycode.log \
    mycode-csh.job -S /bin/sh \


  • The above example requests 8 processors (CPUs, cores); adjust that number to your needs.
  • The flag -pe mpich_ib 8 is the flag that tells SGE to use the parallel environment (PE) mpich_ib and requests 8 processors.
    • GNU/MVAPICH executables must use that PE (not orte_ib),
    • by specifying mpich_ib, not mpich, your job will run on compute nodes connected to the IB fabric,
    • the -pe mpich_ib 8 flag can be embedded (using #$) in the job file, like any other one
  • An other page describes in more detail the available queues.
  • For MPI over IB you must use one of the following queues, sTNi.q, mTNi.q or lTNi.q.
    These correspond to the short, medium and long execution time respectively, (the queue is specified with the -q flag).
  • Options passed to qsub override embedded directives in the job file (including -pe or -q)

Details for Experienced Users

  • At run-time, the scheduler defines the following MPICH specific variables:
    NSLOTS The granted number of slots, or number of processors for this MPI run
    TMPDIR The temporary directory, where the machinefile will be written,
      the file that holds the distribution of processors over the compute nodes

  • Hence you can use, in the jobs file, the commands
    echo number of slots is $NSLOTS to print out the granted value of NSLOTS
    echo machine file is $TMPDIR/machines  
    cat $TMPDIR/machines to print out the name and content of the machinefile

More Examples

Look, on hydra, in ~hpc/tests/mpi/gnu for some examples.

  • To execute them, create a test directory and extract the compressed tar-ball:
hydra% mkdir -p ~/tests/mpi/gnu/gnu+ib
hydra% cd ~/tests/mpi/gnu/gnu+ib
hydra% tar xvzf ~hpc/tests/mpi/gnu/gnu+ib/tests.tgz

  • Build the executable
hydra% make

  • Run (some) of the tests
hydra% source hello-csh.qsub
hydra% source hello-sh.qsub
hydra% qsub hello-csh-opts.job
hydra% qsub -pe mpich_ib 4 hello-csh-opts.job
hydra% qsub -pe mpich_ib 4 hello-sh-opts.job

Run only one job at a time, use qstat to monitor the job, then look at the hello.log file.

-- SylvainKorzennikHPCAnalyst - 12 Jul 2012

Topic revision: r3 - 2013-10-02 - SylvainKorzennikHPCAnalyst
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback