create new tag
, view all tags

How to run OpenMP jobs


OpenMP is a API for multi-platform shared-memory parallel programming. Programs are parallelized by the compiler, using embedded directives, called pragmas, that appears like special comments in the code. These are ignored at normal compilation, but a special flag to OpenMP-aware compilers will tell the compiler to produce a parallelized version.

The parallelization is a shared-memory model and uses multi-threading, not message passing and distributed memory like MPI. All the threads (sub-processes) must run on the same host (i.e., the same compute node).

If you want to run your own multi-threaded applications (assuming you know what you are doing), use the openmp PE to run them.


  • The degree of parallelism is thus de facto limited to the number of available cores on the compute nodes.
  • There is no mechanism on hydra to map the distributed compute nodes to a virtual large shared-memory machine.
  • If you request a lot of slots (processors, cores), your job may wait a long time in the queue until a compute node with that many cores frees up.
  • If you request more slots (processors, cores) than the largest number of cores in all the compute nodes, your job will hang waiting forever, unless you use the -w e flag (to qsub) when queuing that job.

How to Compile

  • All 3 sets of available compilers support OpenMP.
  • To enable OpenMP, use tha appropriate flag at compilation:
    compiler compiler flag
    GNU -fopenmp
    Intel -openmp
    PGI -mp

Do not mix and match object files produced by different compiler, but executables built with either compiler will run fine in the openmp PE (parallel environment).

To Submit a Job

  • The basics on how to submit a job are described in the primer's introduction on job submission, so read that one first.
  • The job file will launch your OpenMP program, but it will not specify the number of processors (threads, slots, CPUs)
  • The qsub file (or command) will request a number of processors (CPUs, cores) via the qsub command and specify the corresponding PE (parallel environment),
  • The job scheduler will grant the request and determine which compute node to use.


A Minimal Job File

  • Let's assume that you want to run the OpenMP executable myompcode
    [t]csh syntax [ba]sh syntax
    hydra% cat myompcode-csh.job hydra% cat myompcode-sh.job
    ./myompcode ./myompcode
  • Note that the env var $NSLOTS is not defined in the job file,
  • The variable NSLOTS will be set by the Grid Engine at execution time and holds the number of granted slots.
  • OpenMP uses the variable OMP_NUM_THREADS to specify the number of threads to use, hence it is set to $NSLOTS.

A Minimal qsub File

  • The corresponding qsub file is
    [t]csh syntax [ba]sh syntax
    hydra% cat myompcode-csh.qsub hydra% cat myompcode-sh.qsub
    qsub -pe openmp 8 \ qsub -pe openmp 8 \
    -cwd -j y \ -cwd -j y  \
    -N myompcode \ -N myompcode \
    -o myompcode.log \ -o mycode.log \
    myompcode-csh.job -S /bin/sh \


  • The above example requests 8 processors (CPUs, cores); adjust that number to your needs.
  • The flag -pe openmp 8 is the flag that tells SGE to use the parallel environment (PE) openmpe and requests 8 processors.
    • OPENMP is the PE to use with and only with OpenMP executables (or similar multi-threaded programs),
    • the scheduler will assign all the slots on a single compute node,
    • the -pe openmp 8 flag can be embedded (using #$) in the job file, like any other one,
    • options passed to qsub override embedded directives in the job file (including -pe).

More Examples

  • Trivial examples, hello.f90 with the corresponding Makefile, can be found under ~hpc/tests/omp/ for each compiler.
  • The corresponding job files are located in ~hpc/tests/queues:
    • test-openmp-gnu.job
    • test-openmp-intel.job
    • test-openmp-pgi.job
    • test-openmp.job
      This last one takes gnu|intel|pgi as argument (pick one):
      qsub -pe openmp 16 test-openmp.job pgi
  • The corresponding log files are in the same location as the job files.

-- SylvainKorzennikHPCAnalyst - 14 Feb 2012

Topic revision: r3 - 2012-07-12 - SylvainKorzennikHPCAnalyst
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback