create new tag
, view all tags

Using the PGI compilers and MPI for the IB fabric

  • Read first Using the PGI compilers
  • MPI support is provided for the PGI compilers and the IB fabric as mvapich
  • To use the IB, programs don't need to be changed, but must be compiled and linked right
  • It is strongly recommended that you use version 13.5p: its PGI's compilers v13.5 with MVAPICH2 v1.9.


  • To include MPI support, the basic configuration can be expanded to point to the IB-capable MPI location (MVAPICH),
  • i.e., something like
    # #
    # Portland Compiler Group Compilers... # Portland Compiler Group Compilers...
    # #
    PGIVER=13.5p set PGIVER  = 13.5p
    PGI=/share/apps/pgi setenv PGI  /share/apps/pgi
    PGIx=$PGI/linux86-64/$PGIVER setenv PGIx $PGI/linux86-64/$PGIVER
    MVAPICH=$PGIx/mpi/mvapich setenv MVAPICH   $PGIx/mpi/mvapich
    LM_LICENSE_FILE=$PGI/license.dat setenv LM_LICENSE_FILE $PGI/license.dat
    MANPATH=${MANPATH}:$PGIx/man:$PGIx/mpi/mpich/man set path = ($PGIx/bin $MPICH/bin $path)
    # if (! $?MANPATH) setenv MANPATH ""
    PATH=$PATH:$PGIx/bin:$MPICH/bin setenv MANPATH ${MANPATH}:$PGIx/man:$PGIx/mpi/mpich/man
    # #
    alias dbx="pgdbg -text -dbx" alias dbx pgdbg -text -dbx


  • A simple example, on hydra=, is under ~hpc/tests/mpi/pgi/ib
  • The Makefile for a trivial 'hello world!" example, is
    # PGI
    PGIVER = 13.5p                        
    PGI    = /share/apps/pgi
    PGIx   = $(PGI)/linux86-64/$(PGIVER)
    # mpi location
    MVAPICH  = $(PGIx)/mpi/mvapich
    # (if MVAPICH is defined in your env, you can ignore the lines above)
    # flags
    CFLAGS = 
    MFLAGS = -I$(MVAPICH)/include -I$(MVAPICH)/include/f90choice
    # compiler/linker
    CC     = pgcc  $(CFLAGS) $(MFLAGS) $(IFLAGS)
    MVAPICC  = $(MVAPICH)/bin/mpicc  $(CFLAGS)
    hello: hello.o
            $(MVAPICC) -o $@ hello.o


  • PGI includes mvapich as part of their distribution. They used mvapich-1.2rc1, built against v13.0 of their compilers.

  • That version of mvapich, still available under /share/apps/pgi/linux86-64/13.5, has a known feature that prevents the grid engine's command qdel from properly killing all the slaves spawn by a qsub - or clean up the slaves if one crashes. The mvapich2-1.9 version solves this. It also has many other features for performance and scalabiliy and you should be able to get improved speedup for your applications.

  • You must recompile your code using 13.5p to use mvapich2-1.9.

  • Should you need to revert to the mvapich distributed by PGI, replace 13.5p by 13.5 in the examples above.

- SylvainKorzennikHPCAnalyst - 28 Jun 2012

Topic revision: r2 - 2013-10-02 - SylvainKorzennikHPCAnalyst
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback