Tags:
create new tag
, view all tags

How to use IDL & GDL on the cluster

Introduction

  • IDL is available on hydra, and uses the pool of the licenses available on the CF-managed machines.
  • Since IDL checks out a license when started:
    • It is not meant to be used on a cluster (i.e., running a slew of IDL jobs), as you will drain off the licenses, run out of licenses, and, prevent all users (on hydra and at CfA) from running IDL.
    • We have a set of special licenses, (i.e., 128 runtime licenses), to enable using IDL in batch mode, meant for use on the cluster.
    • The Grid Engine scheduler will keep track of license use to limit the number of concurrent use, but only if you add the -l idlrt=1 flag to you submission (see below).
  • IDL runtime has a few minor restrictions, check IDL's manual
  • IDL (interactive/regular and runtime) will fail to get a license if
    • the connection to CfA is down
    • the license server at CfA is down
  • GDL is also available on hydra, ver. 0.9, see below.
    GDL is a open source package that is compatible with IDL (v 7.1), see http://sourceforge.net/projects/gnudatalanguage/

Where is IDL on the cluster

  • IDL is installed on hydra under /software/rsi/ (instead of /opt/rsi on CF-managed machines).
  • Matt Phelps has installed versions 6.1 and 6.4 on hydra (a while back).
  • Examples to configure your environment for IDL are in ~hpc/.idl.sh or ~hpc/.idl.setup (sh or csh, respectively), look at ~hpc/.bashrc or ~hpc/.cshrc for examples on how to use them. Make you own copy, adjust it if needed, and edit accordingly your ~/.bashrc or ~/.cshrc file.
  • Since hydra uses the CF licenses, we could install versions 7.0 and 8.1.
    Email me (at hpc@cfa) if you need a more recent version, and let me know why.

How to use IDL runtime licenses

  1. The run-time licenses are for IDL ver. 6.1 (the latest version at the time we purchased them).
    We decided not to pay for s/w maintenance on these.
  2. You need to compile all of your IDL code, including all the routines that may be called, into a single file called a save set.
    You can produce that save set on any machine, but you must use the same version, i.e., 6.1.
    • Your top level cannot be a script (aka MAIN), but must be a routine (i.e., a PRO), that needs no arguments (since they cannot be set).
    • To produce the save set, let's assume your top level routine is reduce and is in reduce.pro, execute the following:
      hydra% idl
      IDL> .run reduce
      IDL> resolve_all
      IDL> save, /routine, file='reduce.sav'
  3. After this, you only need the .sav file on hydra, and any changes you would make to the .pro will not affect the execution of the .sav file, until you regenerate it.
  4. To run IDL and pull a runtime license, your start IDL with the -rt flag, hence your job file should look something like:
    #
    #$ -l idlrt=1
    #
    idl -rt=reduce.sav
  5. You can test you .sav file, by running it interactively, using the -rt switch, on any CF-managed machine, or for a short test run on hydra 's head node.

Note

  1. The name of the .sav file must the same as the top routine to run.
  2. If the .sav is in a different subdirectory, IDL will cd to that subdirectory when started.
  3. Tell IDL to use only one CPU, since all nodes have multiple cores and IDL will, by default, multi-thread large computation, and use that way more than 100% of one CPU. So add the following line in your top procedure:
    CPU, TPOOL_NTHREADS = 1 
  4. We have only 128 runtime licenses, therefore, when submitting a job that will use an IDL runtime license, you have to add the flag -l idlrt=1 to your qsub command (or add the embedded directive #$ -l idlrt=1 in your job file(s).) This will let the job scheduler keep track of license use and limit the number of concurrent running jobs. Still consider limiting the number of concurrent jobs you submit, since it is a shared resources (and thus avoid draining them by a single user).
  5. IDL license sever is dumb and will pull regular (interactive) licenses if:
    • you run out of runtime licenses,
    • the regular IDL license server is running but the runtime server is for some reason down (it has happened).
  6. You cannot prevent using 'regular' license, but by telling the scheduler that you pull a license it will limit the number of concurrent jobs.
The following line in your top procedure (see examples below):
IF lmgr(/runtime) THEN message, /info, 'runtime mode is on' $
    ELSE message, 'not in runtime mode'
only tells you how you started IDL, not what type of license was pulled.

Examples

You can find two examples, on hydra, in ~hpc/tests/idl. Look at test_me.pro and sleep_two.pro.

These are converted to .sav with c-test_me.pro and c-sleep_two.pro.

So copy the .pro files and produce the save sets with:

hydra% idl c-test_me or hydra% idl c-sleep_two

then test them on the head node with:

hydra% idl -rt=test_me.sav or hydra% idl -rt=sleep_two.sav

The procedure test_me allows you to check (w/ lmstat, as described above) that you have indeed pulled a runtime license, while waiting for you to press return.

The procedure sleep_two will print the date, pause for two minutes, print the date again and finish.
It checks whether it is pulling a runtime license or not, i.e., compare

hydra% idl -rt=sleep_two
to
hydra% idl
IDL> sleep_two

The job and qsub files, to run sleep_two.sav, are sleep_two.job and sleep_two.qsub respectively, note the embedded flags in the job file.

To submit a job to run sleep_two

  1. Copy sleep_two.pro, c-sleep_two.pro, sleep_two.job and sleep_two.qsub
  2. Create the .sav file with
    hydra% idl c-sleep
  3. Submit it with
    hydra% source sleep.qsub
  4. It will run, once a license is available, and will produce the log file sleep_two.log.

Monitoring license use, on HYDRA:

  1. to check the current limit on concurrent IDL runtime jobs:
    hydra% grep idlrt_license /opt/gridengine/default/spool/qmaster/exec_hosts/global
  2. to check how many IDL runtime licenses are being used:
    hydra% qhost -F idlrt | head -4
  3. To find out which users are using or requesting R/T licenses:
    hydra% qstat -r -u '*'
    this command can produce a long output, so redirect the output to a file and search for the string idlrt_license
  4. I have updated q+, my PERL wrapper around qstat (alias q+ ~hpc/sbin/q+), and added the -lic option to help track license use.
    hydra% q+ -lic

Monitoring license use, at SAO, on a CF-managed machine

  1. You can check who many runtime licenses are available with:
    cfa0% /opt/rsi/idl_6.4/bin/lmstat -c /opt/idl/license/license.dat -f idl_rt
  2. Each instance of idl -rt=file.sav that checks out a license will grab 6 tokens (for linux), we have 768 tokens (6 * 128).
    So if 240 are "in use", there remains 528 tokens for 88 instance of idl -rt=file.sav.
    Note that for IDL (FlexLM), an instance is when the combination of the user name, host name, and display name are unique.


How to use GDL

  • To use GDL on hydra, setup your LD_LIBRARY_PATH to include /share/apps/plplot/lib, the executable is in /share/apps/gdl/bin/gdl.

  • An example of setup could be (csh syntax, assumes LD_LIBRARY_PATH is already set):
    setenv LD_LIBRARY_PATH $LD_LIBRARY_PATH":/share/apps/plplot/lib"
    alias gdl /share/apps/gdl/bin/gdl

  • GDL does not use any license, since it is open source, but you get what you paid for.

  • GDL is installed on CF-managed machines under /usr/bin/gdl, and on these machines you don't need to set LD_LIBRARY_PATH to a specific value.

-- SylvainKorzennikHPCAnalyst - 30 Jan 2012/Revised 26 Feb 2013

Topic revision: r7 - 2013-02-26 - SylvainKorzennikHPCAnalyst
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2015 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback