Date: Mon, 13 Aug 2012 17:05:40 -0400
From: Chunhua Qi 
To: Ken Young 
Cc: eketo@cfa.harvard.edu, ckatz@cfa.harvard.edu, cqi@cfa.harvard.edu, mgurwell@cfa.harvard.edu, npatel@cfa.harvard.edu,
     jweintroub@cfa.harvard.edu, aargon@cfa.harvard.edu, jzhao@cfa.harvard.edu, rrao@cfa.harvard.edu, qzhang@cfa.harvard.edu,
     tksridha@cfa.harvard.edu, gpetitpa@sma.hawaii.edu, kyoung@cfa.harvard.edu
Subject: Re: Interim Correlator size data file
Parts/Attachments:
   1   OK     62 lines  Text (charset: ISO-8859-1)
   2 Shown   ~68 lines  Text (charset: ISO-8859-1)
----------------------------------------

Hi Taco,

For MIR, we need 3x RAM memory to process the data. For this data set, we could only hope to process one sideband a time. CF has servers
MARS (96 GB RAM) and NEPTUNE (128 GB RAM). Both servers should be able to load one sideband data in IDL unless there is any restriction of
RAM usage from a single user on CF servers. To load in one sideband, e.g. lsb:

IDL> readdata,sideband='l'

Let me know if it works.
On Mon, Aug 13, 2012 at 4:43 PM, Ken Young  wrote:
      � �Last night we ran a science track with 7 antennas and 1.29 second (two
      Walsh cycle) scans. � �The resulting visibilities file is a bit shy of
      50 GBytes, and it is real data taken with a PI-supplied observing script.
      The size of this data set is about what we can expect with a longish
      track using the Interim Correlator + Legacy Correlator, all 8 antennas
      and 10 second integrations. � We do currently run some tracks (for
      example mosaics) with 10 second scans now, so this file is not of an
      unrealistic size for the Interim Correlator era, although it's on the
      large end of what we can expect. � The data directory is

      science/mir_data/120813_03:37:15/

      I tried reading the data into Mir this morning, about 8 hours before
      the track completed, when the visibility file was smaller than 30 GByte.
      I used the machine in Hawaii with the most RAM (16 GBytes), hilodr1.
      After 45 minutes, readdata died with the following message:

      % Unable to allocate memory: to make array.
      � Cannot allocate memory

      % Execution halted at: DBI_CHAN_READ2 � �165
      � /sma/local/mir/idl/pro/sma/dbi_chan_read2.pro
      % � � � � � � � � � � �READDATA � � � � � 78
      � /sma/local/mir/idl/pro/sma/readdata.pro
      % � � � � � � � � � � �$MAIN$

      Later in the day, though still before the track ended, I tried using the
      4 GHz version of Miriad to convert the data into Miriad format. � �After
      21 minutes, smalod died with a segmentation fault.

      So it is not clear that we have any package that can process datasets
      this large. � Within a day or two, the dataset will be available in
      Cambridge, and we can try to put it into Mir using one of the bigger
      CF machines.

      One interesting side note - We were able to run with 1.29 second
      integrations very smoothly for 17+ hours. � As a bonus, the 1.29 second
      scan length is the same as the correlator board dump rate, so with that
      scan length it is impossible for the crates to get out of step with
      each other (requiring a correlatorPause/Resume). � I thought we'd
      have lots of crate restarts, because I expected the crates to have trouble
      pumping out the scans that rapidly, but there were no hiccups at all.
      We could not have done this back when dataCatcher was running on M5.

      Taco