Date: Mon, 13 Aug 2012 16:43:11 -0400 (EDT)
From: Ken Young 
To: eketo@cfa.harvard.edu, ckatz@cfa.harvard.edu, cqi@cfa.harvard.edu, mgurwell@cfa.harvard.edu, npatel@cfa.harvard.edu,
     jweintroub@cfa.harvard.edu, aargon@cfa.harvard.edu, jzhao@cfa.harvard.edu, rrao@cfa.harvard.edu, qzhang@cfa.harvard.edu,
     tksridha@cfa.harvard.edu, gpetitpa@sma.hawaii.edu, tksridha@cfa.harvard.edu
Cc: kyoung@cfa.harvard.edu
Subject: Interim Correlator size data file

   Last night we ran a science track with 7 antennas and 1.29 second (two
Walsh cycle) scans.    The resulting visibilities file is a bit shy of
50 GBytes, and it is real data taken with a PI-supplied observing script.
The size of this data set is about what we can expect with a longish
track using the Interim Correlator + Legacy Correlator, all 8 antennas
and 10 second integrations.   We do currently run some tracks (for
example mosaics) with 10 second scans now, so this file is not of an
unrealistic size for the Interim Correlator era, although it's on the
large end of what we can expect.   The data directory is

science/mir_data/120813_03:37:15/

I tried reading the data into Mir this morning, about 8 hours before
the track completed, when the visibility file was smaller than 30 GByte.
I used the machine in Hawaii with the most RAM (16 GBytes), hilodr1.
After 45 minutes, readdata died with the following message:

% Unable to allocate memory: to make array.
  Cannot allocate memory
% Execution halted at: DBI_CHAN_READ2    165
  /sma/local/mir/idl/pro/sma/dbi_chan_read2.pro
%                      READDATA           78
  /sma/local/mir/idl/pro/sma/readdata.pro
%                      $MAIN$

Later in the day, though still before the track ended, I tried using the
4 GHz version of Miriad to convert the data into Miriad format.    After
21 minutes, smalod died with a segmentation fault.

So it is not clear that we have any package that can process datasets
this large.   Within a day or two, the dataset will be available in
Cambridge, and we can try to put it into Mir using one of the bigger
CF machines.

One interesting side note - We were able to run with 1.29 second
integrations very smoothly for 17+ hours.   As a bonus, the 1.29 second
scan length is the same as the correlator board dump rate, so with that
scan length it is impossible for the crates to get out of step with
each other (requiring a correlatorPause/Resume).   I thought we'd
have lots of crate restarts, because I expected the crates to have trouble
pumping out the scans that rapidly, but there were no hiccups at all.
We could not have done this back when dataCatcher was running on M5.

Taco