Date: Wed, 1 Aug 2012 15:28:04 -0400 (EDT)
From: Ken Young 
To: eketo@cfa.harvard.edu, ckatz@cfa.harvard.edu, cqi@cfa.harvard.edu,
     mgurwell@cfa.harvard.edu, npatel@cfa.harvard.edu,
     jweintroub@cfa.harvard.edu, dwilner@cfa.harvard.edu,
     aargon@cfa.harvard.edu, jzhao@cfa.harvard.edu, rrao@cfa.harvard.edu,
     qzhang@cfa.harvard.edu, tksridha@cfa.harvard.edu, gpetitpa@sma.hawaii.edu,
     rblundell@cfa.harvard.edu, tksridha@cfa.harvard.edu,
     kyoung@cfa.harvard.edu
Subject: Proposed changes to the SMA Data File format

Dear SMA Person,

   Yesterday we had an SMA Science Meeting during which the attendees were
asked about what changes they would like to see made to our data reduction
proceedures which might require changes to the data file format.   I have
put together a list of the changes we discussed yesterday, along with some
other changes we have talked about in the past.    Please take a look
at this laundry list before our meeting next week, and see if you have
any opinion about any of them.

Proposed changes:

1) Change the flagging.   The current method of flagging data bad involves
changing the sign of the weight.   So effectively the sign bit of the
weight is our flag.   That has the serious downside of not allowing us
to unset flags we have set during data processing, without losing all of
the flags set by the online system (for slewing, etc).   I propose that
we add a 32 bit flag variable for each bl record entry.   The lower 24
bits would be reserved for online system flag bits, and the upper 8
bits would be reserved for flags set during data processing.   That would
allow us to unset flags set during data processing, as well as flags
that don't necessarily mean the data is bad such as the Wacky Offsets
flag or the Feed Offsets Mismatch flag.    FYI, here are the flags that
are currently being written to the we_read file - I propose we use these
for the lower 24 bits of the new 32 bit flag:

Gunn PLL unlocked            0x00000001
Cal vane in beam             0x00000002
Bad Sample Statistics        0x00000004
Coordinate Mismatch          0x00000008
Warm dewar                   0x00000010
Antenna drives off           0x00000020
Feed offsets mismatch        0x00000040
Bad IRIG time                0x00000080
M3 Door Closed               0x00000100
Optical pointing mode        0x00000200
Mismatch source names        0x00000400
Track stale                  0x00000800
Wacky pointing offsets       0x00001000
Bad chopper position         0x00002000
Ave. tracking error high     0x00004000
Peak tracking error high     0x00008000
Source changed during scan   0x00010000
Antenna shadowed             0x00020000
Operator set flag            0x00040000
BDA problems                 0x00080000
Waveplate moved during scan  0x00100000
Online software set flag     0x00200000


It would be good to have a way in MIR and Miriad to ignore any of those
flags.

2) Get rid of baseline-based Tsys values in the data file, and use antenna
based Tsys exclusively.   At the moment we store both baseline Tsys and
antenna Tsys.   When the Interim Correlator comes on line, we will have
a new multi-channel continuum detector which will provide separate Tsys
values for every 1 GHz interval in the IF.   Let's not store both antenna
and baseline versions of that!   Since we have redundant information in
our data file, it's hard to be certain which piece of code is using which
type of Tsys.   Let's move to having only antenna Tsys in the data file,
and calculate baseline-based Tsys dynamically whenever it is needed.

3) Our method of handling mosaic observations is a mess.   We encode the
offset position within the source name.   This was done because there was
no obvious set of variables in the MIR dataset that could be used
exclusively for mosaicing.   Let's add a pair of offsets for the RA and
Dec offsets in a mosaic map, and keep the source name constant over
the map.

4) There was a long discussion yesterday about updating the data file
UVW coordinates when a new baseline solution becomes available.   What
would you thing about removing the UVW coordinates from our data files,
and having the online code generate those coordinates when the data is
read, using the antennas file, the hour angle, and the declination?
If we do that, we will remove redundant information, which is
always a good thing, and the person reducing the data can update the
coordinates simply by copying in a revised antennas file.

5) No more byte swapping - store everything in x86 native byte order.

6) Shall we get rid of some or all of the variables we don't use, like
bl.pa, bl.iifc, bl.angres etc?   It'll clean up our files, and remove
mysteries, but it will also mean there will be no unused variables to grab
later on and repurpose for new uses.

7) Shall we rename the variables that we have already repurposed?   For
example, bl.iaq was defined by OVRO to be "Amp Qual Int Code",
but we are using it in dual Rx polarization as the place to store the Rx
associated with the lower numbered antenna on a baseline, so we could
rename this to something like bl.ant1Rx.

Taco