Date: Thu, 28 Feb 2013 11:52:23 -0500 (EST)
From: Ken Young 
To: Jun-Hui Zhao 
Cc: Mark A. Gurwell , cqi@cfa.harvard.edu, aargon@cfa.harvard.edu, David Wilner 
Subject: Re: Chunk order when the SWARM correlator is active
Parts/Attachments:
   1 Shown    79 lines  Text
   2   OK    ~13 KB     Text, ""
----------------------------------------

On Thu, 28 Feb 2013, Jun-Hui Zhao wrote:

> Dear Taco,
>     I need catch up what you guys have done and a few
> questions for you.
>
> Have the test data for the SWARM correlator been transferred to
> RTDC? Where is the location?

There's only a really small amount of genuine SWARM data available.
We have an old-format data file with real SWARM data for one baseline
in 130202_12:38:42 .   Both sidebands contain the USB data, because
SWARM was only recovering one sideband during that test.   I doubt
you'll want to mess with that data set at all, since it's in the old
format, but it's the only *real* on-sky SWARM data we have.   We hope to
get more real SWARM data on March 13th.

We have a large number of data sets taken in the new format which
we will use when the SWARM correlator comes on line.   These data files
were written by a second process running dataCatcher, which writes a
second copy of the data in the new format.   So when the correlator crates
send data to dataCatcher now, dataCatcher writes the data in the old,
standard format, and then sends exactly the same data to a new version of
dataCatcher which writes the data in the new format.   So we have lots of
data sets with real science track data writen in both formats.   These
files live in /sma/rtdata/newFormat/mir_data (as seen by the summit linux
boxes).   The files taken between Jan. 24, 2013 and Feb 19, 2013 are good,
as far as I know.   Alice and I have dumped the header data on some of
those files, and they look OK.   Obviously there could still be problems,
and I'd of course like to hear about any problems you find in those data
sets.   As far as I know, that data is not being transferred back to
Cambridge automatically, so if you don't want to run your code in Hawaii,
you'll need to copy some of those data sets back to Cambridge.   Those
data sets contain neither real nor fake SWARM data - they just are written
in the new format, with the changes involving flags etc that we discussed
last year.   These data sets should allow testing of the aspects of the
data reduction packages which will need to be modified to handle the
header format changes, but which do not actually need SWARM data.

   I have just started writing new format data files with fake SWARM data.
The fake SWARM data will simply be a copy of real chunk s01 data from the
legacy correlator, interpolated onto the big new SWARM correlator chunks.
I would not recommend messing with those data sets yet (they just started
appearing last night).   I want to dump the new data myself first, to make
sure that the zeroth-order bugs are fixed.   I haven't even plotted a
single spectrum from the fake SWARM data yet.   I'll send another email
when I think the fake SWARM data sets are work looking at.   I hope that
will be in the next day or two.
>
> SWARM is the name of the new wideband correlator? Are there any
> technical memo or documents on this new device available to us?

There is a wideband backend Wiki here:

https://www.cfa.harvard.edu/twiki/bin/view/SMAwideband

which contains a very large amount of information about the SWARM
correlator, including schematics etc.   It might be hard to find exactly
what you want at that site, because it's primarily a place where the
engineers building the backend post their test results and documents.
If you have questions about the data that SWARM will actually produce, I
should be able to answer them, of find the answers for you.

> The new large data are written with new format or the DB format
> with adding the broadband spectral chunks? Can you send me the
> .h used in the dataCatcher code for writing the new data file?

I have attached the .h file defining the new format to this missive.

> Do we understand or have resolved the problems for the large
> size testing data (50 Gbytes) produced last fall, which was reported?
> Is the errors/problems due to Miriad code or online code?

Which problems are you referring to?

Thanks,

Taco