___________________
$Updated from the versions since October 24, 2017; Submitted ??, 2018 

Abstract -

A progress report on SMA data status and issues in 2017 is issued based on tests by 
a programmer and reports from SMA users. A probability of having data issues in the 
process of converting SMA archived data into Miriad is computed in this report based 
on the statistics from the examined datasets, a sample of 73 elements. A progress
in patching the data errors is also summarized.

A number of datasets (73) taken from SWARM correlator only during 2017 have been examined 
with SMALOD and UVINDEX in Miriad and data issues are summarized in Table 1. Among the 
total examined datasets, Fourty-six are the datasets taken in January of 2017, the rows 
with grey background listed in Table 1, excluding those broken small datasets, which 
forms a complete sample set for the statistics in this progress report. The data sets 
chosen were based on either users' requests or programmer's testing for Miriad software. 
Column 1 is the filenames of the examined data sets. The column 2 gives brief descriptions 
of each dataset, consisting of sub-columns as follows:

RX Conf.     - "2 rx" stands for dual receivers, and the following two digits mark the 
               pairs of receiever indices (0: 230GHz, 1: 340GHz, 2: 400GHz, 3: 240GHz); for 
               example, rx03 and rx12 means  the receiver pairs of 230-240 and 340-400, 
               respectively; or "1 rx" stands for single receiver, followed by a digit for 
               the receiver code as recorded; 
CO Conf.     - "3C" or "4C" indicates that three or four SWARM spectral chunks were used, 
               and then next two sub-columns give a number of sources (target fields) and 
               the size of each data set in Gbytes, respectively;
Data issues  - marks whether issues are present in the raw archived data; if found, more 
               detailed information are given in the table notes with a color indicating 
               severity levels:
Red    - Fatal errors, not readable due to segmentation faults, need special patches 
         to fix; 
Orange - Critical warnings, readable, the issues might be fixable with a Miriad task 
         other than SMALOD; 
Blue   - Warnings, issues only affect the relevant visibility points that can be 
         flagged or the issues are straightforward and can be automatically repaired 
         by a general parse routine. 
  
Status       - marks the status for each of data sets; a word "Fixed" indicates that the 
               relevant data issues have been fixed and are ready to be further processed 
               in Miriad SMA-WB updated versions given in the Table notes, highlighted in 
	       green. 

A number of 34 datasets, the 1st or the "Good -" group, appear to be smoothly read in and 
converted into Miriad format with no troubles or minor issues. The minor issues mean that 
the on-line data errors can be fixed by the general software repairing kit that has been 
implemented without requiring specific patches. Among the "Good -" group, twenty-one 
datasets were taken from observations in January of 2017. A number of 39 datasets, the 2nd 
group, showed various data issues related to the corrupted headers and/or data that were 
archived from the online process of data acquiring. Among the 39 problematic datasets, 
twenty-five are the production of January in 2017. The statistics indicate that a 
probability of 47% (34/73) of the SMA archived datasets without special programming 
efforts to convert into Miriad, based on all the examined datasets, a large sample of 73 
datasets. From the 2017 January datasets, a smaller sample of 46 but complete, the 
probability is 46% (21/46) in successfully converting SMA archived datasets into Miriad. 
Both samples show that more than half of the datasets have issues require programming 
efforts to implement patches. 

The problematic data sets require patches to fix the specific fatal errors before further 
reduction in Miriad. A pre-processing (step 1) procedure has been created for ensuring SMA 
data to be further processed in Miriad&. One critical process in this step is the data 
examination and make patches when Fatal errors are encountered, which consumes time when 
a data set becomes large. Users are encouraged to send SMA bug reports if spotting issues 
during their data reduction.
____________________
&https://www.cfa.harvard.edu/sma/miriad/swarm/swarmoutsider.html



Table 1: A summary of the data issues from examined SWARM data sets taken in 2017


File name
Data description

 Good -
 RX Conf.
 CO Conf.
 Number of sources
 Size
 Issues
 Status

170101_03:57:48
2 rx03
 3C
 12 target fields 
 20 GB
no
OK

170102_04:54:31
2 rx03
 3C
 6 target fields 
 18 GB
no
OK

170103_04:07:52
2 rx03
 3C
 15 target fields 
 33 GB
no
OK

170106_13:04:50
2 rx03
 3C
 4 target fields16 
 20 GB
no
OK

170110_04:33:27
2 rx03
 3C
 7 target fields 
 45 GB
no
OK

170110_14:15:50
2 rx03
 3C
 11 target fields 
 15 GB
no
OK

170111_04:26:37
2 rx03
 3C
 10 target fields 
 27 GB
no
OK

170112_09:20:48
2 rx03
 3C
 10 target fields 
 17 GB
no
OK

170112_15:53:36
2 rx03
 3C
 11 target fields 
 8 GB
no
OK

170115_03:40:51
2 rx13
 4C
 7 target fields 
 38 GB
no
OK

170117_14:15:24
2 rx13
 4C
 9 target fields 
 14 GB
no
OK

170118_03:31:57
2 rx13
 4C
 11 target fields 
 34 GB
no
OK

170121_04:28:59
2 rx03
 4C
 9 target fields 
 24 GB
no
OK

170121_12:48:47
2 rx02
 4C
 7 target fields 
 6 GB
no
OK

170122_03:03:39
2 rx03
 4C
 5 target fields 
 19 GB
no
OK

170127_03:29:33
2 rx03
 4C
 5 target fields 
 18 GB
no
OK

170128_03:23:19
2 rx13
 4C
 7 target fields 
 26 GB
no
OK

170128_10:57:31
2 rx12
 4C
 6 target fields 
 16 GB
no
OK

170129_16:26:27
2 rx03
 4C
 80 target fields 
 16 GB
warning, readable17
OK

170130_08:54:24
2 rx12
 4C
 6 target fields 
 9 GB
warning, readable17
OK

170131_14:32:00
2 rx12
 4C
 3 target fields 
 9 GB
no
OK

170227_05:48:14
2 rx03
 4C
 6 target fields 
 48 GB
no
OK

170305_03:46:05
2 rx03
 4C
 5 target fields 
 36 GB
no
OK

170306_07:56:22
2 rx13
 4C
 8 target fields 
 42 GB
minor, readable1,2
OK

170421_09:49:44
2 rx03
 4C
 5 target fields 
 28 GB
no
OK

170701_02:37:16
2 rx03
 4C
 6 target fields 
 12 GB
minor, readable9
OK

170701_05:55:39
2 rx03
 4C
 4 target fields 
 65 GB
minor, readable9
OK

170809_02:59:36
2 rx03
 4C
 8 target fields 
 15 GB
no
OK

170923_11:03:17
2 rx03
 4C
 4 target fields 
 13 GB
no
OK

171010_07:53:56
2 rx03
 4C
 5 target fields 
 18 GB
no
OK

171102_02:54:08
2 rx03
 4C
 14 target fields 
 78 GB
minor, readable9
OK

171110_02:32:55
2 rx12
 4C
 16 target fields 
 78 GB
no
OK

171111_06:19:29
2 rx13
 4C
 6 target fields 
 35 GB
no
OK

171127_17:23:16
2 rx03
 4C
 6 target fields 
 35 GB
no
OK

 

 Patched  -
 RX Conf.
 CO Conf.
 Number of sources
 Size
 Issues
 Status

170102_11:48:06
2 rx12
 3C
 ? target fields 
?? GB
yes7
??????

170104_04:20:43
? rx??
 ?C
 ? target fields 
?? GB
yes15
??????

170105_07:58:24
2 rx03
 3C
 ?? target fields 
 ?? GB
yes7
??????

170105_18:15:46
2 rx03
 3C
 18 target fields 
 ?? GB
yes3,16
??????

170106_03:28:16
? rx??
 ?C
 ? target fields 
?? GB
yes15
??????

170107_04:36:42
2 rx03
 3C
 ?? target fields 
 ?? GB
yes7
??????

170108_06:41:09
2 rx03
 3C
 ?? target fields 
 ?? GB
yes7
??????

170109_03:59:30
? rx??
 ?C
 ? target fields 
?? GB
yes15
??????

170109_13:32:51
2 rx03
 3C
 4 target fields 
 ?? GB
yes3
Fixed4

170111_14:12:20
2 rx03
 3C
 ?? target fields 
 ?? GB
yes7
??????

170113_02:44:35
? rx??
 ?C
 ?? target fields 
 ?? GB
yes15
??????

170113_06:07:14
2 rx03
 3C
 4 target fields 
 ?? GB
yes3
??????

170113_10:38:30
2 rx03
 3C
 11 target fields 
 ?? GB
yes3
??????

170114_03:58:22
2 rx13
 4C
 ?? target fields 
 ?? GB
yes7
??????

170116_03:32:16
2 rx13
 4C
 ?? target fields 
 ?? GB
yes3
??????

170117_03:53:23
2 rx13
 4C
 ?? target fields 
 ?? GB
yes7
??????

170122_14:18:47
2 rx02
 4C
 ?? target fields 
 ?? GB
yes7
??????

170123_05:09:20
2 rx03
 4C
 158 target fields 
139 GB
yes1,14
Fixed2
 
170124_03:20:12
2 rx03
 4C
 5 target fields 
 10 GB
yes3
Fixed4
 
170125_03:19:15
2 rx03
 4C
 153 target fields 
104 GB
yes3,5
Fixed6

170125_15:07:22
2 rx03
 4C
 ?? target fields 
?? GB
yes1,7
??????

170125_20:52:18
2 rx03
 4C
 ?? target fields 
?? GB
yes7
??????

170127_11:17:34
2 rx12
 4C
 6 target fields 
?? GB
yes3
??????

170129_04:11:20
2 rx03
 4C
 183 target fields 
?? GB
yes1,3
??????

170131_10:53:50
2 rx13
 4C
 ??? target fields 
?? GB
yes1,7
??????

170210_02:33:39
2 rx03
 4C
 5 target fields 
 17 GB
yes3,11
Fixed13

170305_13:14:52
2 rx13
 4C
 4 target fields 
 12 GB
yes7
Fixed13

170508_04:37:30
2 rx03
 4C
 ? target fields 
 ?? GB
yes7
??????

170616_03:00:50
2 rx03
 4C
 14 target fields 
 45 GB
yes8,9
Fixed10
 
170617_02:38:09
2 rx03
 4C
 17 target fields 
 48 GB
yes8,9,11
Fixed10

170618_04:27:32
2 rx03
 4C
 9 target fields 
 28 GB
yes7,9
Fixed13
 
170626_05:53:29
2 rx03
 4C
 ? target fields 
 ?? GB
yes8
??????

170717_09:31:42
2 rx03
 4C
 9 target fields 
 40 GB
yes8
Fixed6

170719_02:54:43
2 rx03
 4C
 15 target fields 
34 GB
yes7
Fixed13

170731_04:00:53
2 rx03
 4C
 7 target fields 
111 GB
yes3
??????

170808_03:14:55
1 rx0 
 4C
 ? target fields 
?? GB
yes12
??????

170823_07:43:18
2 rx03
 4C
 ? target fields 
?? GB
yes7
??????

171019_07:43:18
??
 ??
 ? target fields 
?? GB
yes8
??????

171127_04:26:09
2 rx03
 4C
 ? target fields 
?? GB
yes7
??????


Table notes:
 1Error report for 170123_05:09:20 -
  Fatal: reference date string = 'Jan 1, -471�' corrupted
  Warning: Variable source has zero or negative size, in UVPUTVR  (skipped).

 2Data issues are patched in Miriad SMA-WB5.0.3 and later versions.

 3Error report for 170124_03:20:12, 170731_04:00:53, 170125_03:19:15 - 
  Segmentation fault (core dumped)  while reading and sorting out the source table.

 4Data issues are patched in Miriad SMA-WB5.0.4  and later versions

 5Error report for 170125_03:19:15 -
  Fatal: reference date string = 'Jan 1, -471�' corrupte fixed
  Header entries for 135 sources or target fields are readable; needed to pay 
  attention to the data, in particular, those samplings near the end of the run.

 6Data issues are patched in Miriad SMA-WB5.0.5 and later versions

 7Error report for 170210_02:33:39, 170508_04:37:30 -
  Segmentation fault (core dumped) while reading the Tsys/source tables.
     
 8Error report for 170626_05:53:29, 170717_09:31:42 -
  No responding while reading headers, corruption in the beginning.

 9Warning: The length of the original source name exceeding the Miriad limits and 
  other data string issues -
  Automatically repaired by SMALOD.

10Data issues are patched in Miriad SMA-WB5.0.6 and later versions

11Fatal: in_read of data sets (170210_02:33:39 170617_02:38:09) has a bizarre 
  souid for the last source entry of codes_read, its coordinates are apparently wrong.

12Error report in reading bl_read of data sets (170808_03:14:55) - 
  Fatal: number of baselines = 60 exceeds the limit of the standard SMA array.

13Data issues are patched in Miriad SMA-WB5.0.7 and later versions.

14Warning [smalod]: Variable source has zero or negative size (around int 5500), in UVPUTVR
  Warning: source ID=158 with coordinates RA=10:01:19.1462 DEC=  2:34:36.6400 but no name given.

15Error report in reading in_read of data sets (170104_04:20:43) -
  Fatal: a skip in inhid. 

16Warning: an entry 'target' with 'RA=00:00:00.0000 DEC=  0:00:00.0000' in source header is loaded.

17Critical warning: Antenna coordinates are all zeros.