Tips for reducing the SMA data with new format 03/02/2018 We have started the new data format since Nov. 15th, 2014. MIR should be ready for processing the SWARM only, ASIC+SWARM and ASCI only correlator data with the same calibration procedure as before. Please check https://www.cfa.harvard.edu/~cqi/mir/mir2018.txt for an example script to reduce the recent SMA data (2018 January). Here is some tips for data reduction using MIR: 0 MIR uvfits output can now be loaded into CASA for imaging. The previous bug in the velocity header has been fixed. For now, only one source/receiver/sideband/chunk uvfits output can be recognized by CASA. If it is a dual receiver data, one receiver has to be specifically selected before outputting to uvfits file. Here is an example of uvfits output: IDL> select,/p,/re, rx=345 IDL> result=fits_out('source_ls4','UVF','source','','','s4','l','','') which will output uvfits filename in capital letters, i.e. SOURCE_LS4.UVF, then you can load it into CASA with the python code MIRFITStoCASA.py (provided by Luca Matra) which can be downloaded from below (https://www.cfa.harvard.edu/~cqi/mir/MIRFITStoCASA.py). To load the uvfits file into CASA: CASA <1> import sys CASA <2> sys.path.append('***path-to-folder-containing-MIRFITStoCASA***') CASA <3> import MIRFITStoCASA CASA <4> MIRFITStoCASA.MIRFITStoCASA(UVFITSname='SOURCE_LS4.UVF',vis='source.ls4.ms') which will use CASA task importuvfits to load uvfits file into CASA and also apply the correct weights scaling factors to the data. A wrapper program AUTOFITS can be used to output uvfits file, just type: IDL> autofits, source='' which will output all the available chunks into uvfits files for a given source, no data filter needed. 1 Spikes can be removed by averaging the adjacent channels with UTI_CHANFIX, e.g. IDL> select,band='s49',/reset IDL> uti_chanfix, channel=9958 which will replace the channel 9958 of s49 with the average of the adjacent 3rd channels in both directions. IDL> uti_chanfix, channel=9958, sample=10 will use the 10th adjacent channels to replace the channel 9958. 2. A new MIR program UTI_CHECKSPIKE is added to detect and fix the spikes in swarm chunks. The program will look for spikes where the difference of data spectra and smoothed spectra is over a threshold. By default the threshold is set as 10. The syntax is IDL> uti_checkspike, source='' you have to specifiy a source name, preferably the gain calibrator source. It will print out any spikes that over the threshold detected for this source. You can modify the keyword NTRIM to ignore the edge channels; THRESHOLD to set different THRESHOLD value. By default the spectra is averaged over all baselines but you can use keyword flag BASELINE to look for the spikes for each baselines, e.g. IDL> uti_checkspike, source='0510+180', ntrim=1000, /baseline The flag FIX can be set to fix the spikes detected in the same way as in UTI_CHANFIX, e.g. IDL> uti_checkspike, source='0510+180', ntrim=1000, /baseline, /fix, sample=3 will look for the spikes based on the data on source 0510+180 for each baseline and if there is any spike, it will replace the channel with the average of the adjacent 3rd channels in both directions and the spike will be fixed on all sources. 3. When you deal with polarization data, for the newformat data (you can tell from the message when you use readdata to load the data into IDL), 'iaq' tag is replaced with 'ant1rx' and 'ibq' is replaced with 'ant2rx'. So the command for the old data IDL> result=dat_filter(s_f,' "iaq" eq "0" and "ibq" eq "1"') can be replaced with IDL> result=dat_filter(s_f,' "ant1rx" eq "0" and "ant2rx" eq "1" ') 4. UTI_AVGBAND has been updated so it can do averaging on swarm chunks. For now, by default, it will average the center 82/104 fraction of the chunk for the continuum regeneration. However, you can specify the starting channel and ending channel of swarm chunks to generate continuum, for example, IDL> uti_avgband, swmch1=1000, swmch2=15000 which will average the swarm chunks between 1000 to 15000 along with the typical range of asic chunks to generate continuum. The bandwidth of the continuum will be calculated accordingly. The weights will be reset by apply_tsys once the bandwidth is correctly set. 4. When you see the following message from readdata % Compiled module: READTSYS2. ******** OS TYPE *********** Platform =x86_64 TSYS reading ... More than 3 receivers data recorded. Quit! This is due to some messed-up receiver headers. The data has been already loaded into IDL, except the tsys values. The "Quit!" is only referred to the tsys reading program. The way to fix is to fix the receiver header first and then rerun the tsys reading program to load the tsys in. However, it's not easy to automatically fix the headers since the receiver headers are somehow messed up randomly. Here is an example how to fix it manually: First figure out the receiver headers in data. You can find the receiver header index by IDL> print, c.rec 230 345 400 240 It means 0 for rx230, 1 for rx345, etc. Below is an example to find out the receiver headers problem on a recent track 170215_11:29:32: IDL> print,uti_distinct(bl.irec,nrec,/many) -1 0 3 IDL> result=dat_filter(s_f,'"irec" eq "-1"',/reset) 145 passed in filter IDL> result=dat_filter(s_f,'"irec" eq "0"',/reset) 321600 passed in filter IDL> result=dat_filter(s_f,'"irec" eq "3"',/reset) 321655 passed in filter As you can see, receiver headers (irec) should not be "-1". It must be mislabelled from either rx230 or rx240 or both. Normally the record number passed for the mislabelled data is the difference of the two headers with correctly labelled data but it's not the case for this track. If you plot the continuum on the irec=-1 data, you will find two points for lsb and one point for usb for each baseline (note plot_continuum will label this data as rx230 data). It means the mislabelled data has both 230s and 240s for lsb and maybe just 230s for the usb. Fortunately it is just int#2183 was affected in this case. So I will put all the mislabelled data as rx230 (with irec as 0, the receiver with smaller record number passed), load tsys values with readtsys2, and then flag out #2183. The commands are: IDL> result=dat_filter(s_f,'"irec" eq "-1"',/reset) IDL> bl[pbf].irec=0 IDL> readtsys2 IDL> select,int=[2183,2183],/reset IDL> flag,/flag The guide line is to find out the mislabeled data, manually label them as one of the existing receiver header (with less records passed) so we can load the tsys values for the whole data with readtsys2, and then flag the mislabeled data out . Let me know if you have any questions on any specific data. 5. Data merging and output for mosaic tracks In previous MIR release, idl2miriad could not export source data with multiple pointings, so we have to append the RA and DEC offsets to the end of the target name and output each pointing individually. This practice makes the data output process rather cumbersome. Some tracks even have more than 100 source names. If we have to merge more than two such tracks, the source numbers will be so many that selecting one target source will generate so called "Program code area full" error as the system memory set aside to compile IDL procedures will not be enough to process the data selecting request. Now MIR can export the data of a target source with multiple pointings. So we don't need the mutiple names for pointings. A new utility program (uti_mosaic_fix) can be used to converge all the mosaic source names back to the target name. To use the utility routine, the target source name should be provided with the keyword SOURCE. For example: IDL> uti_mosaic_fix, source='M51' will converge all the M51* names to just M51. idl2miriad should be able to handle the multiple pointings if the source name "M51" is selected. This procedure should be done before data merging for mosaic tracks. After uti_mosaic_fix, if we want to retrieve a certain pointing data, say related to "M51_+161.04_+139.465" ( "IDL> print, c.source" to check the exact name), we can do (assuming all the related pointings renamed to "M51": IDL> result=dat_filter(s_f,'"source" like "M51" and "offx" eq "161.040" and "offy" eq "139.465" and "wt" gt "0"',/reset) This way we can select the data with the exact pointing with offx (RA offset in arcsec) 161.040 and offy (DEC offset in arcsec) 139.465. Other relational operations (eq, ne, le, lt, ge, gt) can be used to select a range of pointings. 6. If you reduced the data (taken after 2014/11/15) before 2015/02/26, you will have to do the following flagging when loading the data: IDL> res=dat_filter(s_f,'"flags" ne "0"',/reset) IDL> sp[psf].wt=-1 ****NOTE: Since 2015 Feb 26, the above flagging has been added in READDATA. Please report any problem to cqi@cfa.