A report on handling large size SMA data in Miriad :

It now takes 14.6 min to load 20 Gbytes data and 46.9 min for 50 Gbytes.
With the partially updated Miriad (SMA beta 1.4.5 + Miriad_CVS 4.3.4), we
can now handle large-size SMA data with the basic uv tasks - smalod,
uvindex, uvflag, smauvplt ... on a standard linux desktop computer.

1. Test data files from recent testing SMA runs

120505_02:52:05
integration 1.3 sec
# of int    13983
# of bsln   28
# of chan   6145
# of rx     1
size        20 Gbytes

120813_03:37:15
integration 1.3 sec
# of int    48536
# of bsln   10
# of chan   6145
# of rx     2
size        50 Gbytes

2. Computer used

• 4 x processors:

processor	: 
vendor_id	: GenuineIntel
cpu family	: 6
model		: 44
model name	: Intel(R) Xeon(R) CPU           X5687  @ 3.60GHz
stepping	: 2
cpu MHz		: 3591.170
cache size	: 12288 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht 
tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq 
dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt aes lahf_lm ida arat epb dts tpr_shadow vnm
i flexpriority ept vpid
bogomips	: 7182.34
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

• Memory

free -m
             total       used       free     shared    buffers     cached
Mem:         23956      15682       8273          0        363      14282
-/+ buffers/cache:       1036      22919
Swap:        26207        176      26031

• Linux OS

Red Hat Enterprise Linux Workstation release 6.3 (Santiago)

• Miriad software

A hybrid test version: Miriad_CVS 4.3.4 and SMA beta 1.4.5 

3. Summary of test

• Result on 120505_02:52:05 (20 Gbytes)

It takes 14.6 min to convert the data into Miriad with smalod
with no errors from the Miriad uvio routines.

• report from smalod
• report from uvindex

• Result on 120813_03:37:15 (50 Gbytes)

It takes 46.9 min to convert the uv data into Miriad with smalod
by unable calling the relevant uv-routines for a few bad variable 
strings in the data which cause segfaults in Miriad uvio subroutines.

• report from smalod
• report from uvindex
• uv plot from smauvplt

• Errors from 120813_03:37:15 (50GB) file


• Errors reported from smalod

• Error in parsing bsl header: 10 bls reported are inconsistent with 21 bls
  in the visibility plot.
  
  ...
  

   Jun-Hui Zhao  (2012-09-15)

Astrophysicist at SAO
60 Garden St., MS 78
Cambridge, MA 02138
Tel:   617 496 7895
Email: jzhao@cfa.harvard.edu