#----------------------------------------------------------------------------- # IRACproc: IRAC Post-BCD Processing v4.3 PROCESSING #----------------------------------------------------------------------------- (c) 2005 Mike Schuster, Massimo Marengo (mmarengo@cfa.harvard.edu) (c) 2005 SAO - Smithsonian Astrophysical Observatory Harvard-Smithsonian Center for Astrophysics Smithsonian Astrophysical Observatory 30 JUL 2008 * It is assumed that your IRAC BCD products were taken from the archive June 4, 2004, or later. If you have data that is older than this, you can get the latest pipeline calibration by downloading the data again. * The SSC has improved the WCS pointing information in the headers of the BCD frames beginning with the S13.2 pipeline. We recommend that you always use the latest pipeline BCD data. We have therefore set as the default for IRACproc 'USE_REFINED_POINTING = 0' when running MOPEX (i.e. off), as this uses the so called 'super-boresight' pointing from the SSC rather than the less accurate refined pointing for each IRAC band. When processing data from previous versions of the pipeline (S12 and earlier) make sure to switch on the 'USE_REFINED_POINTING = 1' option when running MOPEX (i.e. on), as there was no super-boresight pointing for these pipelines. This logical switch can be found in three configuration files located in the Lists/ subdirectory created after running the fixup routine in Processing #1 below, and should be changed prior to proceeding with Processing #4. The files are fif_tmpl.nl, mosaic_tmpl.nl, and overlap_tmpl.nl. Creating mosaics of your IRAC data with IRACproc is a 3-step process: step 1: Initialize SSC delivered AOR data and create customized configuration files. step 2: Create master image lists for processing (one per channel). step 3: Create mosaics. For additional information on the IRACproc scripts: Documentation for IRACproc scripts can be found in IRACproc/docs/scriptname.html To view options for perlDL scripts type 'pdlscriptname()' at the perldl> prompt. PerlDL script options are specified with the following syntax: OPTION_NAME=>'option_value'. Strings must be specified within quotes ''. To view information files for perlDL scripts type 'help pdlscriptname' at the perldl> prompt. To see options for perl scripts type 'scriptname -h' at the command prompt. Each new run of IRACproc will clobber all intermediary data products for that channel. #----------------------------------------------------------------------------- # Processing Step #1 - Data Initialization #----------------------------------------------------------------------------- Run the script fixup to create the proc_dir directory structure and prepare files for IRACproc. "proc_dir" is where you will process your data. Your working directory will be: //proc_dir/reduce/. Go to the location where all of the processing work will be done. user_name][ cd // Form 1: Make a list (aor.lis) of related AOR directories (i.e. for the same field) to be processed with a single run (relative as well as absolute paths are OK). user_name][ fixup -pmasks mmmyy -aorlist aor.lis proc_dir Form 2: You can also list each AOR (w/ path) on the command line. user_name][ fixup -pmasks mmmyy proc_dir aor1 aor2 ... aorN The script fixup takes several options that should aid with setting up your data for processing. This is a short list of parameters that are likely to change for individual programs. See IRACproc/docs/fixup.html for additional information. See IRACproc/pmask/pmasks.README for details on the SSC Pmasks. USAGE: perl fixup.pl [--help][--ver][--aorlist file] [--link][--bcdpath string][--prefix string] [--pmasks mmmyy][--subarray] proc_dir [aor1 aor2 ... aorN] To reset the configuration files in an existing IRACproc processing directory: ------------------------------------------------------------------------------ user_name][ cd // user_name][ refixup -pmasks mmmyy proc_dir USAGE: perl refixup.pl [--help][--ver][--pmasks mmmyy][--subarray] proc_dir For programs that have many AORs - with one AOR per field: ---------------------------------------------------------- Use the script multi_fixup to run fixup in batch mode. The file aor.lis should be a list of unrelated AOR directories (one AOR per field). The proc_dir/ directory for each AOR will be given the name "Target Name" as listed in any one of the README files in /ch/bcd/. See IRACproc/docs/multi_fixup.html for additional information. user_name][ multi_fixup -pmasks mmmyy -aorlist aor.lis USAGE: perl multi_fixup.pl [--help][--ver][--aorlist file][--link] [--pmasks mmmyy][--subarray] #----------------------------------------------------------------------------- # Additional Processing Step For IRAC Sub-Array Data ONLY #----------------------------------------------------------------------------- If you are processing Sub-Array data, you will need to perform one of the following additional steps before continuing. Method 1: Preferred. Coadds each image pack. To view options type 'subcoadd_bcd()' at the perlDL prompt. To view information file type 'help subcoadd_bcd' at the perlDL prompt, or see IRACproc/docs/subcoadd_bcd.html for additional information. user_name][ cd proc_dir/reduce/ user_name][ perldl perldl> subcoadd_bcd({CH=>1}) perldl> subcoadd_bcd({CH=>2}) perldl> subcoadd_bcd({CH=>3}) perldl> subcoadd_bcd({CH=>4}) Some typical parameters for coadding sub-array image packs: CH IRAC channel: '1', '2', '3', '4' PMASKS Set of Pmasks, as distributed by the SSC, specified by month and year, available options: jun06, dec05, jun05, feb05, jan05, oct04, jun04, mar04 SIGMA Number of sigmas for outlier rejection (default 5) Note: This routine also places the original _bcd.fits, _bdmsk.fits, and _bunc.fits image packs in .bcd_pack_tmp/. Method 2: Very slow. This method should only be used if you really need each layer of a sub-array image pack separately. To view options type 'subsplit_bcd()' at the perlDL prompt. To view information file type 'help subsplit_bcd' at the perlDL prompt, or see IRACproc/docs/subsplit_bcd.html for additional information. To slice the image packs into individual images: user_name][ cd proc_dir/reduce/ user_name][ perldl perldl> subsplit_bcd({CH=>1}) perldl> subsplit_bcd({CH=>2}) perldl> subsplit_bcd({CH=>3}) perldl> subsplit_bcd({CH=>4}) Note: This routine also places the original _bcd.fits, _bdmsk.fits, and _bunc.fits image packs in .reduce_tmp/. #----------------------------------------------------------------------------- # Wrapper/Log For Batch Processing Steps 2 and 3 #----------------------------------------------------------------------------- After running 'fixup' in step 1, there will be a perlDL wrapper file called 'my_iracproc.pdl' in the reduce/ sub-directory. This is useful for batch processing steps 2 and 3 for all 4 IRAC channels at once, and will remain a log of your data processing. Simply use a text editor to modify the file accordingly for each data set. Then, 'cd' into reduce/, start perlDL and type 'my_iracproc()' at the prompt. A duplicate of the original, un-modified file is also placed in the Lists/ sub-directory. #----------------------------------------------------------------------------- # Processing Step #2 - Make Master Image Lists #----------------------------------------------------------------------------- Change dir to the target that you wish to process and start PDL. To view options type 'mkmopex_bcd()' at the perlDL prompt. To view information file type 'help mkmopex_bcd' at the perlDL prompt, or see IRACproc/docs/mkmopex_bcd.html for additional information. user_name][ cd proc_dir/reduce/ user_name][ perldl Make the initial list of images to be processed: perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>1}) perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>2}) perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>3}) perldl> mkmopex_bcd('../Lists/list.par',{FOV=>'all',FT=>30,CH=>4}) Some typical parameters for selecting images (matching is exact): FOV field of view: 'on', 'off', 'sub', or 'all' (default) CH IRAC channel: '1', '2', '3', '4' FT frame time (default 'all') XT exposure time (default 'all') AO AORID (default 'all') WCS_COORDS WCS coordinates (deg) where images are centered - [RA,DEC] WCS_SIZE WCS box size (angular width in deg) where images are located, automatically pads region to include all images that may have pixels within selected region - [RA_SIZE,DEC_SIZE] Each command produces the file Lists/bcd_I.lis. You may remove images from the processing by removing them from the image list: Lists/bcd_I.lis (this can be automated by following the instructions in HOWTO_INSPECT). You may alter the BCD frames with additional preprocessing, however, two guidelines must be followed: 1) the image names must be identical to those originally placed in Lists/bcd_I.lis, and 2) the image list bcd_I.lis must retain its name and location. #----------------------------------------------------------------------------- # Processing Step #3 - Make Mosaic Images w/ Outlier Rejection #----------------------------------------------------------------------------- Create a mosaic of the images in Lists/bcd_I.lis. To view options type 'mkmosaic_bcd()' at the perlDL prompt. To view information file type 'help mkmosaic_bcd' at the perlDL prompt, or see IRACproc/docs/mkmosaic_bcd.html for additional information. perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>1}) perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>2}) perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>3}) perldl> mkmosaic_bcd({NAME=>'',MASTER_FIF=>0,STEPS=>[1,1,1,1,1,1],CH=>4}) Quit PDL - you will need to restart for the next target. perldl> quit By default, copies of the final mosaics and coverage maps are placed in analysis/ with the filenames: I__mosaic.fits and I__mosaic_cov.fits. ---------------- Some typical options for making mosaics: NAME The string placed in the default output image filenames I__mosaic.fits and I__mosaic_cov.fits - default 'IRACproc' CH IRAC channel: '1' (default), '2', '3', '4' STEPS Set of six logical flags to turn on/off stages of processing ALL SIX STEPS MUST BE RUN AT LEAST ONCE - [STEP1,STEP2,STEP3,STEP4,STEP5,STEP6] - defaults are 1 (yes) STEP1 = run overlap correction STEP2 = create median mosaic STEP3 = run 1st layer outlier rejection STEP4 = create metric images STEP5 = create final mosaic (w/ 2nd layer outlier rejection) STEP6 = copy final mosaic to analysis/ subdirectory # FIF options: MASTER_FIF Set this logical flag to use a master Fiducial Image Frame no = 0 (default), yes = 1 (by default created from bcd_I1.lis, bcd_I2.lis, bcd_I3.lis, and bcd_I4.lis) FIF_FILE Set this to use an existing master Fiducial Image Frame (the default location/file is ../Lists/FIF.tbl) FIF_IMG_LIST Set this to use an existing image list for creating the master Fiducial Image Frame (the default location/file is ../Lists/bcd_ok.lis) CROTA2 Position angle of N in final mosaic, CCW from +y-axis 'A' (average, default), '0.' for North up PIXEL_SIZE Mosaic pixel size in arcseconds (default is 0.86267016) # First layer outlier rejection options: CR_FLAG Set this logical switch to flag outliers in BCD Dmask images (_bdmsk2.fits), otherwise routine only clears bit 9 in Dmask - default = 1 (yes) To remove the unnecessary files in an IRACproc processing directory: -------------------------------------------------------------------- user_name][ cd // user_name][ cleanup dir USAGE: perl cleanup.pl [--help][--ver][--dirlist file][dir1 dir2 ... dirN] ---------------- Higher Coverage Mosaics: ------------------------ If your data set is such that the depth of coverage is at least 5 frames at every sky location, you may choose to skip the intermediary steps that perform a first pass at outlier rejection in order to expedite processing by setting CR_FLAG=>0. If however your data set has two epochs of observation (e.g. at different orientations), then you will want to run the the first layer of outlier rejection since this layer is designed to work with bimodal distributions in the pixel stack (CR_FLAG=>1, default). Low Coverage Mosaics: --------------------- The ability of the first layer of outlier rejection (crflag_bcd) to effectively remove transients is dependent on the pointing refinement available for each channel. This is typically much worse for channels 3+4, and can negatively affect coverage on-source in shallow coverage mosaics if the pointing refinement is not used (or not very good). The S13.2 pipeline has significantly improved the pointing refinement for channels 3+4, and IRAC users should download a new copy of their AORs processed with the latest version of the pipeline. Maps (w/o dithering) & Latents: ------------------------------- If your data set is a map that was produced by stepping the array by a large fraction of a frame without dithering the telescope in between steps, then you may want to 'turn off' the masking of latents (bit 5 in the Dmask). The short term latents of bright stars are picked up by the SSC pipeline processing and flagged in the Dmask (bit 5) of subsequent frames. The repetition of the mapping causes these short term latents to overlap in sky coordinates before they have time to fully decay, and so they will literally punch holes in the final mosaic. This effect is caused by the absence of dithering, and is not a consequence of shallow coverage depth; that is, repeating each map position many times will not by itself prevent these overlapping latents. This happens irregardless of any outlier rejection, in other words, adjusting the outlier rejection thresholds will have no effect. This can be avoided by modifying the following parameter files, in Lists/: mosaic.par, overlap.par, median.par, and minabs.par. You would need to change the DCE_Status_Mask_Fatal_BitPattern by subtracting 32 (=2^5). Background Gradients In Large IRAC Mosaics: ------------------------------------------- Most, and possibly all, large IRAC mosaics suffer from significant, artificial background gradients, affecting the measured fluxes and colors for extended emission in all bands. This effect most likely arises from small zodiacal background gradients over the IRAC array, or imperfect flat-fields and geometrical distortion corrections. Although automated background matching is performed as part of the IRACproc processing, it only corrects for constant pedestal offsets for each frame. As a result, small gradients build up over large mosaics manifesting as a position dependent zero-point. To remove the background gradients a surface can be subtracted from each mosaic. The surfaces can be constructed by trend-fitting the pedestal offsets for each frame, as output by the MOPEX overlap correction module. The notes located in IRACproc/docs/BKGFIX_GRAD_NOTES describe one method for finding and removing the gradients in a way that does not add or subtract any net flux from the mosaic. In addition, the notes located in IRACproc/docs/BKGFIX_NOTES describe one method for calculating and subtracting a dark frame from an IRAC mosaic data set. There is also a method for fixing erroneous offset corrections from MOPEX's overlap correction module around extremely bright stars. Bright Stars: ------------- If you have a very bright star (e.g. 1st mag, or brighter) and have only a single IRAC field-of-view, then set BACKGRND=>0. No Dithering/Mapping: --------------------- If your data is neither dithered nor mapped, then DERIV_SCALE should be set to a low value, for example <0.1. Adjusting Outlier Rejection: ---------------------------- If you find that outlier rejection has not been aggressive enough, then reduce the options SCALE_THRESH and/or SCALE_CR_THRESH and rerun mkmosaic_bcd. Be mindful of the coverage of the final mosaic (in the sky background, as well as on top of point sources) - check analysis/I__mosaic_cov.fits. If you find that outlier rejection has been too aggressive, then increase the options SCALE_THRESH and/or SCALE_CR_THRESH and rerun mkmosaic_bcd. Persistent Transients: ---------------------- The mosaic.fits, mosaic_cov.fits, median.fits, and absolute_minimum_mosaic.fits (if created with CR_FLAG=>1) files can be checked and compared across channels for suspected transient events that persist into the final mosaic image. The final mosaic and coverage images are located in: mosaic_I/Combine/ The median image is located in: median_I/Combine/ The absolute minimum mosaic image is located in: minabs_I/Combine/ ---------------------- You may notice the following warning scroll on the screen: mkmopex_bcd: WARNING cannot find file I_########_####_0_1_bmet.fits This is normal when running IRACproc for the first time on a set of images.