HKQuery operates on a list of HK or SED input files, restricting the data contained in those files according to user-supplied rules and produce a single data file as output.
Input files can be either FITS or HK-ASCII format (the human-readable format created by HKQuery by default). Input files NOT in either of these formats will be ignored. An input file list can be a single file or a list of files. In either case, shell name-completion wild cards may be used. Examples:
Operate only on the file "blah.fit":
HKQuery [rules] blah.fit
Operate on all files in the directory "foo", along with the local file "blah.fit":
HKQuery [rules] blah.fit foo/*
When operating on file lists generated by wild cards, HKQuery will process the files in the same order they are listed using the same wild cards. The following will show the processing sequence in the example above:
ls blah.fit foo/*
The formats of files in an input list need not be the same - they can be a mixture of FITS or HK-ASCII files. HKQuery will read each appropriately, and combine their data to produce its output.
HKQuery can also operate on a file from stdin. Because it ignores
files which don't have formats which it recognizes, HKQuery can be
run on the contents of a directory, regardless of file type. It will only
use those which look like HK or SED data files. Of course the screening is
far from fool-proof, as any FITS table will be assumed to contain HK/SED data.
Time stamps come in two forms - the floating point "SCLK" and the
human-readable "SCET". SCLK represents the number of "clock ticks" of the
instruments since the beginning of January 1st, 1980. Each clock tick is
meant to represent exactly one second, which until launch they do precisely.
After launch, a certain amount of drift is expected, which is tracked and
updated once a week. To translate to the SCET, this drift must be taken into
account (not yet implemented!!!).
All HK programs use the SCLK. The SCET is provided in HK-ASCII
output files and on HK2D plots for the sake of readability.
Output is always a single file, to stdout. By default it is HK-ASCII format, a
human readable table format. If the command line argument "-fits" is
used, the output will be in FITS format. If you are expecting lots of
output data, FITS is far more efficient.
Because it can read its own output, HKQuery operations can be chained
together:
There are three ways to restrict data:
The row restrictions can also be specified by a file name (preceded by
the "@" character). The file contains restrictions following the same
rules as above, although they can be newline separated and can also
contain comment lines (following double slash "//" characters).
Get the first, third, and second to last rows from each of the .fit
files in the directory foo/:
Operate on the same input file set, but restrict to rows as described in
the "row_restrictions.txt" file:
The query is performed using a CFITSIO function "fits_select_rows". The
structure of query rules is described in the document "CFITSIO User's
Reference Guide"(Version 2.4) in section 10.10.
Permitted logical operators include ==, !=, <, <=, =<, >, >=, =>, ||,
&&, !, and ~ (approximately equal to). English language and Fortran
style translations may be used in place of the operators (see the
CFITSIO documentation for details).
Arithmetic operators and functions include +, -, *, /, ** (or ^),
abs(x), cos(x), sin(x), tan(x), arccos(x), arcsin(x), arctan(x),
arctan2(x,y), exp(x), sqrt(x), log(x), log10(x), x % y, random(),
min(x,y), max(x,y), and b?x:y (C-style if-then-else). Once again,
English language translations are available for all operators and
functions. These are described in the CFITSIO documentation.
The query can also be specified as a file containing a list of query
rules. The file name must be preceded by the "@" character. The file
can also contain comments following double slash characters ("//").
Pull the packets where "NASATIME" is greater than zero or less than -100
from all .fit files in directory foo/:
Use the query rules in "query_rules.txt" on the same file set:
Get "NASATIME" and "SCTIME_C" data from all .fit files in the directory
foo/:
Get the keywords specified in "my_keywords.txt" from the same files:
Any, all, or none of these rules may be used in a single HKQuery
operation (although each should only be used ONCE in a single
operation). Row restrictions are applied to input data first, then
query rules, with extraction requests last. The omission of any
restriction rule type will be the equivalent of saying "ALL" - i.e. not
including any row restrictions will include all rows in the output, etc.
Get the "NASATIME" data from all .fit files:
Get the "NASATIME" data only when "SCTIME_C" is greater than 0:
Do the same, but we are only interested in the first and last packet in
each file:
Note that because of the order restrictions are run (row, query,
extract), the following chain of HKQuery would produce the same results:
but this would not:
A file of IRAC "commands" may be merged into the output from HKQuery.
using the "-m" or "-merge" command, followed by the name of the file containing
the commands.
All data HK/SED data packets and commands in a Commands File are accompanied by
a time stamp. These time stamps are used to include commands in the output in
sequence with the data.
It is not possible to merge commands into a FITS output file, nor is it
possible to merge commands into a file without timestamps. For this reason,
using the "-m|merge" option will override both the "-fits|FITS" and
"-t|timestamp" options.
Commands are included in the human-readable HK-ASCII
output files as "events", which can be included in plots (see
Plotting Events in the HK2D
documentation).
The online help system for HKQuery is pretty good, at least compared to
the type of help I normally provide with my software. Help can either
be general, which you get by running with the "-h" or "-help" option
without any further arguments:
or it can be about a specific subtopic:
The subtopics available (presently) are:
Topics generally match the command line inputs, although there are some
additional ones.
Help will be sent to stderr. HKQuery will stop after providing help.
When HKQuery is run, a record of its version, who ran it, when, with
what arguments and with what input files is included in the output (both
FITS and HK-ASCII format). Any of this tracking information found in input
files is also included in the output. Thus it should be possible to
reconstruct all processing steps used to get to a given output file from
information contained within it.
The following is a list of command line arguments and a brief
description of what they do:
Timetags - The SCET and SCLK
HK and SED data packets are organized by time stamps which give the (rough)
time measurements were made. By default, these timestamps are maintained in
an HKQuery output file, although this behavior can be shut off using
the "-t|timestamp" option.
Output
HKQuery [some rules] *.fit | HKQuery [some other rules] > out.dat
Data Restriction Rules
Examples:
HKQuery -r "FIRST, 3, -1" foo/*.fit
HKQuery -r "@row_restrictions.txt" foo/*.fit
Examples:
HKQuery -q "(NASATIME > 0) || (NASATIME < -100)" foo/*.fit
HKQuery -q "@query_rules.txt" foo/*.fit
Examples:
HKQuery -e "NASATIME, SCTIME_C" foo/*.fit
HKQuery -e "@my_keywords.txt" foo/*.fit
Examples:
HKQuery -e "NASATIME" *.fit
HKQuery -q "SCTIME_C > 0" -e "NASATIME" *.fit
HKQuery -q "SCTIME_C > 0" -e "NASATIME" -r "FIRST, LAST" *.fit
HKQuery -r "FIRST, LAST" *.fit | HKQuery -q "SCTIME_C > 0" |
HKQuery -e "NASATIME"
HKQuery -q "SCTIME_C > 0" *.fit | HKQuery -e "NASATIME" |
HKQuery -r "FIRST, LAST"
Merging Commands Files
Online Help
HKQuery -help
HKQuery -help [subtopic]
debug
error
extract
FITS
filelist
help
merge
query
rows
Process Tracking
Summary of Command Line Arguments