|FAST Reduction Manual
August 15, 2003
January 22, 2008
There are also a number of distribution programs written by Bill Wyatt that have an adjunct role in the FAST pipeline. These programs are used in two main ways. The first is as a filter to screen and distribute the night's raw data that will not be reduced by roadrunner. The second use of the distribution programs is to distribute the reduced 1-D and partially reduced (up to extraction) 2-D data at the end of each FAST run. A full description of the distribution programs and their parameters can be found at: /home/wyatt/mine/Hectospec/Scripts/Usage.
After each night's data have been processed, the reduced 1-D files
are copied to the FAST archive and then entered into the FAST
database, Fast.db. The raw data is automatically written to tape
at the end of the day.
Summary of roadrunner's programs/functions:
|sorting:||sorts the data by binning, aperture, tilt position and dispersion. roadrunner was originally set up to handle all dispersions, apertures and binnings, but in practice it does not, primarily because HENEAR templates have not been made for every possible combination of settings . Thus roadrunner, at present, is used only to automatically reduce data taken with the 300 l/mm dispersion with binby2 or binby4 binning, apertures of 1.5", 2", 3" and 5" and tiltpos of 600 +/- 10. This covers the bulk of the 300 l/mm data taken with FAST. The roadrunner program does not separate similar sets of data taken before or after a grating or tilt change even if they should be reduced in two or more groups.|
|zerocombine and imstat:||combines the bias files, and runs them through imstat|
|ccdproc:||runs fixpix to interpolate over the two bad columns at pixels 1659 and 2422, then trims the files, subtracts the overscan region and subtracts the combined bias frame. This is not used with FAST III - ccd installed in spring 2006.|
|darkcombine and imstat:||combines darks, runs an imstat to check the mean. If the dark mean is above a certain number set in the roadrunner parameter file, the combined dark file is subtracted from the other files.|
|flatcombine:||flats are combined, run through imstat to see if there are any inconsistencies|
|response:||takes combined flat file, fits the spectrum, divides the flat by the fit producing a file, norm.fits, that will be used to correct for the pixel to pixel variation.|
|flatcorrect+; divides the program and comp files by the norm file.|
|reidentify:||takes a previously identified comp, HENEAR_TEMPLATE, and uses it to reidentify the lines in each comp. Reidentify is run in 2-D so you have a 2-D wavelength solution|
|fitcoords:||takes each wavelength calibrated comp and computes the functions that would make the lines straighter on the slit.|
|transform:||assigns comp to object so the object will have a wavelength calibration and straightens the object on the slit using the functions computed for that particular comp. Output has TF suffix, 0000.<name>TF.fits.|
findall:||finds the spectrum on the slit and determines background|
|apall:||traces the object and extracts the background-subtracted, one- dimensional spectrum; output is 0000.<name>.ms.fits|
|xcsao:||runs the xcsao program on the ms.fits files to find velocity|
|qplot:||quality check; allows the user to inspect and evaluate the velocity measurement|
|gather or gather.raw:||gathers the data into a database table or tables|
|greport:||summarizes the data by program numbers|
|distribute:||copies the data to the distribution directories|
|inform:||emails a script to the P.I.'s telling them how to pick up their data|
First, log in to the fast account.
After each night, data are automatically transferred from Mt Hopkins to Cambridge via cron jobs run on the tdc2 computer. This transfer often can not be complete until 11:30 a.m. EST, depending on the time of sunrise at Mt Hopkins and whether DARK exposures were taken at the end of the night.
The final automatic transfer is not made until just after 12:00 pm MST so that all files are guaranteed to be stored in the pervious night's data directory. If you are reducing FAST data before this time, it is prudent to manually run the transfer program to be sure all data is in Cambridge:
fastmirror -off -1If all data has been transferred, you will see a line beginning:
No files left to get from ...The data are copied into:
/data/fast/rawdata/<year>.<month><day>(for example : /data/fast/rawdata/2003.0925)
where year is four digits, month and day are two each.
Before reducing the data, cd to the transfer directory, for example, in /data/fast/rawdata/2003.0925, and type:
mkfastlogThis will look at the prior camblog.ps from the previous available night to extract the last log page number. Then, it reads the fastlog.lis file, taking comments and writing them to the appropriate files. It lastly generates the new camblog.ps file for your inspection. Note especially any files named "test" or "focus" or ones with special comments regarding the object's row on the CCD.
gather.rawThis will create 4 database tables and show how many objects are in each:
If the fast_rawscan.rem.db has no objects, great: you are finished in terms of raw data distribution, and can go on and reduce that night's data in the reduction directory.
fast_rawscan.db all files fast_rawscan.rawdist.db files to be distributed fast_rawscan.rem.db files to be removed because they are not standard reductions fast_rawscan.reduce.db files to be reduced
If, however, there are files in the fast_rawscan.rem.db, you
will have to distribute them. You might actually want to look
at the files and can do so by typing:
vurawscan -b < fast_rawscan.rem.db | moreThe -a and -b keys simply give you slightly different output reports. Another key, -h, gets one or more extra keywords, for example,
vurawscan -a < fast_rawscan.rem.db | more
puts the P.I. and Program in the report.
-h "pi program"
To move the files to the directory from which the P.I. can pick them up, type:
distrib.raw -i fast_rawscan.rawdist.db
This puts them in dated subdirectories in /data/fastdist/distribute/fastraw/ ProgNNN/<year>.<month><day>/ where NNN is the program number. You will note that fast_rawscan.rawdist.db has more objects than does fast_rawscan.rem.db. That is because the rawdist files include BIAS and DARK frames as well as photometric and velocity standards. Susan usually edited this file to remove the standards and anything else she knew the P.I. did not need. However, it was not intended to be edited and it is not expected you to edit it. Once the files have been placed in the raw data distribution directory, you can notify the P.I. that they are there to be picked up. You do this with the command inform "-r date-range", for example,
inform -r 2003.0925Raw data are distributed daily so it is more typical to use just one day rather than a range.
inform -r 2003.0925-2003.0927
The inform command creates and sends email that contains a script which will allow the P.I. or her delegate to copy the data to her local disk. The scripts are created because the program distribution directories, with the exception of velocity and photometric standards, are closed to the P.I.s. This was done to maintain security without assigning a different group name to each separate program. This means the P.I. cannot go to the data distribution directories, type an "ls" and see what is there. However, if the P.I. knows the path name, which includes file name and number, she can pick up the data. Thus, a list of the files with pathnames is included in the email sent by the inform program. The raw data files remain on the distribution disk in this location for a period of 6 months before they are removed. The email sent to the P.I. notes this fact.
The list of program numbers, names, P.I. and the person or people to whom the data will be distributed are kept in the fastprogs.db file, which is currently located in /home/fast/progdb. Note that the fastprogs.db file is a tab-separated file. It will occasionally need editing, usually to add new programs. If the program number and name do not match the P.I. listed in fastprogs.db, gather.raw will fail. It will produce an error file and a fix file. Often it is easiest to edit the error file to fix the headers. Whatever you edit, your fixfile should end up with lines looking something like:
sethead 0151.sn2003gh.fits P.I.="Kirshner"Then you type:
sethead 0151.sn2003gh.fits PROGRAM="#2 SN Observations"
chmod +x fixfileOnce you have fixed the files so the headers are correct, you should delete all the fast_rawscan.*.db files and run gather.raw again.
Before leaving the transfer directory and when there are files to be reduced by the P.I., create a listing of those file names from the fast_rawscan.rem.db. To do this, type:
headoff FILENAME < fast_rawscan.rem.db > remlistYou will copy this list, called remlist, to the reduction directory when it is created.
Summary of the gather commands for raw data:
vurawscan -b < fast_rawscan.rawdist.db
distrib.raw -i fast_rawscan.rawdist.db
inform -r date range
eclThis is an alias that will put you into the correct IRAF login directory and start up IRAF with roadrunner loaded.
mkdir <year>.<month><day>for example, mkdir 2003.0925; mkdir can be run either in or outside of IRAF.
cd reddir$2003.0925 orand copy the raw data into that directory. I prefer to do this in a window outside of IRAF with the command:
cd ~/ared/2003.0925 (if outside IRAF)
cp /data/fast/rawdata/<year>.<month><day>/*.fits .But you can use the imcopy command instead of cp if you are in an IRAF window.
Also, copy the remlist (files to be removed) for that night with the command:
cp /data/fast/rawdata/<year>.<month><day>/remlist .Then, in the IRAF window, but same subdirectory, remove the files that will not be reduced with the command,
gethead *.fits aperture disperse tiltpos naxis2 | more
gethead *.fits aperture disperse tiltpos naxis2 > indata
roadr > road.logThe > road.log diverts the roadrunner output from the screen to the file, road.log, so you have a log of the reduction process.
tail -f road.logThis allows you, if you wish, to see what is happening. There is no graceful exit from this program; you have to Control C out of it after roadrunner has finished in the IRAF window Be careful that you are not in the IRAF window when you use Control C as you don't want to interrupt the roadrunner program. You can also check road.log after it has finished running by typing:
more road.logIn checking this log there are a number of things to look for:
imdel 0123.nameTF,and then run transform with the correct comp:
transform input=0123.<name> output=0123.<name>TF fitnames=0124.COMP
imcopy *TF.fits ../or alternatively, in the reduction directory, above the set directories, type:
imcopy set1dir/*TF.fits .This allows the one dimensional spectra for a night to be extracted together, in the same directory. It makes archiving and distribution significantly easier.
The qplot plot marks the absorption and emission lines for the velocity of the best template. You will notice that we run the object against ten templates and the template with the highest r value is the one chosen. There are times when you may wish to force the velocity to a different template or to the results of the emsao program. To force a different template, type t and then the number of the template you want, such as 3. The templates are listed from highest r value to lowest and the number of that sequence is the number to use when you want to select a different template. If you like what you see, type s and then x. The x stands for correlation and this will put your chosen correlation velocity in the header keyword, VELOCITY.
The emission line program, emsao is a program to find a velocity based on emission lines only. I seldom run emsao because the emission line template, femtemp, will most likely have the highest r value when there are good emission lines in the spectrum. Unlike the correlation program run by xcsao that matchs the Fourier transform of the templates against the Fourier transform of the object, emsao actually fits the emission lines and comes up with a final velocity based on the velocities of the lines. You may want to run emsao when emission lines are present, but are weak and not marked. In such a case, you would either want to confirm that the emission line velocity matchs the absorption line correlation velocity or find an emission velocity if the correlation velocities aren't correct. Sometimes too, the femtemp template has an incorrect velocity because it has mistaken the N line, at a resting wavelength of 6583 Angstroms, for the H Alpha line, with a resting wavelength is 6563 Angstroms.
To force the emsao velocity to be the final velocity, type s and then e.
scombine 0124.<name>.ms,0125.<name>.ms 2425.<name>.sumIf you have a summed file, make sure you add it to the outfile list created by beepbeep by editing outfile. I don't combine everything that has more than one exposure. If both have high r values, I usually don't bother; if one is terrible, but the other is ok, again I don't do it. However, if both are marginal, say with r values in the 4 range, you might be able to get a better velocity by combining them and then running xcsao on the combined file, above, 2425.<name>.sum.fits. You should also run hedit on the new file 2425.<name>.sum to change the title to <name>.sum. As you have noticed, we don't have a good numbering system for combined files. I try to use some part of the file numbers that are combined; nevertheless you should be aware that the keyword, DISKFILE, will be that of the first file to be combined, here 0124.
gethead -ht -n 2 *ms.fits velocity czerr czxcr velqual besttemp | justify > zoutThis creates the file zout which is a list of the objects you reduced with their velocities, errors, r-values, quality check, and template used. Among the things I look for are:
Next, outside of the IRAF window, I make a hard copy of zout by typing:
enscript -d(printer) -r zoutI don't know if you will want to do this, but if so, this is not a list that should be casually shared; it really is for your use only. FAST data are proprietary and we have to take the security of that data seriously. I suggest, instead, that you create a directory owned by fast where you keep the zout files. Permissions on this directory, and the files within it, should be set so that only user fast and group fast can read it.
splot 0000.<name>.msand then hitting the ) key. The ( key takes you in the other direction through the four spectra.
After you have finished with the night's reductions you will want to copy the .ms.fits files to the FAST archive. The archive currently is located on /data/mc4/fast/archive/ and the files are stored by night, each night's directory having the form, <year>.<month><day>, for example, 2003.0925.
There are scripts set up for the transfer. In the reduction directory, e.g. /data/mc4/fast/reductions/2003.0925/, type
s1 = datefor example,
s1 = 2003.0925Then type
cl < ../mvlistThis creates that night's directory in the archive and copies the data into it. It also copies a few other files that may be helpful if someone ever wants to look back and see how the data were reduced. Of particular importance is the database directory.
If you have reduced the data in 2 or more sets, you will also want to copy the database and log files created by the roadrunner program in each set directory to the final archive. So, after you have done the above transfer, go into each of the subdirectories to copy those files to the archive. Go into the first subdirectory and type:
cp /data/mc4/fast/reductions/mvlistold .Edit the mvlistold so that it has the correct date. Then type:
cl < mvlistoldThen go into the other subdirectories and do the same thing, editing the set1 designation (to set2 for example) so each logfile will have a unique name and won't overwrite the logfile you copied earlier.
Once the data are in the archive, go to that file and do a quick count of the fits files to make sure all of that night's data really got transferred into the archive. After that check, cd to the directory above, /data/mc4/fast/archive/. Run chmod to set permissions:
chmod -R 750 2003.0925 (for example)The -R, which is recursive, changes the permissions on the directory, in the above example, 2003.0925, as well as on the files within it.
Next, copy the data from the archive into the Fast database. To do this, first type:
scanfast 2003.0901-2003.0916 (date range of run)This creates a database file, called Scanfast.db, and a summary of the files for that night by p.i. and program number are shown on the screen. If this looks ok, you are ready to put the night's data into the FAST database, Fast.db. If it doesn't, you might need to fix p.i or program number and then run scanfast again.
To enter the data from Scanfast.db into Fast.db, type:
addscanThis takes a few moments to run because, in addition to adding the files to the database, it is indexing them by ra, dec, title, etc. for faster retrieval. The program will show you the number added and the total number of files in the database.
Finally please note that the FAST archive and the FAST database
files are not open to anyone other than the fast group members;
the security of this data is a high priority. Also, while the
database can be rebuilt from the FAST archive, the FAST archive itself
is the final storage for reduced FAST files. The Computer Facility
automatically backs up the disks where these files reside; I do not
make a backup.
At the end of a run, the .ms.fits and TF.fits files are distributed to a disk where they can be picked up by the P.I.s This is done by the same set of programs that make distribution of raw data to P.I.s easy.
I'll use concrete examples but the input, set with -i, and output, set with -o, names are arbitrary and the date range would actually be the date range of the run.
Go to the archive file, /data/mc4/fast/archive/, and type, for example:
gather -o gather.sept03.db 2003.0901-2003.0916This creates the database file "gather.sept03.db" Of course you may name the output file whatever you want, but as this file will eventually be moved to: /data/mc4/fast/archive/gatherdb.dir/ to keep a record of what was done, it is important to differentiate the runs by date.
greport -i gather.sept03.db -o gatherlog.sept03This gives you a summary of what you have already seen from running gather but in a form you can easily keep and this file will be moved to /data/mc4/fast/archive/gather.log.dir/
distribute -i gather.sept03.db -1The -1 on the distribute command line puts the data into the fast1d directory which is:
The other distribution directories are:
inform -1 2003.0901-2003.0916which sends, to every p.i. with data from the run, a script with a list of the files, with full path names, to enable each p.i. to copy her reduced data.
Go to your reduction home directory, /data/mc4/fast/reductions/ and repeat the above with the following modifications:
gather -X TF.fits -o gather.sept03.db 2003.0901-2003.0916
greport -i gather.sept03.db -o report.sept03
distribute -i gather.sept03.db -2 (for 2-D)
inform -2 2003.0901-2003.0916