Processing TRES Spectrograph Data
Jessica Mink, 2018-October-22 2021-April-20

1. When the log,"60logsheet", for a night has been mailed, data taking has stopped.
2. Make sure the night's data has been completely transfered from Mt. Hopkins to Cambridge, check ~/araw/[yyyy.mmdd] . If the most recent file is tres.log or tres.log~ and the last data file is a COMP, BIAS, or DARK, the data transfer is complete. It is transfered at 9:00 am Cambridge time, but in the winter there may still be data until 10:00 am, so wait to process if the transfer is not complete. And sometimes the transfer doesn't work until later. You can also check nflwo60.sao.arizona.edu:/tres to check the raw data there and use sftp to transfer the night if you want to start the pipeline before the automatic transfer has been completed.
3. Run cpraw [yyyy.mmdd] to transfer the raw data for one night from the unmodifiable raw directory to the modifiable one, that is, from ~/araw/[yyyy.mmdd] to ~/traw/[yyyy.mmdd]. This also gives us a second online copy of raw data. Early in TRES' career, its Cambridge raw data disk crashed and it took a full month to recover using backups. This program also inserts RFN, a reduced file number of the form [yyyymmdd].[nnn], into the header of each file and makes tab-separated databases of all raw files and all raw object files. The latter can be searched using the -r option of tresearch.
4. Run the Lars Buchhave's TRES pipeline from ~/idl/tres using reduce_tres [yyyy.mmdd]. This csh script can read from either a list of dates on the command line or a list of dates, one per line, in a file invoked as @[file]. reduce_tres1 is called to run the IDL pipeline by invoking tres_reducex.pro . Raw data is read from ~/traw/[yyyy]/[yyyy.mmdd]/ and reduced data is written to ~/tred/[yyyy-mm-dd]/ in the IRAF .ec format, that is, with a nonlinear wavelength solution in the header. Spectra are cross-correlated against a library of template spectra, and the results with the best template for each spectrum are written to the directory ~/tred/[yyyy-mm-dd]/classifications/ . There are three files for each spectrum.
5. In reduce_tres1, after the IDL processing is complete, the classification/ .log files are collected and run through readlog, a C program which extracts information from the log files, recreates the name of the FITS files which match them, which is a bit tricky because the time in the two file headers is in totally different formats and units. The output is piped into a local tres.db file which is then appended to the master TRES tres.db file. A local tresps.db file is also created for the Postscript graphs of the results, and that is appended to the master TRES tresps.db file.
6. Also in reduce_tres1, the barycentric velocity correction is extracted from the Postscript file containing a plot of the RV31 order and appended to the FITS spectrum header as BCV. The midtime of the observation as Solar System Barycentric Julian Date, extracted from the corresponding log file for tres.db is added to the FITS header as HJDN. HJDN is converted to Geocentric Julian Date and added to the header as GJDN, and as a modified Julian Date of the midtime, MJD-MID.
7. Lastly in the reduction, Kepler objects are extracted from the database into local tresk.db and treskps.db files, and the results are reformatted into a local tres.kfop file for the date, and mailed to Dave Latham and Allyson Beiryla with the subject "[yyyy-mm-dd] TRES data run".
8. Check the results using the trescheck [yyyy-mm-dd] program. It lists the full pathnames of the Postscript files it displays in the window from which you run it. If an image has a bad cosmic ray, capther the pathname for that file and save it in a file of pathnames. I use ~/tred/fix.list
9. If no spectra need to be corrected, run treslist [yyyy-mm-dd] to send email messages containing information about Kepler, K2, K2standards, and TESS spectra and a message noting that no cosmic rays needed to be removed/

   or

   If spectra are corrected, run tresfixset [file of PS pathname] ( or tresfixset1 [PS pathname] ) in ~/tredto set up a [yyyy-mm-dd]fix file with the spectrum to be fixed and the blaze function ( FLAT*fits ) in the repackBlaze/ subdirectory. The original spectrum and its associated classification files are moved to a directory call [yyyy-mm-dd]bad. If there are multiple bad spectra on one date, they are all placed in the same []fix/ and []bad/ directories.

   In ~/idl/tres, run idl, then its apexam task and open the [yyyy-mm-dd]fix/ directory which contains spectra to repair, then open those spectra one at a time. For each, use the cursor to pan and the scroll wheel or arrows to zoom and unzoom. When you have identified the bad pixels, hit "t" to go into pixel mode, and then use the "a" key at each edge of the bad data. Zoom back out so you can make sure that there are no more particle hits, Use the "Save" command in the dropdown File menu to save the spectrum and "Open" the next one. When they are all edited, exit from both apexam and IDL because problems occur if you try to start up apexam from inside IDL when you've already run it once.

   tresupdate [@file] or [yyyy-mm-dd]fix [yyyy-mm-dd]fix [...]fix Reruns classify_tres on the spectra in the ...fix files only. Adds revised entries to tres.db and tresps.db. Runs tresfix [@file] or [yyyy-mm-dd][fix] [yyyy-mm-dd][fix] [...][fix] to run tresfix1 on each directory to copy spectra and classification information from fixed directories back into the archive and move ...fix and ...bad to ~/tred/Fix/ and ~/tred/Bad, respectively. Reruns tresblaze [yyyy-mm-dd] to create addition copies of each spectrum with the blaze function divided out in a [yyyy-mm-dd]/b/ subdirectory

10. Data is now ready to be searched with tresearch about which much more has already been written.