Automating Reduction of Multifiber Spectra
from the MMT's Hectospec and Hectochelle

Douglas J. Mink, William F. Wyatt, Nelson Caldwell, Maureen A. Conroy, Gabor Furesz, Susan P. Tokarz
Smithsonian Astrophysical Observatory, Cambridge, MA 02138


Hectospec, the MMT's low resolution, 300 fiber spectrograph, posed some interesting and unique problems for reduction software development. To address these issues as well as the usual ones encountered in a multifiber spectrograph, we built a semi-automatic reduction pipeline, SPECROAD, using host IRAF CL scripting. The pipeline is based on existing IRAF tasks, modified when necessary, with some locally-developed procedures added. For quality control purposes, some human interaction was kept in the process. The pipeline takes raw data and produces one-dimensional, wavelength calibrated, sky subtracted, and velocity correlated spectra that are automatically distributed to PIs and added to a database. We are currently developing a pipeline for the 240-fiber Hectochelle high-dispersion spectrograph and we will discuss the additional issues which have complicated its data reduction.


Hectospec and Hectochelle are multi-fiber spectrographs with robot-controlled fiber placement. They are mounted on the MMT 6.5-meter telescope and used in its f/5 mode, with a field of view one degree in diameter. Hectospec (Fabricant et al. 2005) is low-resolution, with 300 fibers covering 3700-9200 angstroms at 1.2 angstrom/pixel using its 270 line/millimeter grating. Hectochelle (Szentgyorgyi et al. 1998) is high-resolution, with 240 fibers at 0.03 angstrom/pixel. Both have dual-chip/dual amplifier detectors giving four readouts per exposure. An earlier paper (Mink et al 2004) described the entire pre-observation to archive process; here we will describe how raw data from the telescope is put into a scientifically useful form.

IRAF in a Modern Pipeline

We used the IRAF CL scripting language as the basis for this pipeline because of our experience using it for the FAST Spectrograph and other data reduction problems over the years. IRAF's pre-existing tasks for extracting spectra from 2-dimensional images (Valdes 1991) were easily modified to do what we needed. By remaining in IRAF, our own existing IRAF SVDFIT package (Mink and Kurtz 2001) for background removal and XCSAO package (Kurtz and Mink 1998) for redshift cross-correlation could be reused. IRAF's cross-platform compatibilities have meant that processing could be moved between Solaris and Linux fairly easily.

Hectospec Procedure

The SPECROAD master script first sorts the raw data files. Calibration files, including bias and dark frames, comparison lamp exposures, and sky and dome flat field images are processed using IRAF mscred.ccdproc, removing overscan, dark counts if necessary, and trimming. After merging the four amplifiers of each multiextension file into single images, each set of dome, sky flat and comparison files is combined into a single image, removing cosmic rays at the same time. Bias and dark images are examined but usually not used. Image overscan regions provide a better measure of their bias level, and dark current is very low and stable. Each task adds information to the image header keeping an image from being processed twice and allowing processing to be resumed at any point.

The spectrograph is so stable that only one set of contiguous calibration lamp exposures are taken per night. Small adjustments in wavelength calibration are made for each object by calculating shifts in the night sky lines and applying that shift when the extracted spectra are linearized. Comparison wavelength solutions from previous runs can be used as reference spectra to get an automatic wavelength solution for comparison spectra on a given night.

SPECROADCAL creates a flat field using the IRAF apextract.apflatten task to flatten the combined calibration lamp image, and 300 flat field and calibration spectra are extracted from the images. From this point on all operations are on 300 separate spectra, though they reside in a single IRAF multispec file. Dispersion functions are fit to all of the fibers, usually automatically based on earlier fits. Linear spectra of a flat field pointing, usually of the twilight sky, but sometimes of a reflective panel on the dome, are linearized using the dispersion functions fit above. Throughput correction spectra are made by ratioing the spectrum from each fiber to the mean of several fibers so that the color and response of fibers can be compared.

SPECROADOBJ processes object spectrum images in the same way that calibration images are processed through the wavelength linearization step, except that multiple exposures of the same pointing are not combined until after they have been linearized in wavelength. The combined spectra are then throughput corrected, sky absorption at 6800 and 7600 Angstroms is divided out, and extra flux at the red end of the spectrum due to an LED source in the fiber positioner in the instrument is removed. Spectra are then split into separate files, and the SKYPROC program removes the sky background from each object spectrum individually.

Sky Removal

Hectospec gets good response beyond 9200 Angstroms, but to use information from object spectra redward of 7000 Angstroms, OH emission from the night sky needs to be removed. A two-pronged approach has been used in the SKYPROC script. First, we select the closest object-free sky spectra on the same CCD chip as each object, and combine them. The point spread functions differ enough between the two detectors that the skylines are inaccurately removed if spectra from the two detectors are mixed. The results are then fit to the object+sky spectrum and removed by treating the sum as a single eigenvector and using our SVDFIT package (Mink and Kurtz 2001) which was originally developed to fit and remove multiple components of a sky spectrum. This method has proven to be better at removing the majority of the troublesome emission lines than subtraction scaled by matching either single or multiple night sky spectral lines. Figure 1 shows a typical galaxy spectrum before and after sky removal. Note that the bright mercury and oxygen lines are first removed entirely. [figure]


The pipeline suite of scripts uses a simple, light-weight method to manage multiprocessing. An array stores the process ID of each job when forked, up to a user-settable number. When the maximum number of jobs to be run in parallel is reached, the scripts wait on the first job, check its exit status and copy its output to standard output and error. If another job is to be run, the cycle is repeated until done.

A complication is that IRAF processes require write access to their parameter files, so they must be copied to one temporary directory per job run in parallel. The uparm variable is pointed to the appropriate directory just before the job is forked.

Hectospec and Hectochelle pipelines do enough I/O that two or three jobs can be run for each processor in a system if the I/O is not a bottleneck. The single-threaded parts of the pipeline are observed to limit the real-time gain to a factor of about 2.5 with our existing CPUs and Gigabit Ethernet connections.

Redshifts and Quality Control

After the sky is removed, each spectrum is cross-correlated against a set of templates using our IRAF rvsao.xcsao task. The RVSAO package includes a quality checking task, rvsao.qplot which labels emission and absorption lines on a graphical display, allowing the resulting radial velocities to be rapidly checked interactively. So far, we have been able to keep up with the data stream and check every spectrum we reduce.


Hectochelle data is processed in a similar manner, using the appropriate parameters to define the selected echelle order and chip binning. Unlike Hectospec, each pointing (multiple exposures on a field with the same fiber positions) has its own associated calibration lamp and dome flatfield spectra. So far we have only automated a few binning-filter combinations; calibrating the remaining configurations will keep us occupied for a while.

Ongoing Changes

Now that the Hectospec pipeline for the low resolution grating is stable, upgrades are being added to process spectra taken with the 600 line/mm grating which gives 0.56 Angstroms per pixel. Several grating tilt positions are allowed, providing spectra centered on different wavelengths. Since the tilt is software controlled, multiple tilt positions can be used per night but each position requires its own set of calibration spectra.


Fabricant, D., Fata, R., Roll, J., Hertz, E., Caldwell, N., Gauron, T., Geary, J., McLeod, B., Szentgyorgyi, A., Zajac, J., Kurtz, M., Barberis, J., Bergner, H., Brown, W., Conroy, M., Eng, R., Geller, M., Goddard, R., Honsa, M., Mueller, M., Mink, D., Ordway, M., Tokarz, S., Woods, D., Wyatt, W., Epps, H., Dell'Antonio, I. 2005, PASP, 117, 1411

Kurtz, Michael J. and Mink, Douglas J., 1998, PASP, 110, 934

Mink, D. J., Wyatt, W. F., Roll, J. B., Tokarz, S. P., Conroy, M. A., Caldwell, N., Kurtz, M. J., Geller, M. J. 2005, ADASS XIV, 228

Mink, Douglas J. and Kurtz, Michael J. 2001, ADASS X, 491

Szentgyorgyi, Andrew H., Cheimets, Peter, Eng, Roger, Fabricant, Daniel G., Geary, John C., Hartmann, Lee, Pieri, Mario R., Roll, John B. 1998, SPIE, 3355, 242

Valdes, F. 1991, ADASS I, 417