Raw Data Flow

The raw data generated by each of the 4 SDSU controllers is transmitted over a dedicated fibre optic link to a PCI card in a data acquisition Computer. This data acquisition computer runs a ukirt Data Handling System (DHS) and records the data along with various header data collected from various sources in a standard UKIRT NDF HDS container. These files are written to a file system located on a RAID5 volume local to the aquisition computer. Thus, there are 4 seperate raw data disks for WFCAM, each located on a seperate aquisition computer.

Whilst observing is taking place, 4 other computers dedicated to data reduction read the raw data from the aquisition machines via NFS and reduce it using oracdr running the so called "summit pipeline" to produce various data quality parameters along with reduced images and catalogues.

In the morning following a night of WFCAM observing, the raw data are written to LTO-1 tapes on drives located in the acquisition computers. Thus, 4 seperate data tapes, one for each camera, are written in parralell.

In addition, the 4 Data Reduction computers also write copies of the raw data (now transformed into compressed FITS files) to LTO tapes. These sets of tapes are mailed to CASU in the UK where the data are processed and ingested into the WFCAM science archive.

Raw data flow schematic

Note. The LTO tape systems in use operate most efficiently with a 64k blocksize. In Linux, this is achieved by running 'mt setblk 0' to configure the drive for variable blocksize, then using 'tar -b128 ...' to specify to tar that it should generate 64k (128 times 512-bytes) blocks. It is necessary to use the same parameters when reading the tapes.

Pipeline Processing

The WFCAM raw data is pipeline processed at UKIRT during observing. This gives the observing team near-real-time feedback on the quality of the data they are aquiring and allows them to assess whether the current weather and conditions are appropriate for the observations they are executing or planning to execute. To do this, the summit pipeline carries out a reasonably complete data reduction sequence, including flat fielding, microstep processing and mosaicing, followed by object detection and catalogue creating, followed by astrometric and photometric calibration.

In addition, the raw WFCAM data shipped to CASU in the UK is processed by an offline pipeline to generate reduced data for ingest into the WFCAM science archive. The offline pipeline has the advantage that it does not have to deal with data in the order it was observed, for example, calibration frames such as darks or flats can be built up from data spanning the entire night, or even several nights.

It is expected that data will be dispatched to CASU within approximately 7 days of being taken. Allowing a further 7 days for transit and a further 7 days for processing / ingest, data should be at the science archive within 3 weeks of being observed.

Access to WFCAM data

In general, wfcam users (including UKIDSS, PATT, UH, Japanese, etc) will obtain their WFCAM data from the WFCAM science archive in the UK. This will be reduced data processed by the offline (UK) pipeline. In order to gain access to the archive, WFCAM PIs must register with the WSA.

It is not expected that the majority of users will require access to raw data. Those that do may request it from CASU.

Reduced data products from the summit pipeline will be availiable to observers at UKIRT. This includes calibrated catalogues and mosaiced images. In general, it will be possible for observers to take away small amounts of this data (eg catalogs and a few mosaics) if they wish. Likewise, small amounts can be supplied to remote PIs. Both of these are on a best efforts basis, and will probably be limited to a few gigabytes in practise.

Some projects may have a valid case for expedited access to bulk data, where the ~6 week delay between observations being taken and data becoming availiable in the science archive presents a serious compromise to the sciencetific potential of the project. PIs who believe that this applies to their project should raise the issue with thier support scientist well in advance.

Japanese and University of Hawaii WFCAM runs

In addition, it is realised that retrieving bulk raw data from the UK may be difficult and slow for people located a long way from the UK (eg Hawaii). Strictly on a best efforts basis, we may be able to write data to either LTO-1 or LTO-2 tape or USB-2 interfaced hard disks at JAC. These tapes or disks must be supplied by the observer. If you wish to take advantage of this, you must contact your support scientist well in advance of your data being taken, typically 2-3 weeks before the run. Last minute requests will likely be unsuccessful. In general, there will be a delay of 1-2 weeks after your data are taken before the data are availiable on your tapes or disks - you will not be able to take the data away at the end of your run.

Access to raw and/or reduced data at the summit