WORKING WITH WFCAM DATA AND THE WFCAM SCIENCE ARCHIVE

This cookbook describes:

1. Data access (see also the Accessing UKIDSS or Accessing Flexed Data links on the sidebar),
2. The processes involved in generating complete tiles from the "pawprints" (or "Multiframes") in the WSA,
3. How to make use of the Multi-Extension FITs catalogues supplied with each Multiframe, and
4. How to get started with SQL searches of the source tables hosted by WSA (i.e. how to get photometry data without downloading the actual images!). This includes a very brief introduction to SQL and the Starlink table viewer/editor TOPCAT.

Commands are mostly from the Starlink software collection, which is available from http://www.jach.hawaii.edu/software/starlink/.
You may need to update your Starlink installation for some of the commands described below to work properly!

• For notes on KAPPA, type showme sun95 or go to http://docs.jach.hawaii.edu/star/sun95.htx/sun95.html
• For notes on CCDPack, type showme sun139 or go to http://docs.jach.hawaii.edu/star/sun139.htx/sun139.html
• For notes on GAIA, type showme sun214 or go to http://docs.jach.hawaii.edu/star/sun214.htx/sun214.html

The WFCAM Science Archive also host a Frequently asked Questions webpage.

Finally, if you are still having problems (having read the notes below), please contact your UKIRT support astronomer (if you have one), Chris Davis at the JAC (c.davis@jach.hawaii.edu) or the WSA helpdesk (wsa-support@roe.ac.uk).

1. Access to Survey and non-Survey Data from WSA

Registering

PI-led projects (PATT, Service, UH, etc.) must be REGISTERED, by the PI, with WSA. THIS DOES NOT HAPPEN AUTOMATICALLY! Projects may be registered at any time, though the end of the semester (or shortly after the last data have been acquired) is a good time to do this.

To register your PATT, Service, Japanese or UH programme, click on the nonSurvey link on the WSA sidebar and fill in the registration box. IMPORTANT: If you observe the same field in multiple filters, select the filters in the last row of the form. If you don't, a merged source catalogue won't be created (see below). When your data have been ingested in the archive you will be sent an email informing you that data are available. Login at WSA and try the "Archive Listing" link on the WSA sidebar to see what's available... The WSA "Start here" link is a good starting-point for first-time users!

File-naming convention

The MEF is probably of the form: w{date}_{num}_sf_st.fit

For nonSurvey data, initially only Flat File access may be available. In this case images are fully reduced and photometry (FITS) catalogues are made available - one for each MEF (see below). However, if a region has been observed in multiple filters, then merged tables (Source tables), where multi-band photometry for each source are collated and presented in the same row, are only made available when "full" data access is attained. Moreover, the FITS catalogues available with "flat file access" don't include columns of magnitudes, although integrated fluxes (aperture photometry) are provided for each source, along with instrumental magnitudes (zeropoints) and other calibration coefficients.

A full description of the data products available in the archive is given at the WSA website.

Data processing steps are described in a series of papers (note in particular Hambly et al. (2007) and Irwin et al. (2008)). See also the technical notes at CASU: http://casu.ast.cam.ac.uk/surveys-projects/wfcam/technical

The WFCAM Archive - having a quick go...

The regions covered by the latest release are shown on the surveys page. Note that the date given for each data release pertains to ESO releases; world data releases will be approximately 18 months later. For example, DR1 (and DR1plus) had their world data release in January 2008.

• Archive Listing on the WSA side-bar is perhaps the quickest way to access large volumes of data. If you enter a range in RA/DEC and select a filter you should get a list of observations (the frame type should be set to "stack" - see above). Try for example just changing "filter" to H2 and leaving everything else as default. Submit should give a list of observations; if you click on "view" for one of the observations you will get 4 GIF images showing the reduced data. The FITS Img and Cat files listed in the same table are Multi-Extension Fits (MEF) images and Multi-Extension Fits catalogues - basically four images or four tables per file, again as described earlier.
  One thing to note is that the RA and Dec listed in the "archive listing" table are actually located midway between the four images. The WFCAM arrays are separated on sky (as described ) so you don't actually get data at the coordinate listed in the table!


• Get Image is another useful tool, which allows you to get a small snap-shot image around a target coordinate. This time, data are returned for the coordinate you enter. For example, if you enter 5:35:27,-5:4:00 as coords and submit you'll get a list of reduced images (one per filter) which you can view as GIFs or download as FITs images. Setting X-size and Y-size to 10 arcmin gives a nice view of Haro 5a/6a (or there-abouts).


• Region is used in a similar way to "Get image", although this time it returns a source table, with multi-band photometry listed in various apertures. Again, if you enter 5:35:27,-5:4:00 (and select "UKIDSS Galactic Clusters Survey" and "Source table/merged catalogue") in the Programme/Survey and Table pull-down menus, you'll get a table centred on Haro 5a/6a which can be saved in various formats: HTML - the default - simply displays the table in your browser.
  For your favourite source you can set the search radius to 0.05 (3 arcsec), resubmit the query, and get photometry for just that one source. If you click on "view" in the resulting table, you even get a snap-shot image in each of the available filters.

Colour images direct from the WSA

As an example, login as a UKIDSS user and try the following coords: 20:38:35 (2000) 42:37:30 with the survey set to GPS and filters K_1, H and J (multiple-epochs in a particular filter will be listed as K_1, K_2, etc.). These should give a nice colour picture of the massive star forming region W75N.

3. Catalogues

File naming convention

  w20060702_00204_sf_st.fit        (the MEF containing 4 images)
  w20060702_00204_sf_st_cat.fits   (the MEF containing 4 catalogues)

Inspection & Concatenation

Open the MEF catalogue first, e.g.:

> gaiadisp w20060702_00204_sf_st_cat.fits &

In the pop-up window, highlight and "open" the table you're interested in (this may take a few moments). Now, from the main GAIA file pull-down menu open the image MEF fits file (in this case w20060702_00204_sf_st.fit), and select the associated image from the list of four. Finally, "plot" (or "search" and "plot") in the table pop-up window should mark the sources on the image.

Alternatively, you can extract the four images from the MEF as described above, and similarly extract the four table files using the STILTS tcopy command:

> stilts tcopy w20060702_00204_sf_st_cat.fits#1 w204cat.fits > stilts tcopy w20060702_00204_sf_st_cat.fits#2 x204cat.fits > stilts tcopy w20060702_00204_sf_st_cat.fits#3 y204cat.fits > stilts tcopy w20060702_00204_sf_st_cat.fits#4 z204cat.fits

NB: if you put the above into a c-shell script, remember to put " " around the #1, #2, etc. in your script; otherwise c-shell will think everything after the # is a comment!

Open the image in GAIA; then open the catalogue file with "Data Servers - Local catalogues - Load from file". The sources should automatically be marked on the image, and the catalogue displayed.

Individual tables in a catalogue can also be opened in TOPCAT (note that if you omit the #3 below, only the first of the four tables in the multi-extension fits catalogue file will be opened by TOPCAT):

> topcat w20060702_00204_sf_st_cat.fits#3 &

If you have combined the 16 images to create a WFCAM tile, you may wish to do this to the 16 tables in the four MEF fits catalogues that cover the same tile. This can be done with a simple c-shell script using STILTS and the tcopy and tcatn commands. For example, write the four tables from each MEF catalogue to separate files as described above, then concatenate all sixteen tables with, e.g.

> stilts tcatn nin=16 \ in1=w1cat.fits in2=w2cat.fits in3=w3cat.fits in4=w4cat.fits \ in5=x1cat.fits in6=x2cat.fits in7=x3cat.fits in8=x4cat.fits \ in9=y1cat.fits in10=y2cat.fits in11=y3cat.fits in12=y4cat.fits \ in13=z1cat.fits in14=z2cat.fits in15=z3cat.fits in16=z4cat.fits \ out=combined.fits

You should now be able to display a full tile image in GAIA, and over-plot (with "Data Servers - Local catalogues - Load from file") a full tile catalogue. However, note that the above does not deal with sources in the overlap regions, which will appear in multiple MEF tables. Also, zero-points will change from camera to camera; you may thus be better combining only tables for each camera, i.e. for each corner of the tile.

Either way, you should be able to open the combined catalogue in TOPCAT for further processing (see below).

Photometry Calibration with MEF Catalogues

• MAGZPT = 23.98 / Photometric ZP (mags) at airmass of 1
• NIGHTZPT= 23.0 / Average photometric ZP (mags) for night
• NUMZPT = 1317 / Number of standards used

   Mag = ZP - 2.5*log10(flux/exptime) - extin*(airmass-1) - apcor - percorr 

The parameters needed to convert aperture fluxes to magnitudes are stored in each FITS catalogue header. To view these parameters (MAGZPT, EXP_TIME, EXTINCT, AMSTART/AMEND, APCOR* and PERCOR), open the catalogue in Gaia: in the main Gaia window click on "View - Fits header". The resulting pop-up will contain five tabs; those labelled 2,3,4,5 contain the headers for each of the four cameras. Note that these can be listed alphabetically...

For a more detailed description of these parameters see this document from CASU, or consult the notes on catalogue generation. The WFCAM photometry system is described here. The archive also provide a nice cookbook on working with WFCAM flat files and catalogues.

Excluding saturated sources

The simplest way to flag sources as saturated is to make use of the "Peak_height" column and "Sky_level" column information in the MEF catalogue, and compare these with the saturation value SATURATE in the catalogue header. Objects may be flagged as being saturated if they reach 90% of the saturation level with respect to the sky; i.e. if peak height > 0.9*(saturate-skylevel).

TOPCAT is an interactive graphical viewer and editor for tabular data. Both TOPCAT and STILTS require a Starlink installation (although strictly speaking catalogue analysis alone doesn't require the full Starlink suite - see the TOPCAT webpage below). Non-Starlink tools include HEASARC's fv, and ESO's Midas package seems to handle the FITS MEFS also.

Further info on TOPCAT (including software download) is available at http://www.starlink.ac.uk/topcat/
Further info on STILTS is available here: http://www.starlink.ac.uk/stilts/

4. A SESSION WITH THE WSA ARCHIVE - SQL and TOPCAT

The WFCAM Science Archive (WSA) can be used to download tables of photometry, either from one of the UKIDSS surveys, or from a specific PATT or UKIRT/Service programme. Here we provide an example session based on data taken in a service programme; a session querying UKIDSS data would be very similar, except that obviously the user must login as a UKIDSS user, and search a different source catalogue, probably using somewhat different parameters (see below). Login is not necessary when querying public data from a World Data release.

The WFCAM Science Archive (WSA) - Downloading Source/Photometry Tables

• Registering: Necessary for non-survey data (PATT, service, UH or Japanese programmes) - see above.
• Accessing data: When non-survey data become available in the archive, WSA will email you with a password. (UKIDSS data releases are announced on the UKIRT and UKIDSS homepages.) Return to the WSA website and login: for PATT or service data, the Community is nonSurvey; for UKIDSS data the Community will probably be derived from your email address, e.g. jach.hawaii.edu.
• SQL query: Go straight to the Freeform SQL interface (link on the sidebar). First-time users should check out the "notes and tips" link, or the "Cookbook". SQL examples are also given in the appendix of Phil Lucas' GPS survey paper (MNRAS, 2008). An example query is given here:

      select sourceID, ra, dec,
      mergedClass, pStar,
      jAperMag3, jAperMag3Err,
      hAperMag3, hAperMag3Err,
      kAperMag3, kAperMag3Err,
      h2AperMag3, h2AperMag3Err,
      jppErrBits, hppErrBits, kppErrBits
      from userv1635Source
      where pstar >0.990
      /* Exclude artifacts */
      and mergedClass !=0
      /* Exclude multiple detections of same source */
      and (PriOrSec=0 or PriOrSec=framesetID)
  

  This query searches the userv1635Source source catalogue created for service programme u/serv/1635. The source catalogue associated with the GPS, for example, is called gpsSource.

  Hopefully, most of the above is self-explanatory (note: JHK and narrow-band H2 data were obtained for this particular project). pStar and mergedClass define the likelihood that a source in the catalogue is a star; the text within "/*" and "*/" is a comment; PriOrSec is used to exclude duplicate sources in overlap regions between frames; the ErrBits parameters are used to identify cross-talk artifacts, bad pixels and/or saturated (or non-linear/near-saturation) sources. (pNoise and pSaturated are two other useful parameters.)

• Schema Browser, or "what on earth are hppErrBits?" To establish which additional parameters are available for query and/or download, and to get a description of all catalogue entries, click on Schema Browser on the side bar. Once there, select WSA nonSurvey from the new sidebar and click on your project.
• Submitting a query: When you are happy with the SQL query, select VOTable file as the Data Format and Submit. WSA may initially claim "zero rows returned"; be patient as it continues the search. You should eventually see the first 30 lines of the table displayed in your browser.
• Downloading a data table: Finally, save the table to your local disk as VOTable ascii. You may have to right-click on the link to avoid your browser displaying the text in the browser window (rather than saving to a file). The VOTable ascii table should be readable in TOPCAT (see below) and GAIA.

VOTable Photometry Tables and TOPCAT

Below we give a few tips on displaying WSA photometry in TOPCAT, Starlink's very nifty interactive graphical viewer and editor for tabular data.

• Getting started: Having installed TOPCAT (available here), run TOPCAT by simply typing topcat on the command line, and Open the VOTable ascii file. The file name will be displayed top-left in the main TOPCAT window.
• Displaying table contents: Click on Display Table Cell Data (4th icon - main window, or just double-click on the table in the "Table List") to see the contents of the table.
• Creating new/deleting obsolete columns: Right-click on a column in the table browser to define a "new synthetic column". Fill in the "name" (the column heading) and "expression" used to define the new data (the rest are optional). For example, a column listing H-K/J-H values could be defined as hAperMag3-kApermag3/jApermag3-hAperMag3. Note that column headings should not include mathematical characters (these will mess up subset selection - described below), so instead of H-K/J-H, the column heading should be something like HmKoverJmH.

• Choosing subsets: TOPCAT allows you to select a portion of the tabulated data using a parameter string. Different criteria can be used to select different "subsets" of the data. Multiple subsets can then be displayed on the same line or scatter plot. For example, blue and red stars could be selected and plotted with different symbols on the same Colour-Colour or Colour-Magnitude diagram.

  To select a subset of the opened table, click on Display Row subset (7th icon - main window); click on the green "+" in the new "Row subsets" window and enter a subset "name" and "expression". E.g. a subset called All Stars in L1448 could have an expression:

     equals(region,"L1448") && pstar > 0.995 
  

  Or you could select a subset based on RA or Dec range. Note that if RA and Dec are strings (e.g. 4:20:30.5 -5:30:20), then this should work:

     radiansToDegrees(dmsToRadians(Dec)) >= -5.5
  

  A second, tighter subset could then be selected by again clicking on the "+" button and entering:

     equals(region,"L1448") && pStar > 0.995 && pNoise < 0.003 &&
     jAperMag3Err<0.1 && hAperMag3Err<0.1 && kAperMag3Err<0.1 &&
     jppErrBits == 0 && hppErrBits == 0 && kppErrBits == 0
  

  The above criteria (all typed in on one long line - stretch the window and/or scale the column widths!) should select only data with decent point-source photometry. Limiting the maximum and minimum magnitudes of sources (i.e. targets too faint to be believed, or too bright to be in the linear regime of the array) may be another way to do this, e.g.:

     equals(region,"L1448") && pStar > 0.995 && pNoise < 0.003 &&
     jAperMag3Err<0.1 && hAperMag3Err<0.1 && kAperMag3Err<0.1 &&
     kAperMag3>10.5 && jAperMag3>10.5 && jAperMag3<21.0
  

  Sources within a given RA and DEC range can also be selected, e.g.:

     equals(region,"L1448") && pStar > 0.995 && pNoise < 0.003 &&
     jAperMag3Err<0.1 && hAperMag3Err<0.1 && kAperMag3Err<0.1 &&
     kAperMag3>10.5 && jAperMag3>10.5 && jAperMag3<21.0 &&
     ra < 309.855 && ra > 309.655 && dec < 42.43 && dec > 42.255
  

  Note that RA is tabulated in decimal degrees (360 degs = 24 hrs in RA). Finally, a subset of just the red sources, with H-K/J-H > 0.7 could be selected:

     equals(region,"L1448") && pStar > 0.995 && pNoise < 0.003 &&
     jAperMag3Err<0.1 && hAperMag3Err<0.1 && kAperMag3Err<0.1 &&
     kAperMag3>10.5 && jAperMag3>10.5 && jAperMag3<21.0 &&
     HmKoverJmH > 0.7
  

  TIP: to edit a parameter in an existing subset selection expression, double-click on the expression in the "Row subsets" window; you should then be able to edit it.

  Rather than type in longer and longer subset strings, a subset can be saved to a new, smaller table. The user may then simply work with this smaller table (see below). Alternatively, it is possible (I think) to select a subset of a subset: if you highlight a subset sample in the "Row Subsets" window, then define a new subset, the next subset will in fact be a "sub-sub-set", that is, a subset of the highlighted subset!

• Plotting data (Click on the scatter plot below to see an example session): Any of the above subsets can be plotted on the same axes. With the original table highlighted in the "Table List" in the main TOPCAT window, click on the Scatter plot icon in the main TOPCAT window.

  In the Scatter Plot window, the X and Y-axes to be plotted can be selected bottom-left, and the "Row Subsets" to be displayed can be chosen bottom-tight. The symbol size/colour can be altered (by clicking on the symbol key bottom-right), as can the labelling of the key.

  The axes scales and labels can be set from the Scatter-Plot window panel (third icon), and GIF and EPS versions of plotted figures saved to disk.

  Lastly, if a second table is opened in the main TOPCAT window, these new data can be included on the same plot. In the Scatter-Plot window, click on the add a new dataset icon (just below the graph) to include these new data.

  In the example below, a subset of "all stars" are plotted in blue; a "red stars" subset is over-plotted in red, and data from a second and third table - that define the Main Sequence dwarf and giant branches - have been drawn as black lines.
  
  

  

  Click above to see the full TOPCAT session.

• Saving subsets as new (smaller) tables: Once you've gone to the trouble of creating new columns, deleting obsolete columns, and establishing which sources are, e.g. red, you can save just this subset of data to a new table. In the main TOPCAT window, click on Save Table (second icon). Select the "Row subset" to be saved - so in our example this would be the "Red Stars" - the "order" in which rows are to be sorted, and set the output format to "VOTable table data", "FITS" or "ASCII".
• Over-plotting sources onto a WFCAM image: New tables can be re-opened in TOPCAT at a later date. Alternatively, open the FITS or ASCII files in GAIA. GAIA should easily handle the RA and DEC parameters, which are both stored in decimal degrees (you may need GAIA version 4.0 or above). The FITS table can be opened under "Data servers -> Local catalogues". The ASCII table must be imported as a text file: "Image analysis - > position -> Import plain text". (The ascii file is delimited with spaces; I had more luck with this file when over-plotting the positions onto a WFCAM image than with the FITS file...)

Miscellaneous WSA Tips

Photometry for just a few targets: can be obtained using the "Region" link. Log on to your PATT project, or as a UKIDSS user (described above), and go straight to "Region". Enter the RA,Dec of your target and - for individual sources - set the Search radius to 0.03 (2 arcsec). Select the catalogue to be queried and submit.

IMPORTANT - If you suspect that a source is only detected in one or two bands, select the "detection table" rather than the "merged table". This will return separate rows for each filter. Filters are identified by number: 3 is J, 4 is H, 5 is K, 6 is H2, etc. If for example a target was only detected in K, only one row will be returned. You can also examine the data (a gif cut-out) by clicking on View in the returned table.

5. SCRIPTS

• extract-wfcam-images.prg

• bin-wfcam-images.prg

• makemosaic_wfcam.prg
• makemosiac_wfcam_basic.prg
• makemosaic_wfcam_tile.prg
• makemosaic_wfcam_corner.prg

• clean_wfcam_images.prg

• align-and-subtract.prg

• wfcam-table-concat.prg

• surface_fit_wfcam.prg (fits and removes a coarse surface to background regions)

6. MORE INFORMATION

• CASU have produced a web page describing how to get sensible colour-magnitude and colour-colour diagrams from the archive: casu.ast.cam.ac.uk/surveys-projects/wfcam/science/2005a-science-data-examples