Mechanical

WFCAM interlock failure:
See e.g., 20061107.002. On a stuck countdown fault or if you have difficulties switching filters, quickly check the main WFCAM DM screen. If there is an interlock fault the interlock status will be red on this screen. You must follow the procedures in the first response to 20061107.002 in order to get going again. Interlock faults occur when the WFCAM CCS loses the filter paddle position and forces you to park the filters then redatum in order to refind the positions and prevent the paddles colliding.

Hexapod stuck/star looks like a doughnut:
Typical symptom of hexapod stuck fault is a guide star that looks like a doughnut due to being out of focus. First, toggle the telescope focus between 0 and 1 to see if the hexapod responds. If it does, try refocusing, if it doesn't, slew to zenith, reset the hexapod and retry. The canonical fault for this is 20051214.010 and please respond to this fault if you have an occurrence of this but no time is lost.

Light leak in dark frames:
Since the filter paddles do not completely mask the arrays when in the blank (dark) position, there is a light leak which affects dark frames when the ambient light is bright. The DR should report a light leak. To avoid this, the beginning of night dark sequence should be started no earlier than five minutes before sunset. This fault was originally misunderstood (20050619.001) but the current fault with more details and an image can be found in  20070417.003.

Sequence console pops up error after WFS:
See 20080215.001. The sequence console pops up the error "Failed to set configuration (phase 1)". This is due to the focus mechanism hitting the limit switch because the WFS focus offsets were left in and WFCAM is now datumming the filters. While the filters are finding datum, open the wfcamMain.dl DM screen and put 0.0 into the focus offset field and wait for the datum process to finish.

---

Arrays/electronics

Arrays fail to enable/blank first frame:
The controller for camera 2 had a fault which resulted in camera 2 not enabling on the power on command sent by the TSS, however this is now believed fixed (see 20050403.001). If you do see any of the arrays blank after being enabled (powered on), then disable the arrays, remove inst. (or  ocs_down), run down the low level software, power cycle the controller for the problem array and restart everything.  Take a quick frame using the read noise or flush sequence to check that the camera is now enabled.

Read noise is high:
The DR will report high read noise, and if the read noise is so high that it's clear there is an electronic problem, will exit. First thing to try is a flush sequence and then another read noise sequence. If the noise is lower but not nominal, repeat. If the noise is still very high, run down the observing software, power cycle the controllers (or the controller for the problem array) and restart.

Dark current is high (especially camera 4):
The DR reports dark current although it will not exit if the current is high. Note that camera 4's DR always reports large dark currents for long exposures (typically in the hundreds). These numbers can be ignored. It is due to a read-out issue which puts a bias level into the array. This is steady and is removed in dark subtraction. See 20061116.004.

Channel edge problem:
Channel edge problem can occur in all arrays and there's essentially nothing you can do about them. Camera 2 currently has a "weak" edge problem which appears to subtract out nicely. Cameras 3 and 4 will likely exhibit the majority of the "hard" channel edge problem. Software work is continuing at the ATC to try and provide a fix. The canonical fault is 20070118.002.

Arrays fail to disable:
See 20070429.004 (original) and 20070809.001 (latest info and the canonical fault). Occasionally the arrays do not disable at the end of the night or at the end of daytime checks. The obvious symptoms of this are 1) the arrays remain powered on in the sequence console and 2) the TSS does not get the powered off response in the wfcamControl log (please do watch for this message when disabling the arrays). If this problem occurs, you should remove inst. run down the low level software, restart the software, add inst. enable and then disable the arrays. Sometimes this is not enough, in which case a full ocs_down is needed.

---

Computing/data acquisition

Countdown freezes at zero: "nonfatal":
The canonical fault is 20061106.003. For essentially all zero countdown faults you must remove inst and run down the low level software (it is unlikely you will be able to disable the array). When the software is run down, check the camera status. Do a nuke if necessary (and there is no harm in doing one anyway when the software is run down). If everything is OK, restart the low level software and again check status. If everything is running, add inst, enable the arrays and restart observations.For easier reference, follow the flowchart.

Countdown freezes at zero due to machine failure: message involving "integer":
In this case the stuck countdown is a symptom of a wfacq machine having died. You will need to reboot whichever one is involved.

For rebooting, first try soft reboot via kvm switch as given below:

  kvm (this gives an error message that you can ignore)
  connect to irtkvm as "admin" with the usual !password
  connect to the machine in question (double click)
  then log in as root and type '/sbin/shutdown -r now'

If a soft boot doesn't bring it back (if you don't get an response on the kvm switch) try a hard reboot:

  "connect wfacqN power", (give the admin's password when prompted for) and reboot using the command "/boot wfacqN".

After the reboot you must remount the raw data disk using "/local/bin/remount wfcamN". Do this from your own account using your own password and make sure the DR for the relevant camera has been stopped. If the remount sticks, hit ctrl-c and it should continue. To make sure, remount the disk a couple of times.

Acquisition fails to write data file:
Occasionally WFCAM fails to write a raw data frame. This usually goes unnoticed since we are using the -skip option in the DR, but the DR might complain about the mosaic. Unfortunately there's little that can realistically be done about this at the telescope. See 20060429.002.

Disk not mounted/ompobslog can't see data:
If the DR does not see the raw data and/or ompobslog doesn't see the data then the relevant raw data disk needs remounting (if ompobslog doesn't see data, then the problem disk is on WFACQ1). Stop the DR, and from your own account on any summit machine run "/local/bin/remount wfcamN" where N is the machine number. Use your own password when prompted. If the remount sticks, hit ctrl-c to continue. Redo the remount if necessary and then restart the DR and /or ompobslog.

Failed to set configuration (phase 1):
See Sequence console pops up error after WFS and fault 20080215.001.

---

Data reduction

DR exits on first frame:
Typically due to an array or all of them being disabled. If the arrays were enabled earlier but one is blank, the relevant controller has to be power cycled (see Arrays fail to enable/blank first frame). If the arrays were not enabled, then stop the sequence, enable the arrays, restart both the sequence and the DR.

Read noise is high:
See Read noise is high. Generally flush the array and redo the read noise sequence. If this doesn't help, the controllers may need to be power cycled. If the read noise is unusually high, the DR will exit as it's telling you there is a problem that needs fixing. If this is the case, it's usually best to call for help.

Dark current is high (especially camera 4):
See Dark current is high (especially camera 4). In just about all cases this can be ignored, especially if it's in camera 4 as the DR provides an incorrect dark current value in the hundreds.

Light leak in dark frames:
The DR will exit if it detects the known light leak during initial darks (see Light leak in dark frames). If this occurs, wait until 5 minutes before sunset before starting the darks sequence.

Channel edge problem:
The DR will report channel edge problems but in general will not exit. There is little that can be done at the moment (a software fix is being looked at) apart from marking the bad frames in the night log. Restart the DR from the start of the next group.

No suitable dark:
The will exit if it cannot find a suitable dark for the current observation. This is to warn you that the data you are taking cannot be calibrated. You must make sure that a correct dark is taken, in fact it is recommended to take at least 5 of them. They must have the same exposure time and number of coadds. The DR will exit for the same reason if the darks sequence was not taken at the start of the night or in daytime checks.

DR can't see the raw data:
See Disk not mounted/ompobslog can't see data. If the DR just sits there producing dots and not reducing the raw data that you know have been taken, then it is likely the raw data disk is not mounted. Stop the DR, remount the disk using "/local/bin/remount wfcamN" and your own password. If the remounting sticks, use ctrl-c to make it continue. Run the remount script twice if necessary and then restart the DR.