WFCAM daytime checks

WFCAM daytime checks from Hilo

Before starting any daytime checks, make sure that you contact the UKIRT daywork leader first to get clearance to run WFCAM. You will also need to get clearance to run the TCS in simulation mode if you are doing the full check and nighttime darks as you will need the TCS running to get all the necessary headers in the data files. Note that if ETS are running the TCS you will not be able to run TCS_SIM without causing the TCS running at the summit to crash. The darks sequence will also require that the dome lights are off and the dome is closed.

Full daytime check with nighttime darks
Full daytime check without nighttime darks
Regular engineering tests
Potential problems and fixes

Full daytime check with nighttime darks (needs TCS running)

check daywork; call summit; send ukirt_ops wfcam daywork e-mail
log in to ohi as observer with ssh -Y observer@ohi
launch WFCAM engineering interface with wfcamControlEng
start low level camera software for all cameras [0a]
run simulated tcs [1a]; wait for startup messages and launched windows
run ocs_up [2a]; wait for startup messages and launched windows
add wfcam instrument software to ocs [3]; confirm on sequence console
enable arrays for all four cameras [4]; confirm on sequence console
send WFCAM flush and readnoise measurements from query tool to ocs queue; start ocs queue
start wfcamReduce; confirm data directory presence and reduction started
start wfcamNoise; confirm nominal readnoise
start local ohi ompobslog; confirm data present and add readnoise results shift comment
start remote kauwa and hokulea ompobslogs; confirm data present
send WFCAM darks (and any specially-requested darks) from query tool to ocs queue; start ocs queue
wait for dark observations to finish; wait for dark reductions to finish
stop and exit wfcamReduce pipelines; run wfcamReduceNuke
disable arrays for all four cameras [5]; confirm disabled in log
remove wfcam instrument software from ocs [6]
delete all the deferred MSBs from the deferred area by dragging them to the waste bin
type exit and ctrl-c in and close tcs_sim_cli window
run ocs_down [7]
run tcs_nuke [8a]
stop low level camera software for all cameras [9]; confirm stopped and ignore noticeboard in log
run a drama nuke on kauwa
exit wfcamControlEng
log out of ohi
send ukirt_ops wfcam status e-mail report with instrument status, darks status, and final frame

Full daytime check without darks (TCS not necessary)

Log into Ohi as "observer" (using "ssh -Y observer@ohi") and and type "wfcamControl". This will launch the WFCAM control GUI. Run up the low level software from the GUI by hitting the "start" button and check that all four cameras are running by clicking on "status". Ignore any beeps you get when hitting buttons on the GUI, this behaviour is quite normal.
On Ohi, type "ocs_up" and wait for all the OCS screens to launch.
Back on the WFCAM control GUI, hit the "add inst" button, wait for the sequence console to become "live", and then hit the "enable" button to activate the arrays.
Send the "WFCAM flush" (optional) and then the "WFCAM readnoise measurement" sequence to the queue and start the queue, or send directly via "send for engineering" and start the sequence in the sequence console.
While the read noise data are being taken, log onto Kauwa as "observer" (using "ssh -Y observer@kauwa") and type wfcamObs. From this GUI, hit the wfcamReduce button to start the DRs on all four WFDR machines. Check that the DRs can see the raw data and the read noise level as reported by the DRs.
Assuming all is well, quickly run ompobslog on both Kauwa and Hokulea and ensure they see the data.
Disable the arrays from the WFCAM control GUI (NB. please make sure that you get the response in the GUI logging pane that the arrays are powered off).
Type "ocs_down" on Ohi, then "drama_nuke".
Run down the WFCAM low level software, check the status to make sure the software has run down successfully, exit wfcamControl and log out. You can ignore any messages in the wfcamControl logging pane that say "Failed to unlink noticeboard file".
Exit the DRs and nuke them (with -nogaia). The nuke can also be done using the "wfcamReduceNuke" button in the wfcamObs GUI. Log out.
Exit wfcamObs on Kauwa and log out.
Log into each wfacq machine as root. cd to /export/data/raw/wfcamN/<utdate> (where N is the number of the wfacq machine you are logged into) and delete both the raw data and the .*.ok files.
Log into Kauwa as observer, cd to /ukirtdata/raw/wfcam/<utdate> and delete the .lastobs file.

Steps 12 and 13 ensure that the nighttime observers will start with a blank slate and that their first frame of the night will be frame 1, You do not need to carry out these steps, but if you don't, you must in form the TSS/observers or send email to ukirt_ops explaining that data files already exist for tonight.

Regular engineering tests (data in engineering directories)

Log into Ohi as "observer" (using "ssh -Y observer@ohi") and and type "wfcamControlEng". This will launch the WFCAM control GUI. Run up the low level software from the GUI by hitting the "start" button and check that all four cameras are running by clicking on "status". Ignore any beeps you get when hitting buttons on the GUI, this behaviour is quite normal.
From the GUI, hit the "ocs_up -simTel -eng" button which will launch the OCS.
Again on the WFCAM control GUI, hit the "add inst" button, wait for the sequence console to become "live", and then hit the "enable" button to activate the arrays.
Send the "WFCAM readnoise measurement" sequence to the queue and start the queue, or send directly via "send for engineering" and start the sequence in the sequence console.
Log in as observer to each of the four WFDR machines. In each one, type "oracdr_wfcam_eng" followed by a nuke (-nogaia) and run the DR with "oracdr -loop flag -log sf -nodisplay -skip" or with the options you prefer.
When finished, disable the arrays from the WFCAM control GUI (NB. please make sure that you get the response in the GUI logging pane that the cameras are powered off). You can ignore any messages in the wfcamControl logging pane that say "Failed to unlink noticeboard file".
Hit the ocs_down button in the GUI (also best to type drama_nuke on Ohi afterwards as well) and then stop the low level software, checking the status to make sure the software has run down successfully. Exit the GUI and log out.
Exit the DRs and nuke them (with -nogaia), Log out.

Potential problems and fixes

Countdown stuck at zero or low level software does not run up successfully: Refer to flow chart in fault 20061106.003
"add inst" does not activate the sequence console or the arrays won't enable: Usually a problem with the OCS. Run down the low level software, do an ocs_down and then drama_nuke on Ohi, run up the low level software, ocs_up and try again. This will usually clear the problem. Refer to the flow chart in 20061106.003 if there are problems with the low level software.
Data are taken successfully, but DR can't see the frames: One or more raw data disks are not mounted. Log into Kauwa as yourself, type "/local/bin/remount wfcamN" where N is the number of the camera that you need to remount" and when prompted enter your password. You should make sure the DR for the dismounted disk/s isn't running when this is done. If the remounting stops at some point, ctrl-c will get it to continue ands then repeat the command.
Ompobslog does not show data: the raw data disk on wfacq1 is not mounted. Follow instructions above.
Arrays do not power off: When disabling the arrays, you must make sure you get a response saying the cameras are powered off in the wfcamControl logging pane. If not, run down the low level software, fix any problems with reference to the flowchart in 20061106.003, run the low level software back up again, ocs_up, add inst, enable and then disable arrays. If you cannot disable arrays, then get someone at the summit to power off the arrays using the disable buttons on the controllers.