How to run RTS2-F and the robot on flwo48 as observer. Created 10/01/10 Updated 06/22/17 ***********STARTING RTS2-F (robot servers) Type Rstatus in any xterm to find out whether the RTS2-F servers are running. The servers should run all the time and must be running before GOrobot (see below) is issued. Hereinafter, RTS2-F refers to the servers. 1) If RTS2-F has crashed, make sure the robot is not running in Telshell, if it is, stop it with CTRL-C, then in any xterm type: Rstop NOTICE: Rstop clears the queues but it also saves them to a file that Rstart uses to reload the queues. 2) To restart RTS2-F and the monitoring, first ensure that the real-time system is up, otherwise in any xterm type gokep; once the system is up, in any xterm type Rstart. Then type GOrobot in the Telshell. Also type GOweather and GOqueues in any xterm in desktop 2 (2nd from the left in the task bar at the bottom of your VNC window). Then, click the box at the top of the GOweather GUI, to set the state to ON (or Standby if the weather is inclement). NOTICE: a) Rstart will reload the queues from the last file that Rstop saved. Double-check their contents. b) GOweather is different from STARTweather, which restarts weather sensing, see the next paragraph. 3) If the robot senses the weather is inclement (see GOweather) but you are CERTAIN (using the MMT skycam and confirming with the 1.5m observer) that the weather is clear and the sensing is wrong, you may use the command KILLweather to disable ALL weather sensing. Use with great caution. To restart weather sensing after a KILLweather or after a larger system failure, type STARTweather. 4) On flwo48, watchdog jobs monitor the robot's activity. They verify that the robot is running, that data are being acquired at night, that the dome is not open under weather conditions over closing thresholds (wind, RH, precipitation, clouds) and that the CCD is operating normally. If there is any violation, the robot warns the 1.5m observer, and requests a clearing of the violation every 10 minutes, until the observer clears the violation and issues a clear_robot command on flwo48. On flwo60, /var/log/chk.robot.log shows the status of any violation every 10 minutes, and on flwo48, /var/log/rob.clear shows when the violation was cleared. ***********STATES and MODES of RTS2-F (once Rstart is issued) You will find RTS2-F in one of several possible modes listed in the table below. The robot's actions depend on the mode, which may change with the state (e.g., night), through observer input, or as a function of weather conditions. The modes are displayed with the GOweather GUI or with MONrobot in any xterm. mode state weather action ----- ----- ------ ----- ON day any wait for domeflats ON dusk bad wait for good weather ON dusk good open, take skyflats ON night bad if open and flag close_bad_weather (under DOME) is true, stop and close ON night good open if needed, observe OFF any any wait for observer input STANDBY day any wait for domeflats STANDBY dusk bad wait for weather; if flag weather_open (under DOME) is true, switch to ON upon good weather STANDBY dusk good wait for ON, remain closed STANDBY night bad wait for weather; once flag weather_open (under DOME) is true, switch to ON upon good weather STANDBY night good wait for ON mode. If weather_open flag (under DOME) is true, switch to ON, open if needed, observe ************STARTING THE ROBOT To start, enable the robot to allow it to run. There are 3 ways to do it, with GOweather (preferred; click ON at the top of the GUI), or MONrobot (the "curses" xterm), or RTS2-G (the RTS2-F web GUI). 1) GOweather: This GUI shows the weather status and allows you to change the state to ON (good weather) or STANDBY (bad weather), see the instructions in the ROBOT USERS GUIDE. Notice that GOweather displays the weather status is different from STARTweather, which starts weather sensing. 2) MONrobot: In any xterm, type MONrobot. You may enable (disable) the robot there with F4 (F3); press F10 or CTRL-C to exit MONrobot. You may also leave MONrobot running and put its xterm in the background. If you use VNC, in some variants the F keys may not work. If that's your case, just type "on" inside the MONrobot window, then RET. 3) RTS2-G: on the Seamonkey browser on flwo48, click on RTS2-F in the top menu bar to bring up the main page. Then click on "View or change current state" and finally on the appropriate button. The username is observer with the usual pwd. Once the sate is "ON" type "GOrobot" in the Telshell to run the robot. Several default actions of GOrobot can be modified with the following flags: -D Skip dome flats -d Use alternative dome flats list: /home/observer/RTS2-F/FLATS/ALTFILT Currently, this is the default list for flats. -e Skip evening sky flats -m Skip morning sky flats -R FORCE tele reference -F Skip focus run at the start -P Run FIXpoint at the start, to fix the pointing -E get 2 defocused images for collimation The syntax is, e.g., GOrobot -D -F. In addition to "tele ref" the Telshell "FIXpoint" command adjusts the pointing using astrometry on a short exposure near zenith. The same may be achieved by adding the target "FIXpoint" in your queues (more than once if so desired by setting a period for repeats or the time of observation). The Keplercam dewar is filled automatically at about 5 pm and 7 am. These times vary seasonally to allow for dome flats before the evening LN2 fill. When the telescope points at the flat screen, a lot of LN2 spills out of the dewar if it's full, so it's best to be at zenith for each fill. The telescope drives are always enabled, thus avoiding the need for human intervention. Dome flats will start about 2 hours before sky flats and take about 30 minutes for the standard set. Stop the robot with STOProbot in any xterm. It can be stopped and restarted (type "GOrobot" in Telshell) at will. Type ftime to see when the robot will start dome and sky flats and when the dewar will be filled. To see how long dome flats will take, type showdt (showdt -h provides help). Unless disabled with a flag, first u and then r sky flats are attempted at dusk; if they fail, they are attempted again at dawn, r first, then u. The filters used for skyflats may be changed by changing the contents of files ~observer/ROBOT/FLATS/SKYFILTx where x=1, 2. ************VIEWING ROBOT LOGS To view a night's detailed log, use SHOWlogs [yymmdd] To view a log of dome flats, use SHOWdomes [yymmdd] To view a log of sky flats, use SHOWskies [yymmdd] To view a list of a night's exposures, use SHOWnight [yymmdd] or the Nightlog button in GOqueues (see below) To view the .heartbeat file showing the health of the system, type SHOWheart. ************DATABASE If you have a catalog that you wish to add to the database, you can have it harvested, or enter it with the queue GUI (see next section). See an example catalog in ~observer/ROBOT/RTS2-F-example.mctR. To add targets from a catalog file to the database, harvest it: sftp the file to ~observer/ROBOT/CATALOGS. If you do this before 2pm MST, the harvester will process your catalog at that time. For manual entry, after the sftp, in any xterm type harvestcat. After it runs, you see some messages and your catalog is moved to CATALOGS/PROCESSED. Catalogs are checked for formatting errors. The harvester sends an email to observer at flwo48 with a brief diagnostic. If there is a problem with your catalog, it is not moved to PROCESSED. You may modify a number of the properties of a target in the database with command "rts2-target." Use "man rts2-target" to view its many options. Some of these are: rts2-target -p priority --pi PI name --program program name --proper-motion proper motion --delete-targets --tempdisable seconds --airmass lower:upper --lunarDistance lower:upper --lunarAltitude lower:upper --maxRepeats maximal number of observations You may list the properties of a target in the database with command "rts2-targetinfo" For example: rts2-targetinfo SDSS1029 1110 O 10:29:13.350 +26:23:31.80 -02:37 1.22 +55 18 28.08 268 29 54.44 rising SDSS1029 KCAM:'ampcen=2 autoguide=OFF tempdisable 1d filter=r for 4 { E 300 } ' \-- expected duration: 20m 00.000s rts2-targetinfo %X% lists the information in the database for targets that contain X (regardless of case). Use "man rts2-targetinfo" to view its many options. ************QUEUES: general options To use queues, set up your queue each night as a manual queue (OR see below for recurring observations). The guts of RTS2-F are the "selector" and the "scheduler" which pick targets to observe. The scheduler selects targets automatically from the database, unless queues are populated. Automatic scheduling follows the rules for each target in the database, including their priorities. The options available in the GOqueues GUI are calibration, manual or service queues. Queues are always enabled, but do nothing unles they are populated with targets. The selector visits each queue and observes available targets under the constraints for each queue. The order is a repeating cycle of calibration, manual and service queues, until all queues are empty. To see what you wrought, simulate a night's run: once you have created and saved queues, click the "Simulate" button to see how your queues will be executed. Also click "Update" in the simulator window. With this option, you can go back to each queue GUI to correct any problems in your queues. If the simulation is not what you expect compared to the number of targets and exposures in the queues, exit GOqueues, restart it, and again click on "Simulate." That should fix the problem. If the queues are all empty and "Automatic selector" is enabled in the queues GUI, RTS2-F selects targets from the database and observes them until the end of the night. If "Automatic selector" is disabled, the robot idles unless and until a target is available in the queues. It prompts so you may let it wait, or just exit (with ctrl-C in Telshell or STOProbot in any other xterm). To enter targets into a queue, in any xterm type GOqueues. The queues are shown top to bottom in the order the selector follows, checking them for available targets: calibration, manual and service. There are then 4 options: a) Manually, click "Add targets", bottom right, third entry from the top. All the targets in the database appear (the window takes a minute to pop up). To expand the original window and the target window for a better view, click the corners and drag with the mouse. Find your targets by scrolling down; you can select RA, DEC or name ordering by clicking on the appropriate button at the top of the window. You may also start typing the name of a target and the list is moved to the targets whose names start with what you typed. Select targets by CTRL+clicking on them and then on "Add" at the bottom middle of the window. If you have a sequence of contiguous targets, click on the first one, then SHIFT+click on the last one, then "Add." For VNC other than the preferred TightVNC, make sure you have the proper settings to do this (some may just not work). You now see your targets in the queue window. Targets that are currently not visible are highlighted in magenta; a window pops up and asks whether you would like to switch its start time to an appropriate value (click Yes or No). Targets that will not be visible during the current night normally should not go into a queue; a warning window pops up (usually click skip); if you are adding a target to the service queue, you may want to click "add anyway" but be careful. b) From a catalog, click "Add catalog" which will prompt you for the location of your catalog; click on open once you find it and click on its name in the window that pops up. The targets entered from a catalog into a queue are also added to the database if they are not in it already. The information for a target in the catalog replaces that in the database if the target is already there, except for records that show, e.g., when and how many times the target was observed. It is critical that, if you are not repeating observations such as for a transit, you make sure to check "Remove observed" before you load your catalog. If you try to load a catalog and "Remove observed" is not checked, the load will fail, GOqueues will hang, and you will have to Rstop/Rstart, type GOweather, switch the mode to "ON" by clicking on the GOweather button for the state (or "Standby" if the weather is bad), check "Remove Observed" and reload your catalog, all of which will waste significant time. c) From a file, click "Load from file." This is useful if you have previously saved a queue with the "Save to file" button or if you wish to restore an "autosaved" queue. The latter are written automatically as observations proceed each night. d) From a file again, if you have previously saved all the queues with the 2nd button from the bottom left corner of the GOqueues window, the 1st button lets you reload all the queues. If you use FIFO, rearrange your targets with the arrows (the barred arrow takes the target to the top). If you selected Meridian West-East, the observing order is decided by the scheduler; see below to check what the scheduler will observe. You can delete any of the targets you entered by clicking the trash can symbol (it may be hidden, it's top+right). Once done with the adding, close the targets window by clicking "Exit." Once you are happy with your targets, click the "Save to server" button, then "Done." If you wish to analyze your night, click "Simulate." Once updated to the total number of targets in your queues, click "Edit" to view the simulation. It will show your targets with the times of execution of observations and with red highlighting violations of constraints (which may render the targets unoservable) or when a target is planned to be observed at the same time slot as another. If you would like your targets observed at fixed times, use the "T" for each target and enter values in the "from" and "to" boxes. You can use just one of these to tell the robot to start by about the "from" or to end by about the "to" time. But, be aware that if you use Meridian West-East mode, a target has to be on top to have its time taken into account by the system. A better approach is to put all time-critical targets into the manual queue: (1) select again the main queue window, or relaunch it (type "GOqueues" in any xterm) (2) click Edit next to manual queue (e.g. NOT service queue, as before) (3) add targets to the queue, select FIFO mode (top right corner of the window) (4) click on the T button to specify start/stop MST times. For observations after midnight MST you must specify the next day. Click "Set" once done, or Cancel/Clear to cancel the time (5) make sure that targets in the manual queue are arranged as they should be observed. The system will not re-arrange their order (6) make sure to leave enough time between targets so all the exposures you are scheduling can be taken before the start time for the following exposure (command rts2-targetinfo TARGETNAME shows an estimate of the time required for each target, including overheads) (7) in a case such as a transit, leave the number of exposures in your catalog set to 1, but uncheck "Remove observed" (right, near bottom) so the target remains in the queue until the "to" time and is observed repeatedly (8) if you are not repeating observations such as for a transit, make sure to check "Remove observed" before you load your targets (9) click "Save to server" ************QUEUES: A new option, the Scheduling Window The Scheduling Window (SW) is used to coordinate start/end times with other queues. The SW allows targets from the bottom (in terms of priority), e.g. service queue, to overrun the start time of targets in an upper, e.g. manual or calibration queue. It should be used if you don't need exact start times for your observations - the robot will then optimize the schedule. An SW may be specified for each queue, in a box marked "Window", just above the "Skip Targets" button. As an example, consider target C1 in the calibration queue with start time 22:00 MST, and targets S1, S2 and S3 in the service queue without start times. Assume it is 21:50 MST, and S1, S2 and S3 have scheduled durations of 20 minutes. If SW is set to 0, the robot will be idle or will select short-duration targets from the database to fill the 10 minutes remaining until 22:00, when C1 is scheduled to start. If you set SW to 10 minutes, at 21:50 the robot will pick up S1, and delay the start of C1 to 22:10. The general rule is to set SW slightly larger than the durations you see in the service queue, with typical durations of 20 minutes. For such durations, 25 minutes would be a good choice. Then targets with start time T can start from T till T+SW, and the system uses those extra SW minutes to optimise the overall schedule. If you want your target to start strictly at a given time, set SW to 0. If you want to start your target no later than some given time, subtract SW from the given time and use it as start time. The SW does not affect end times. ***********QUEUES: recurring observations To set up recurring observations in the service queue, type GOcircular in any xterm. To add targets to the file, type: GOcircular TARGET1 TARGET2 ... To remove targets from the file, type: GOcircular -TARGET1 -TARGET2 ... You may mix add/remove, e.g.: GOcircular TARGET1 -TARGET2 ... The command prints out circular.q, shows the syntax to load targets, and prompts to reload the service queue; hit RETURN if satisfied. GOcircular updates the file: /home/observer/ROBOT/QUEUES/circular.q. A cron job loads this file into the service queue each day at 14:00MST. If you wish to raise the likelihood that a target in the service queue will be observed, you may do so by editing circular.q and placing the target in question at or near the top of the file (make sure not to disturb the formating of the file). ************MONITORING THE ROBOT With the GUI: (1) Type "GOqueues" in any xterm. (2) If the robot is active and the state is night, near the top of the window you will see the next target picked by the selector, and which queue it is selected from ("automatic" if from the autonomous scheduler, which is unlikely to be your choice; this indicates that you arrange the target incorrectly in the queue). (3) In the "queue" window (after you click on the "Edit" button next to the queue name), you will see current order of targets. This might change during the night; if you are using "Highest elevation", "West-east", "Meridian West-east", the queue order will be changed during the night. The queue order will not change if you use "FIFO" or "Circular". (4) You can access the night-logs by clicking the "Nightlog" button in the queue window (or use SHOWnight in any xterm, see entry above). Please make sure you are viewing the current night (select night start date by clicking "Select night", you will see nights with observations in bold). If there are no observations for the current night, Nightlog shows the last night with data; see the timestamp for each observation to make sure. (5) Click "Update" or "Load from server" frequently to update the screen. Automatic updates will be added soon. (6) You can run the real-time GUI with the command GOweather. The system updates the weather values shown in the window. With MONrobot: In any xterm, type MONrobot to view a cornucopia of detail. There are 3 sub-windows in the MONrobot window: .-------. | | 2 | .1|-----. | | 3 | `-------' Move down to "SEL" in sub-window 1 (with the down arrow key), then hit TAB to go to sub-window 2, then use the down arrow to move down to the service queue. There you will see your targets lined up. Another 2 TABs will bring you back to sub-window 1. While in the SEL area, you can also view the states of the calibration, manual and service queues. Items with a "W" on their left can be modified by moving down to each item with the down arrow and hitting RET. You will be offered a menu with options you select with the down arrow and hitting RET again. If the item is a number, text or a list of these, you can edit the values and hit RET when done. The "selector_queue" allows you to switch between the queues the robot will execute (the options are calibration, manual or service). You can switch between these anytime. Move down to WEATHER in sub-window 1; there you see current weather conditions and limits such as for wind and humidity. Hit TAB to move to sub-window 2, then use the down arrow to view additional readings.