-
Notifications
You must be signed in to change notification settings - Fork 1
Procedure for Diagnosing Processing Output
These instructions show how to run the notebook provided to check and plot the results of template processing, search processing, and fail files. The notebook is located at gw_workflow/JobHelper/jobHelper.ipynb
.
-
SSH into your preferred DES machine as the
desgw
user- I prefer des91 or des90 as they are fast
-
Type
O4
and hit Enter. This command brings you to the DESGW O4 directory with all the event files -
Navigate to your chosen event's JobHelper directory, which should be at
O4/{event_id}/gw_workflow/JobHelper/
, where{event_id}
is replaced with your event's ID -
Run the command
mi38_jupyter
- This command initializes the correct conda environment and starts the jupyter notebook server
-
Open a new different terminal window on your own computer. In this terminal, run the command
ssh -K -N -L localhost:9999:localhost:9999 desgw@{machine_id}.fnal.gov
, replacing{machine_id}
with the name of the DES machine you are using, e.g.des91
.-
This step may differ if you are using Windows OS, but I think it works the same way as any normal ssh command
-
If there is already a Jupyter server running on your current DES machine, the port 9999 may already be in use (Jupyter will warn you about this on startup in the original terminal window). In this case, this command may cause errors at some point. If this happens, change every 9999 to 10000 in this command, as Jupyter will automatically increment the port number by 1.
-
-
In the original terminal window, it may take some time for the Jupyter server to be ready. Once it is ready, it will say "To access the server, open this file in a browser, or copy and paste one of these URLs:". Copy and Paste the URL given into a web browser, as indicated.
- The URL should begin with
http://localhost:9999/...
- The URL should begin with
-
Your web browser will open and show the JobHelper directory in Jupyter. Doubleclick
jobHelper.ipynb
and it will open in a new tab.
The notebook itself should require very little editing, if at all. Run each cell using Shift+Enter. Alternatively, in the Jupyter menu bar, use Run > Run All Cells
.
There are some plots in the notebook. The first one shows a histogram of templates based on how many CCDs have completed Single Epoch Processing for each template. The second plot shows histograms of search exposures, based on how many of their CCDs have *.gz
files or *.FAIL
files. The third plot breaks down the types of failure modes which are given in the above *.FAIL
files.
If any errors arise while running this notebook, contact Isaac McMahon.