cta-observatory · ranieremenezes · Oct 4, 2023 · Oct 5, 2023 · Oct 5, 2023 · Oct 6, 2023
diff --git a/docs/user-guide/IT_analysis_NSB.rst b/docs/user-guide/IT_analysis_NSB.rst
@@ -0,0 +1,39 @@
+.. _IT_data_NSB:
+
+IT Cluster analysis: NSB_matching
+=================================
+
+TODO
+----- 
+
+1. Database (joint observations): choose format and share the needed file
+
+Quick start tutorial
+--------------------
+
+Update the config_h5.yaml file with the time range, target name and bad runs.
+
+``list_from_h5.py`` to create the lists with the runs
+
+``nsb_level.py`` If there is no file called "config_general" or if you want to set another name to the config file, you can use the option "-c PG1553_config_general.yaml". This is actually valid for all scripts in the NSB branch. This script computes the NSB level for each LST run found by the first script. For a single LST run, it takes around 50 min to compute the NSB level, so we launch the jobs in parallel. For each job, it creates one txt file with information about the NSB. 
+
+``collect_nsb.py`` This script stacks the information from the txt files created above separated by NSB level.
+
+``nsb_setting_up_config_and_dir.py`` Creates the directories for DL1 MAGIC data separated by observation period and processes MAGIC data up to DL1.
+
+To merge the subruns into runs, the M1 and M2 runs, and then runs into nights, we do:
+
+``nsb_merge_subruns.py``
+
+``nsb_merge_M1_M2_runs.py``
+
+``nsb_merge_M1_M2_night.py``
+
+``nsb_coincident_events.py`` Find the MAGIC-LST coincident events and organize them by NSB level.
+
+``nsb_stereo_events.py`` Computes the stereo parameters for the coincident runs.
+
+
+
+
+
diff --git a/docs/user-guide/IT_analysis_data_MC.rst b/docs/user-guide/IT_analysis_data_MC.rst
@@ -0,0 +1,228 @@
+.. _IT_data_MC:
+
+IT Cluster analysis: data + MC reduction tutorial
+=================================================
+
+1) The very first step to reduce MAGIC-LST data in IT Container is to have remote access/credentials to this cluster, so provide one. Once you have it, the connection steps are the following:  
+
+| Authorized institute server (Client) &rarr;  ssh connection to CTALaPalma &rarr; ssh connection to cp01/02  
+
+2) Once connected to the IT Container, install MAGIC-CTA-PIPE (e.g. in your home directory in the IT Container) following the tutorial here: https://github.com/cta-observatory/magic-cta-pipe
+
+3) Do not forget to open the magic-lst environment with the command `conda activate magic-lst` before starting the analysis
+
+Preparations and DL0 to DL1
+---------------------------
+
+In this step, we will convert the MAGIC and Monte Carlo (MC) Data Level (DL) 0 to DL1 (our goal is to reach DL3).
+
+Now copy all the python scripts available here to your preferred directory (e.g. /fefs/aswg/workspace/yourname/yourprojectname) in the IT Container, as well as the files `config_general.yaml`, `MAGIC_runs.txt` and `LST_runs.txt`.
+
+The file ``config_general.yaml`` must contain the telescope IDs and the directories with the MC data, as shown below:  
+
+    mc_tel_ids:
+        | LST-1: 1
+        | LST-2: 0
+        | LST-3: 0
+        | LST-4: 0
+        | MAGIC-I: 2
+        | MAGIC-II: 3
+
+    directories:
+        | workspace_dir : "/fefs/aswg/workspace/yourname/yourprojectname/" 
+        | target_name   : "CrabTeste"
+        | MC_gammas     : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/sim_telarray"
+        | MC_electrons  : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/Electrons/sim_telarray/" 
+        | MC_helium     : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/Helium/sim_telarray/" 
+        | MC_protons    : "/fefs/aswg/data/mc/DL0/LSTProd2/TrainingDataset/Protons/dec_2276/sim_telarray"
+        | MC_gammadiff  : "/fefs/aswg/data/mc/DL0/LSTProd2/TrainingDataset/GammaDiffuse/dec_2276/sim_telarray/"
+
+    general:
+        | target_RA_deg          : 83.633083 #RA in degrees
+        | target_Dec_deg         : 22.0145   #Dec in degrees
+        | SimTel_version         : "v1.4"    
+        | LST_version            : "v0.9" 
+        | focal_length           : "effective" #effective #nominal
+        | MAGIC_runs             : "MAGIC_runs.txt"  #If there is no MAGIC data, please fill this file with "0, 0"
+        | LST_runs               : "LST_runs.txt"  
+        | proton_train_fraction  : 0.8 # 0.8 means that 80% of the DL1 protons will be used for training the Random Forest
+        | env_name               : magic-lst
+
+
+
+The file ``MAGIC_runs.txt`` looks like that:  
+
+    | 2020_11_19,5093174
+    | 2020_11_19,5093175
+    | 2020_12_08,5093491
+    | 2020_12_08,5093492
+
+
+
+
+The columns here represent the night and run in which you want to select data. Please do not add blank spaces in the rows, as these names will be used to i) find the MAGIC data in the IT Container and ii) create the subdirectories in your working directory. If there is no MAGIC data, please fill this file with "0,0". Similarly, the `LST_runs.txt` file looks like this:
+
+    | 2020_11_18,2923
+    | 2020_11_18,2924
+    | 2020_12_07,3093
+
+
+Note that the LST nights appear as being one day before MAGIC's!!! This is because LST saves the date at the beginning of the night, while MAGIC saves it at the end. If there is no LST data, please fill this file with "0,0". These files are the only ones we need to modify in order to convert DL0 into DL1 data.
+
+In this analysis, we use a wobble of 0.4°.
+
+Night sky background estimation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Before processing the Monte Carlo simulations, we need to estimate the NSB level of our data. We do it by calling the following script:
+
+.. code-block:: console
+
+    $ python nsb_level_MC.py -c config.yaml
+
+    Process name: nsb
+    To check the jobs submitted to the cluster, type: squeue -n nsb
+
+If the file config.yaml is not provided, this script will automatically search for the file "config_general.yaml" in the same directory where you call the script.
+This script will save a series of files named TARGETNAME_LST_nsb_RUNNUMBER.txt with information about the NSB level of each run, and usually takes 50 min to run.
+
+DL0 to DL1
+^^^^^^^^^^
+
+To convert the MAGIC and SimTelArray MCs data into DL1 format, you first do the following:
+
+.. code-block:: console
+
+     $ python setting_up_config_and_dir.py
+
+
+    ***** Linking MC paths - this may take a few minutes ******
+    *** Reducing DL0 to DL1 data - this can take many hours ***
+    Process name: yourprojectnameCrabTeste
+    To check the jobs submitted to the cluster, type: squeue -n yourprojectnameCrabTeste
+
+Note that this script can be run as 
+
+.. code-block:: console
+
+    $ python setting_up_config_and_dir.py --analysis-type onlyMAGIC  
+
+or  
+
+.. code-block:: console
+
+    $ python setting_up_config_and_dir.py --analysis-type onlyMC  
+
+if you want to convert only MAGIC or only MC DL0 files to DL1, respectively.
+
+
+The script ``setting_up_config_and_dir.py`` does a series of things:
+- Evaluates the average NSB level over all runs based on the files generated by the script nsb_level_MC.py.
+- Creates a directory with your source name within the directory ``yourprojectname`` and several subdirectories inside it that are necessary for the rest of the data reduction.
+- Generates a configuration file called config_step1.yaml with telescope ID information, adopted imaging/cleaning cuts, average NSB level. It then puts this configuration file in the directory created in the previous step.
+- Links the MAGIC and MC data addresses to their respective subdirectories defined in the previous steps.
+- Runs the scripts ``lst1_magic_mc_dl0_to_dl1.py`` and ``magic_calib_to_dl1.py`` for each one of the linked data files.
+
+In the file ``config_general.yaml``, the sequence of telescopes is always LST1, LST2, LST3, LST4, MAGIC-I, MAGIC-II. So in this tutorial, we have  
+
+    | LST-1 ID = 1  
+    | LST-2 ID = 0  
+    | LST-3 ID = 0  
+    | LST-4 ID = 0  
+    | MAGIC-I ID = 2  
+    | MAGIC-II ID = 3 
+
+If the telescope ID is set to 0, this means that the telescope is not used in the analysis.
+
+You can check if this process is done by typing  
+
+.. code-block:: console
+
+    $ squeue -n yourprojectnameCrabTeste
+
+or
+
+.. code-block:: console
+
+    $ squeue -u your_user_name
+
+in the terminal. Once it is done, all of the subdirectories in ``/fefs/aswg/workspace/yourname/yourprojectname/CrabTeste/DL1/`` will be filled with files of the type `dl1_[...]_LST1_MAGIC1_MAGIC2_runXXXXXX.h5` for the MCs and `dl1_MX.RunXXXXXX.0XX.h5` for the MAGIC runs. The next step of the conversion of DL0 to DL1 is to split the DL1 MC proton sample into "train" and "test" datasets (these will be used later in the Random Forest event classification and to do some diagnostic plots) and to merge all the MAGIC data files such that in the end, we have only one datafile per night. To do so, we run the following script:
+
+.. code-block:: console
+
+    $ python merging_runs_and_splitting_training_samples.py  
+
+
+
+    ***** Splitting protons into 'train' and 'test' datasets...  
+    ***** Generating merge bashscripts...  
+    ***** Running merge_hdf_files.py in the MAGIC data files...  
+    Process name: merging_CrabTeste  
+    To check the jobs submitted to the cluster, type: squeue -n merging_CrabTeste
+
+
+This script will slice the proton MC sample according to the entry "proton_train_fraction" in the "config_general.yaml" file, and then it will merge the MAGIC data files in the following order:
+- MAGIC subruns are merged into single runs.  
+- MAGIC I and II runs are merged (only if both telescopes are used, of course).  
+- All runs in specific nights are merged, such that in the end we have only one datafile per night.  
+- Proton MC training data is merged.
+- Proton MC testing data is merged.
+- Diffuse MC gammas are merged.
+- MC gammas are merged.
+
+Coincident events and stereo parameters on DL1
+----------------------------------------------
+
+To find coincident events between MAGIC and LST, starting from DL1 data, we run the following script:
+
+.. code-block:: console
+
+    $ python coincident_events.py
+
+This script creates the file config_coincidence.yaml containing the telescope IDs and the following parameters:
+
+    mc_tel_ids:
+        | LST-1: 1
+        | LST-2: 0
+        | LST-3: 0
+        | LST-4: 0
+        | MAGIC-I: 2
+        | MAGIC-II: 3
+
+    event_coincidence:
+        | timestamp_type_lst: "dragon_time"  # select "dragon_time", "tib_time" or "ucts_time"
+        | window_half_width: "300 ns"
+        | pre_offset_search: true
+        | n_pre_offset_search_events: 100
+
+        time_offset:
+            | start: "-10 us"
+            | stop: "0 us"
+
+
+
+It then links the real LST data files to the output directory [...]DL1/Observations/Coincident, and runs the script lst1_magic_event_coincidence.py in all of them.
+
+Once it is done, we add stereo parameters to the MAGIC+LST coincident DL1 data by running:
+
+.. code-block:: console
+
+    $ python stereo_events.py
+
+This script creates the file config_stereo.yaml with the follwoing parameters:
+
+    mc_tel_ids:
+        | LST-1: 1
+        | LST-2: 0
+        | LST-3: 0
+        | LST-4: 0
+        | MAGIC-I: 2
+        | MAGIC-II: 3
+
+    stereo_reco:
+        | quality_cuts: "(intensity > 50) & (width > 0)"
+        | theta_uplim: "6 arcmin"
+
+
+It then creates the output directories for the DL1 with stereo parameters [...]DL1/Observations/Coincident_stereo/SEVERALNIGHTS and [...]/DL1/MC/GAMMAorPROTON/Merged/StereoMerged, and then runs the script lst1_magic_stereo_reco.py in all of the coincident DL1 files. The stereo DL1 files for MC and real data are then saved in these directories.
+
diff --git a/docs/user-guide/index.rst b/docs/user-guide/index.rst
@@ -8,3 +8,4 @@ User Guide
 
    getting-started
    magic-lst-scripts
+
diff --git a/magicctapipe/image/tests/test_calib.py b/magicctapipe/image/tests/test_calib.py
@@ -18,7 +18,6 @@ def tel_id_MAGIC():
 
 
 def test_calibrate_LST(dl0_gamma, config_calib, tel_id_LST):
-
     assigned_tel_ids = [1, 2, 3]
     for input_file in dl0_gamma:
         event_source = EventSource(
@@ -67,7 +66,6 @@ def test_calibrate_LST(dl0_gamma, config_calib, tel_id_LST):
 
 
 def test_calibrate_MAGIC(dl0_gamma, config_calib, tel_id_MAGIC):
-
     assigned_tel_ids = [1, 2, 3]
     for input_file in dl0_gamma:
         event_source = EventSource(
@@ -89,7 +87,6 @@ def test_calibrate_MAGIC(dl0_gamma, config_calib, tel_id_MAGIC):
         config_extractor_magic = {extractor_type_magic: config_magic["image_extractor"]}
         magic_clean = {}
         for k in [1, 2]:
-
             magic_clean[k] = MAGICClean(camera_geoms[k], config_magic["magic_clean"])
         calibrator_magic = CameraCalibrator(
             image_extractor_type=extractor_type_magic,

diff --git a/magicctapipe/io/tests/test_io_monly.py b/magicctapipe/io/tests/test_io_monly.py
@@ -210,7 +210,6 @@ def test_load_mc_dl2_data_file_opt(p_dl2_monly, gamma_dl2_monly):
     """
     dl2_mc = [p for p in gamma_dl2_monly.glob("*")] + [p for p in p_dl2_monly.glob("*")]
     for file in dl2_mc:
-
         data_m, _, _ = load_mc_dl2_data_file(
             str(file), "width>0", "magic_only", "simple"
         )
@@ -434,7 +433,6 @@ def test_load_dl2_data_file_opt(real_dl2_monly):
     Check on event_type
     """
     for file in real_dl2_monly.glob("*"):
-
         data_m, _, _ = load_dl2_data_file(str(file), "width>0", "magic_only", "simple")
 
         assert np.all(data_m["combo_type"] == 0)

diff --git a/magicctapipe/resources/test_config_general_4LST.yaml b/magicctapipe/resources/test_config_general_4LST.yaml
@@ -5,7 +5,25 @@ mc_tel_ids:
     LST-4: 5
     MAGIC-I: 0
     MAGIC-II: 0
+
+directories:
+    workspace_dir : "/fefs/aswg/workspace/raniere/"
+    target_name   : "CrabTeste"
+    MC_gammas     : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/sim_telarray"
+    MC_electrons  : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/Electrons/sim_telarray/" 
+    MC_helium     : "/fefs/aswg/data/mc/DL0/LSTProd2/TestDataset/Helium/sim_telarray/" 
+    MC_protons    : "/fefs/aswg/data/mc/DL0/LSTProd2/TrainingDataset/Protons/dec_2276/sim_telarray"
+    MC_gammadiff  : "/fefs/aswg/data/mc/DL0/LSTProd2/TrainingDataset/GammaDiffuse/dec_2276/sim_telarray/"
 
 general:
+    target_RA_deg          : 83.633083 #RA in degrees
+    target_Dec_deg         : 22.0145   #Dec in degrees
+    SimTel_version         : "v1.4"    
+    LST_version            : "v0.9" 
+    LST_tailcut            : "tailcut84"
     focal_length           : "effective" #effective #nominal
+    MAGIC_runs             : "MAGIC_runs.txt"  #If there is no MAGIC data, please fill this file with "0, 0"
+    LST_runs               : "LST_runs.txt"  
+    proton_train_fraction  : 0.8 # 0.8 means that 80% of the DL1 protons will be used for training the Random Forest
+    env_name               : magic-lst
 
diff --git a/magicctapipe/scripts/lst1_magic/lst1_magic_event_coincidence.py b/magicctapipe/scripts/lst1_magic/lst1_magic_event_coincidence.py
@@ -227,7 +227,6 @@ def event_coincidence(input_file_lst, input_dir_magic, output_dir, config):
     tel_ids = np.unique(event_data_magic.index.get_level_values("tel_id"))
 
     for tel_id in tel_ids:
-
         tel_name = TEL_NAMES[tel_id]
         df_magic = event_data_magic.query(f"tel_id == {tel_id}").copy()
 

diff --git a/magicctapipe/scripts/lst1_magic/lst1_magic_stereo_reco.py b/magicctapipe/scripts/lst1_magic/lst1_magic_stereo_reco.py
@@ -188,7 +188,6 @@ def stereo_reconstruction(input_file, output_dir, config, magic_only_analysis=Fa
     Two_arrays_are_used = Number_of_LSTs_in_use * Number_of_MAGICs_in_use > 0
 
     if (not is_simulation) and (Two_arrays_are_used):
-
         logger.info(
             "\nChecking the angular distances of "
             "the LST and MAGIC pointing directions..."
@@ -238,7 +237,6 @@ def stereo_reconstruction(input_file, output_dir, config, magic_only_analysis=Fa
     multi_indices = event_data.groupby(["obs_id", "event_id"]).size().index
 
     for i_evt, (obs_id, event_id) in enumerate(multi_indices):
-
         if i_evt % 100 == 0:
             logger.info(f"{i_evt} events")
 
@@ -256,7 +254,6 @@ def stereo_reconstruction(input_file, output_dir, config, magic_only_analysis=Fa
         tel_ids = df_evt.index.get_level_values("tel_id")
 
         for tel_id in tel_ids:
-
             df_tel = df_evt.loc[tel_id]
 
             # Assign the telescope information
@@ -361,6 +358,7 @@ def stereo_reconstruction(input_file, output_dir, config, magic_only_analysis=Fa
 
 def main():
     """Main function."""
+
     start_time = time.time()
 
     parser = argparse.ArgumentParser()
Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,3 +8,4 @@ User Guide

		getting-started
		magic-lst-scripts