Repetition #43

cwjames1983 · 2023-06-21T13:36:10Z

This is a pretty large PR for FRB repetition behaviour!

It's composed of four components:

the general method, encapsulated in repeat_grid.py;
Miscellaneous additions/fixes to the main codebase
Specific work for modelling CHIME repetition in /papers/repeaters
Some extra work for FRB 210912

The current major issue with this is that it uses the old survey files, since I began this before the survey class was updated.

Hence, what I propose is to accept this PR, merge it with the main code, then I get a new clean checkout, and update all analyses to the new format (this won't be done urgently however, so maybe I just rename everything "old"?)

…n paper

papers/Repeaters/BeamData/all_freq_beam_10_bins.npy

profxj · 2023-06-28T17:36:29Z

papers/Repeaters/Obsolete/check_normalisation.py

@@ -0,0 +1,98 @@
+""" 


I'm assuming this code in Obsolete need
not be reviewed. But let me know if that's
not right.

This is true. Obsolete is for files which may in theory be useful, so I don't want them deleted; but they have no direct relevance to the analysis, so they're kept out of the way.

papers/Repeaters/Surveys/CHIME_150_fbar.dat

profxj · 2023-06-28T17:39:41Z

papers/Repeaters/add_mc_calculations.py

+import matplotlib
+import time
+from zdm import beams
+beams.beams_path = '/Users/cjames/CRAFT/FRB_library/Git/H0paper/papers/Repeaters/BeamData/'


here and throughout, can we avoid hard-coding paths?

Yep, I thought I had removed all of these, but clearly not! All done now. Actually, most of this was obsolete stuff I just hadn't deleted...

profxj · 2023-06-28T17:41:27Z

papers/Repeaters/utilities.py

@@ -0,0 +1,446 @@
+"""


please move this under zdm, e.g. in a folder named chime

same comment for any of the code in this current folder that would be needed by others to do "simple" CHIME analyses

It's not needed for simple CHIME analysis - originally, yes it was, but not anymore! See file zdm/scripts/load_chime_data.py

i.e. the file is staying where it is :-)

profxj · 2023-06-28T17:53:26Z

zdm/repeat_grid.py

+        p2no_reps = np.exp(-N2reps)
+
+
+def poisson_z(Robs,*args):


move this code elsewhere?

Moved to the recycle bin - I'd honestly forgotten this was in there, it's a relic from a first attempt at an analytic calculation. So deleted.

profxj · 2023-06-28T17:53:51Z

zdm/repeat_grid.py

+
+
+
+def calc_p_rep(Cr,V):


move this code elsewhere?

e.g. repeater_utils.py

Also legacy code, now deleted

profxj · 2023-06-28T17:54:42Z

zdm/scripts/repeaters.py

@@ -0,0 +1,191 @@
+""" 
+This script creates zdm grids and plots localised FRBs


I assume this doc is incorrect. Please update

Yes, indeed it is - I'd forgotten about this, this file is now updated in several ways

profxj · 2023-06-28T17:55:23Z

zdm/survey.py

-        if self.NFRB==0:
-            raise ValueError('No FRBs found in file '+filename) #change this?
-
+        #if self.NFRB==0:


rm this code

actually, this is OldSurvey

we should not be using this anymore, right?

Correct, we shouldn't be using it. Maybe we shortly shift this to a separate "old_survey.py" file, to make development easier? However, I had to change this, since it's vital to allow surveys to have no FRBs in them. So I'll keep this code there, but we can discuss what to do about old_survey. I think we need to keep the code for posterity, but maybe shift it somewhere? Or I guess, this is what version control is for. Once we are 100% sure we have converted over all old .dat files, then we delete

profxj · 2023-06-28T18:00:25Z

zdm/survey.py

@@ -32,7 +32,6 @@ class Survey:



Just realized here that this branch has not pulled in main yet,
so I am going to stop reviewing until we do.

profxj · 2023-07-08T20:46:02Z

@cwjames1983 -- I was aiming to generate p(z|DM) for CHIME on this branch
so started on the README.txt file in papers/Repeaters, e.g. this step

################## dm_bias_fit.py  ####################

This produces a fit to the DM bias from CHIME Cat1.
Output is placed in "DMBiasFit/". The main figure
produced is "chime_bias_fit.py.

The key output of this has already been copied
to "survey.py" in /zdm/ to model the response of
the CHIME experiment.

This script also produces Figure 4 of the paper.

But the file chime_dm_bias.dat does not appear
to have been checked in to data/Misc (maybe call
that folder CHIME instead?). Please do so when
you have a chance. thx

No idea why the remote was changed, but merging before pushing changes...

profxj · 2023-07-14T13:02:59Z

papers/Repeaters/evaluate_mc_calculation.py

+from zdm import beams
+
+
+beams.beams_path = '/Users/cjames/CRAFT/FRB_library/Git/H0paper/papers/Repeaters/BeamData/'


are these the same files as in zdm/papers/Repeaters/BeamData/ ?

Yes, and this is an old pathname, now deprecated and removed. Files now shifted to zdm/data/BeamData/CHIME/

profxj · 2023-07-14T13:06:46Z

zdm/iteration.py

+    const *= factor
+    return const
+
+def OldConvertToMeaningfulConstant(pset):


profxj · 2023-07-14T13:13:04Z

zdm/repeat_grid.py

+
+        self.calc_Rthresh(Exact=Exact,MC=MC,doplots=doplots)
+
+    def update(self,Rmin = None,Rmax = None,Rgamma = None):


define Rmin, Rmax, Rgamma in doc string

profxj · 2023-07-14T13:14:22Z

zdm/repeat_grid.py

+        return Rc,Ntot
+
+
+    def calc_Rthresh(self,Exact=True,MC=False,Pthresh=0.01,doplots=False,old=[False,False,False]):


explain what Exact does

profxj · 2023-07-14T13:19:43Z

zdm/scripts/repeaters.py

+from zdm import survey
+from matplotlib import pyplot as plt
+
+def main():


I don't think this script does much of anything.
Remove this file?

You are correct - I've now updated it to be useful

profxj · 2023-07-14T13:21:07Z

zdm/survey.py

@@ -88,7 +84,8 @@ def get_efficiency_from_wlist(self,DMlist,wlist,plist,
        """
        efficiencies=np.zeros([wlist.size,DMlist.size])
        if addGalacticDM:
-            toAdd = self.DMhalo + np.mean(self.DMGs)
+            # the following is safe against surveys with zero FRBs


How many of these changes need to propogate to the new Survey class?

I've now fully debugged the new survey class to be robust against surveys with 0 and >0 FRBs. The methods are quite different in the new survey class.

profxj

Ok, finished a careful and comprehensive pass

…ties

…etition files to standard location

…hanges to plotting scripts for Repeaters paper.

profxj · 2023-10-18T18:31:39Z

zdm/scripts/repeaters.py

@@ -1,7 +1,14 @@
 """ 
-This script creates zdm grids and plots localised FRBs
+This script shows how to use repeating FRB grids.


I tried running this script, but it fails right away.

profxj · 2023-10-18T18:34:25Z

zdm/scripts/load_chime_data.py

@@ -5,6 +5,9 @@
 It simply loads in CHIME data and makes sure
 everything can be read correctly.

+It also demonstrates how to generate "repeat grids"


I commented out 2 items that weren't used (and were causing my run to crash)

after that, 3 lovely plots were generated in a CHIME/ folder! :)

profxj

got the load_chime_data.py script to run successfully
with sensible looking figures generated.

I'm convinced this is ready

Clancy James added 7 commits May 30, 2022 06:46

Begun changing files

7a86ba1

First working repeater simulation script

842dc1c

Finally debugged code, included hack to avoid low-zero problem

bc4c2c5

Added a whole bunch of updates from writing the repeaters paper

177a3a7

Massive upload of many repeater files, and 210912

cb89dd8

Huge commit of final files prior to first submission of FRB repetitio…

4b0ed70

…n paper

Adding many small CHIME beam and survey files to the repository

507763c

profxj self-requested a review June 28, 2023 17:32

profxj requested changes Jun 28, 2023

View reviewed changes

profxj and others added 7 commits July 8, 2023 13:49

this and that

876fe94

Minor changes, especially adding chime dm bias data

12cab13

Added very minor changes to various files

40bee47

Finished resolving conflicts, but yet to test

5518d4e

Modified all CHIME repeater simulation files to use old CHIME surveys

7525c43

Added trivial energy calculation script

dab966b

Merge remote-tracking branch 'origin/repetition' into repetition

337cfb6

No idea why the remote was changed, but merging before pushing changes...

profxj reviewed Jul 14, 2023

View reviewed changes

profxj requested changes Jul 14, 2023

View reviewed changes

Clancy James added 6 commits July 17, 2023 16:03

Forgot small change required on fit single bursts - now imports utili…

006284b

…ties

Adding CHIME ecsv files

64a5a7d

Refactored surveys to include beam and width information. Updated rep…

a8c29d1

…etition files to standard location

Added script to investigate CHIME threshold

d6e6036

Fixed no FRB problem in new survey. Updated CHIME ecsv files. Minor c…

7c89ed4

…hanges to plotting scripts for Repeaters paper.

Removed all absolute paths

4bf7d8e

Clancy James added 5 commits September 28, 2023 14:51

First lot of responses to Xs requests on the pull request review

93f7d7b

Final pass at addressing xs comments

09433ef

resolved conflicts

2a75337

fixed bug in plotgrid2

c93c4fb

fixed most bugs related to test scripts

8e939bc

profxj reviewed Oct 18, 2023

View reviewed changes

profxj approved these changes Oct 18, 2023

View reviewed changes

mionr edit

c94aadb

cwjames1983 merged commit a28d973 into main Oct 20, 2023
6 checks passed

profxj deleted the repetition branch October 21, 2024 14:56

		@@ -0,0 +1,191 @@
		"""
		This script creates zdm grids and plots localised FRBs

		from zdm import beams


		beams.beams_path = '/Users/cjames/CRAFT/FRB_library/Git/H0paper/papers/Repeaters/BeamData/'


		self.calc_Rthresh(Exact=Exact,MC=MC,doplots=doplots)

		def update(self,Rmin = None,Rmax = None,Rgamma = None):

		return Rc,Ntot


		def calc_Rthresh(self,Exact=True,MC=False,Pthresh=0.01,doplots=False,old=[False,False,False]):

Repetition #43

Repetition #43

Conversation

cwjames1983 commented Jun 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

profxj commented Jul 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

profxj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

profxj left a comment

Choose a reason for hiding this comment