Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change std outlier condition to be multiple of median std #960

Open
wants to merge 2 commits into
base: ctapipe_0.17
Choose a base branch
from
Open

Change std outlier condition to be multiple of median std #960

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c2e2cce
Change std outlier condition to be multiple of median std
maxnoe Apr 4, 2022
d04522b
Merge branch 'master' into outlier_conditions
rlopezcoto May 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 2 additions & 5 deletions lstchain/calib/camera/flatfield.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ class FlasherFlatFieldCalculator(FlatFieldCalculator):
help='Interval of accepted charge values (fraction with respect to camera median value)'
).tag(config=True)
charge_std_cut_outliers = List(
[-3, 3],
help='Interval (number of std) of accepted charge standard deviation around camera median value'
[1/3, 3],
help='Interval (fraction with respect to camera median value) of accepted charge standard deviation'
).tag(config=True)
time_cut_outliers = List(
[0, 60], help="Interval (in waveform samples) of accepted time values"
Expand Down Expand Up @@ -343,9 +343,6 @@ def calculate_relative_gain_results(
# median of the std over the camera
median_of_pixel_std = np.ma.median(pixel_std, axis=1)

# std of the std over camera
std_of_pixel_std = np.ma.std(pixel_std, axis=1)

# relative gain
relative_gain_event = masked_trace_integral / event_median[:, :, np.newaxis]

Expand Down
6 changes: 3 additions & 3 deletions lstchain/calib/camera/pedestals.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,13 +38,13 @@ class PedestalIntegrator(PedestalCalculator):

"""
charge_median_cut_outliers = List(
[-3, 3],
[-4, 4],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this changed 3->4?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even if we assume that all pixels follow a common distribution, cutting an interval of 3 sigma will always remove ~5 pixels erroneously. That's way to harsh.

With 4 sigma, the expected number of erroneously rejected pixels is 0.1:

In [10]: def expected_rejected_pixels(a, b):
    ...:     return (1 - (norm.cdf(b) - norm.cdf(a))) * 1855
    ...: 

In [11]: expected_rejected_pixels(-3, 3)
Out[11]: 5.008121697347684

In [12]: expected_rejected_pixels(-4, 4)
Out[12]: 0.11750030720087512

help='Interval (number of std) of accepted charge values around camera median value'
).tag(config=True)

charge_std_cut_outliers = List(
[-3, 3],
help='Interval (number of std) of accepted charge standard deviation around camera median value'
[1/3, 3],
help='Interval (factor of median over pixels of std over events) of accepted charge standard deviation',
).tag(config=True)

time_sampling_correction_path = Path(
Expand Down