[Gotthard2][Dark] Calculate noise bpixs in multiple iterations

Description

@mramilli mentioned that reiterating on calculating NOISE_OUT_OF_THRESHOLD badpixel type for gotthard2 showed a difference in the number of badpixels identified.

In the below figure are his test results. Each plot is the difference in bpix numbers for each gain for one of the production constants.

Below are my test results showing that sometimes you can get a different number of badpixels with more iterations than 2.

This MR exposes an iteration parameter to control it if needed (default 5).

Now GH2 darks will iterate to calculate the more accurate number of badpixels. each iteration costs =< 1ms, so no obvious performance drawback on dark processing.

How Has This Been Tested?

Relevant Documents (optional)

Types of changes

Checklist:

Reviewers

@mramilli @schmidtp @kluyvert

changed title from Calculate bpixs for gotthard2 in multiple iterations to [Gotthard2][Dark] Calculate noise bpixs in multiple iterations

So uhm, it seems quite obvious to me the number of bad pixels will increase by applying the NOISE_OUT_OF_THRESHOLD criterion multiple times. As values outside of the region are set to nan when flagged, they are removed for the following median estimation, causing it likely to shift. Values previously inside the region are then outside, causing them to be flagged as well. At some point these median shifts will (happen to!) not cause any pixels to slip out, and the bad pixel mask no longer changes.

The question, why would we want to do this? This will almost always make the allowed noise interval smaller than badpixel_threshold_sigma * std by shifting the median around.

@mramilli

because a GH2 module has only 1280 strips, and a very noisy strip can skew the distribution a lot, causing some potentially bad strips to be missed. This is an attempt to remove the very bad strips at first and then calculate new outliers when these are gone.

assigned to @ahmedk

added 259 commits

• 4b1b01ca...fe329b7e - 257 commits from branch master
• 642b4f6b - Calculate bpixs for gotthard2 in multiple iterations
• 2a4bd898 - fix for the second module

Compare with previous version

added Bug label

The tests ran as expected and the failed status is related to new badpixels constant values:

test_report.pdf

--- Compare HDF5 files  ----
REF: /gpfs/exfel/d/cal_tst/reference_folder/SA1_XTD9_HIREX/SA1_XTD9_HIREX-DARK
NEW: /gpfs/exfel/data/scratch/xcaltst/test/feat/iterate_to_calculate_noise_bpix/SA1_XTD9_HIREX/SA1_XTD9_HIREX-DARK
const_BadPixelsDark_GH2-0149.h5
  ~ CHANGED: /data (Data changed)
const_BadPixelsDark_GH2-0150.h5
  ~ CHANGED: /data (Data changed)
const_Noise_GH2-0149.h5 - ✓ no changes
const_Noise_GH2-0150.h5 - ✓ no changes
const_Offset_GH2-0149.h5 - ✓ no changes
const_Offset_GH2-0150.h5 - ✓ no changes

added 1 commit

• 014e8a94 - improve comment and use for loop instead

Compare with previous version

added 1 commit

• 1fafed6b - move OFFSET_NOISE_EVAL_ERROR into the loop as well as it is calculated from noise map also

Compare with previous version

Let's go with the iterative approach for now. I tried some covariance-based anomaly detection which seems to work well, but it require some additional tuning for now.

Anything left to do? LGTM for the current revision.

There is nothing left to do, I missed seeing your past comment that we go with this for now.

Thanks for the review!

added Test CAL Data Affected label

merged

mentioned in commit dea013ae

changed milestone to %3.15.2