[AGIPD] Only use bad pixels from darks in agipdutils.baseline_correct_via_stripes

Description

In response to https://git.xfel.eu/calibration/planning/-/issues/96, @jsztuk asked to only use bad pixels obtained from darks to be used for masking in baseline correction via stripes.

How Has This Been Tested?

@jsztuk verified the correction result with this change.

Types of changes

• Changed behaviour (non-breaking change which fixes an issue)

Reviewers

@jsztuk @ahmedk @kluyvert

changed milestone to %3.4.3

LGTM

`GAIN_THRESHOLDING_ERROR' is not needed as we already select only pixels in HG, this bit is only relevant for pixels in MG or LG gain as there is an issue with gain decoding (i.e. it is not possible for some pixels to separate MG from LG based on the gain value informations)

There are two classes BadPixels and BadPixelsFF BadPixelsFF.FIT_FAILED bits values are used internally in FF processing notebook. To fill bits for BadPixels, they are grouped

for mod in modules:
    bpmask[mod] = np.zeros(fit_data[mod]['mask'].shape).astype(np.int32)
    bpmask[mod][ any_in(fit_data[mod]['mask'], BadPixelsFF.NO_ENTRY.value) ] = BadPixels.FF_NO_ENTRIES.value
    bpmask[mod][ any_in(fit_data[mod]['mask'], 
                        BadPixelsFF.GAIN_DEVIATION.value) ] |= BadPixels.FF_GAIN_DEVIATION.value
    bpmask[mod][ any_in(fit_data[mod]['mask'], 
                        BadPixelsFF.FIT_FAILED.value | BadPixelsFF.ACCURATE_COVAR.value |
                        BadPixelsFF.CHI2_THRESHOLD.value | BadPixelsFF.GAIN_THRESHOLD.value |
                        BadPixelsFF.NOISE_PEAK_THRESHOLD.value | BadPixelsFF.PEAK_WIDTH_THRESHOLD.value) ] |= BadPixels.FF_GAIN_EVAL_ERROR.value

Aha, I had missed that translation from BadPixelsFF to BadPixels.

LGTM

added 1 commit

• a1b8f7c0 - Move BadPixelsFF to cal_tools.agipdutils_ff and clarify internal usage

Compare with previous version

Thanks for clarification of the BadPixelsFF enum. I've moved it to agipdutils_ff and clarified the comment to prevent any confusion in the future.

EDIT: Force-pushed to fix missing comma

added 1 commit

• 9c60cfd0 - Move BadPixelsFF to cal_tools.agipdutils_ff and clarify internal usage

Compare with previous version

i have installed the branch (i.e. before @schmidtp moved BadPixelsFF to agipdutils_ff :)) and runs a small test on r21 from p002929. I processed data with --xray-gain and w/o xray-gain.

1. The bls values are the same in both cases

2. data is properly corrected. I compared results obtained with the old version and the current branch. the same set of corrections was applied to the data. The plot shows mean image for one train after bug fix (this branch):

For comparision, the same train before the bug fix:

resolved all threads

LGTM

Thanks for review and also the extensive testing from @jsztuk

merged

mentioned in commit c134799a