[AGIPD][CORRECT] Manual photon energy input and consistency histograms for photonization

Description

Through indoctrination I completely there may be non-elastic scattering in https://git.xfel.eu/detectors/pycalibration/-/merge_requests/657 and coupled the photon energy used for division closely to the FEL photon energy. This MR removes photon-energy as a constant condition since it was never used anyway and instead uses it for photonization input, by default again falling back to XGM values for elastic scattering.

Also after discussions with @moellerj we agreed on consistency plots to quickly gauge if photon rounding is applicable:

I'm not particularly proud of the current implementation, but it's what I managed to achieve with both minimal invasion of the existing design and as little performance cost as possible. Since I did not find a clean way to track per-worker reduction memory in the presence of maxtaskperchild, I instead opted to lock a single array across all workers.

Number-wise it decreases performance for the gain correction step in my testing from (5.6 +- 1.93) s to (12.8 +- 1.76) s up to a total of 97.2 s for one set of files (7%). It may be possible to improve this with custom Cython code as opposed to the very generic np.histogram.

How Has This Been Tested?

in_folder = "/gpfs/exfel/exp/MID/202221/p003217/raw" # the folder to read data from, required
sequences = [0] # sequences to correct, set to -1 for all, range allowed
run = 722 # runs to process, required

karabo_id = "MID_DET_AGIPD1M-1" # karabo karabo_id
karabo_id_control = "MID_EXP_AGIPD1M1" # karabo-id for control device

# Correction parameters
blc_noise_threshold = 5000 # above this mean signal intensity now baseline correction via noise is attempted
cm_dark_fraction = 0.15 # threshold for fraction of  empty pixels to consider module enough dark to perform CM correction
cm_dark_range = [-30.,30] # range for signal value ADU for pixel to be consider as a dark pixel
cm_n_itr = 4 # number of iterations for common mode correction
hg_hard_threshold = 1000 # threshold to force medium gain offset subtracted pixel to high gain
mg_hard_threshold = 1000 # threshold to force medium gain offset subtracted pixel from low to medium gain
noisy_adc_threshold = 0.25 # threshold to mask complete adc
ff_gain = 1.0 # conversion gain for absolute FlatField constants, while applying xray_gain
photon_energy = -1.0 # photon energy in keV, non-positive value for XGM autodetection

# Correction Booleans
only_offset = False # Apply only Offset correction. if False, Offset is applied by Default. if True, Offset is only applied.
rel_gain = True # do relative gain correction based on PC data
xray_gain = True # do relative gain correction based on xray data
blc_noise = False # if set, baseline correction via noise peak location is attempted
blc_stripes = True # if set, baseline corrected via stripes
blc_hmatch = False # if set, base line correction via histogram matching is attempted
match_asics = False # if set, inner ASIC borders are matched to the same signal level
adjust_mg_baseline = True # adjust medium gain baseline to match highest high gain value
zero_nans = False # set NaN values in corrected data to 0
zero_orange = False # set to 0 very negative and very large values in corrected data
blc_set_min = True # Shift to 0 negative medium gain pixels after offset corr
corr_asic_diag = False # if set, diagonal drop offs on ASICs are corrected
force_hg_if_below = True # set high gain if mg offset subtracted value is below hg_hard_threshold
force_mg_if_below = True # set medium gain if mg offset subtracted value is below mg_hard_threshold
mask_noisy_adc = False # Mask entire ADC if they are noise above a relative threshold
common_mode = True # Common mode correction
melt_snow = False # Identify (and optionally interpolate) 'snowy' pixels
mask_zero_std = False # Mask pixels with zero standard deviation across train
low_medium_gap = True # 5 sigma separation in thresholding between low and medium gain
round_photons = True  # Round to absolute number of photons, only use with gain corrections

use_xgm_device = 'SA2_XTD1_XGM/XGM/DOOCS'  # DoocsXGM device ID to obtain actual photon energy, operating condition else.

Types of changes

• New feature (non-breaking change which adds functionality)

Reviewers

@ahmedk @kluyvert