Through indoctrination I completely there may be non-elastic scattering in https://git.xfel.eu/detectors/pycalibration/-/merge_requests/657 and coupled the photon energy used for division closely to the FEL photon energy. This MR removes photon-energy as a constant condition since it was never used anyway and instead uses it for photonization input, by default again falling back to XGM values for elastic scattering.
Also after discussions with @moellerj we agreed on consistency plots to quickly gauge if photon rounding is applicable:
I'm not particularly proud of the current implementation, but it's what I managed to achieve with both minimal invasion of the existing design and as little performance cost as possible. Since I did not find a clean way to track per-worker reduction memory in the presence of maxtaskperchild, I instead opted to lock a single array across all workers.
Number-wise it decreases performance for the gain correction step in my testing from (5.6 +- 1.93) s to (12.8 +- 1.76) s up to a total of 97.2 s for one set of files (7%). It may be possible to improve this with custom Cython code as opposed to the very generic np.histogram.
How Has This Been Tested?
in_folder="/gpfs/exfel/exp/MID/202221/p003217/raw"# the folder to read data from, requiredsequences=[0]# sequences to correct, set to -1 for all, range allowedrun=722# runs to process, requiredkarabo_id="MID_DET_AGIPD1M-1"# karabo karabo_idkarabo_id_control="MID_EXP_AGIPD1M1"# karabo-id for control device# Correction parametersblc_noise_threshold=5000# above this mean signal intensity now baseline correction via noise is attemptedcm_dark_fraction=0.15# threshold for fraction of empty pixels to consider module enough dark to perform CM correctioncm_dark_range=[-30.,30]# range for signal value ADU for pixel to be consider as a dark pixelcm_n_itr=4# number of iterations for common mode correctionhg_hard_threshold=1000# threshold to force medium gain offset subtracted pixel to high gainmg_hard_threshold=1000# threshold to force medium gain offset subtracted pixel from low to medium gainnoisy_adc_threshold=0.25# threshold to mask complete adcff_gain=1.0# conversion gain for absolute FlatField constants, while applying xray_gainphoton_energy=-1.0# photon energy in keV, non-positive value for XGM autodetection# Correction Booleansonly_offset=False# Apply only Offset correction. if False, Offset is applied by Default. if True, Offset is only applied.rel_gain=True# do relative gain correction based on PC dataxray_gain=True# do relative gain correction based on xray datablc_noise=False# if set, baseline correction via noise peak location is attemptedblc_stripes=True# if set, baseline corrected via stripesblc_hmatch=False# if set, base line correction via histogram matching is attemptedmatch_asics=False# if set, inner ASIC borders are matched to the same signal leveladjust_mg_baseline=True# adjust medium gain baseline to match highest high gain valuezero_nans=False# set NaN values in corrected data to 0zero_orange=False# set to 0 very negative and very large values in corrected datablc_set_min=True# Shift to 0 negative medium gain pixels after offset corrcorr_asic_diag=False# if set, diagonal drop offs on ASICs are correctedforce_hg_if_below=True# set high gain if mg offset subtracted value is below hg_hard_thresholdforce_mg_if_below=True# set medium gain if mg offset subtracted value is below mg_hard_thresholdmask_noisy_adc=False# Mask entire ADC if they are noise above a relative thresholdcommon_mode=True# Common mode correctionmelt_snow=False# Identify (and optionally interpolate) 'snowy' pixelsmask_zero_std=False# Mask pixels with zero standard deviation across trainlow_medium_gap=True# 5 sigma separation in thresholding between low and medium gainround_photons=True# Round to absolute number of photons, only use with gain correctionsuse_xgm_device='SA2_XTD1_XGM/XGM/DOOCS'# DoocsXGM device ID to obtain actual photon energy, operating condition else.
Types of changes
New feature (non-breaking change which adds functionality)
