Feat/agipd fixed gain correction

Overview

This MR builds upon https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/438. It enables the AGIPD pre-correction constants retrieval notebook and the correction notebook to use the new fixed gain mode.

For constant retrieval, this means:

• detect the gain mode from slow data (reusing code from https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/438)
• use gain mode parameter when asking CalCat for constants
  • for FF and PC constants, the constants are queried without the gain mode parameter for now as requested by @jsztuk
  • don't try to load threshold constants

For correction, this means:

• detect the gain mode from slow data (reusing code from https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/438)
• if fixed gain mode, very explicitly disable some correction bools only relevant for adaptive gain mode

For agipdlib.py, this means:

• add gain_mode attribute to AgipdCorrections
• if in fixed gain mode, skip everything to do with thresholds (init, computation)
  • keep the gain array, but simply set all entries based on the gain mode we're in

I'm setting the MR to target the feat/agipd-add-fixed-gain branch of the previous MR to make diffs a bit more reasonable to look at for now. But still, sorry, the diffs are a bit big. Did minor refactoring of adjacent code.

How has this been tested?

I ran the retrieval notebook with these settings:

in_folder = "/gpfs/exfel/d/raw/SPB/202130/p900188"  # path to input data, required
out_folder = "/gpfs/exfel/data/scratch/hammerd/test/fixed-gain-correction-notebook/from-notebook"  # path to output to, required
sequences = [-1]  # sequences to correct, set to [-1] for all, range allowed
modules = [0, 4, 5, 8]  # modules to correct, set to [-1] for all, range allowed
run = 263 # runs to process, required

karabo_id = "SPB_DET_AGIPD1M-1" # karabo karabo_id
karabo_da = ['-1']  # a list of data aggregators names, Default [-1] for selecting all data aggregators

h5path_ctrl = '/CONTROL/{}/MDL/FPGA_COMP' # path to control information
karabo_id_control = "SPB_IRU_AGIPD1M1" # karabo-id for control device
karabo_da_control = 'AGIPD1MCTRL00' # karabo DA for control infromation

cal_db_interface = "tcp://max-exfl017:8015#8045" # the database interface to use

And the correction notebook with these:

in_folder = "/gpfs/exfel/d/raw/SPB/202130/p900188"  # path to input data, required
out_folder = "/gpfs/exfel/data/scratch/hammerd/test/fixed-gain-correction-notebook/from-notebook"  # path to output to, required
sequences = [0] # sequences to correct, set to -1 for all, range allowed
modules = [0, 5] # modules to correct, set to -1 for all, range allowed
run = 263 # runs to process, required

karabo_id = "SPB_DET_AGIPD1M-1" # karabo karabo_id
karabo_da = ['-1']  # a list of data aggregators names, Default [-1] for selecting all data aggregators
receiver_id = "{}CH0" # inset for receiver devices
path_template = 'RAW-R{:04d}-{}-S{:05d}.h5' # the template to use to access data
h5path = 'INSTRUMENT/{}/DET/{}:xtdf/' # path in the HDF5 file to images
h5path_idx = 'INDEX/{}/DET/{}:xtdf/' # path in the HDF5 file to images
h5path_ctrl = '/CONTROL/{}/MDL/FPGA_COMP' # path to control information
karabo_id_control = "SPB_IRU_AGIPD1M1" # karabo-id for control device
karabo_da_control = 'AGIPD1MCTRL00' # karabo DA for control infromation
cal_db_interface = "tcp://max-exfl017:8015#8045" # the database interface to use

# Correction parameters
blc_noise_threshold = 5000 # above this mean signal intensity now baseline correction via noise is attempted
cm_dark_fraction = 0.66 # threshold for fraction of  empty pixels to consider module enough dark to perform CM correction
cm_dark_range = [-50.,30] # range for signal value ADU for pixel to be consider as a dark pixel
cm_n_itr = 4 # number of iterations for common mode correction
hg_hard_threshold = 1000 # threshold to force medium gain offset subtracted pixel to high gain
mg_hard_threshold = 1000 # threshold to force medium gain offset subtracted pixel from low to medium gain
noisy_adc_threshold = 0.25 # threshold to mask complete adc
ff_gain = 7.2 # conversion gain for absolute FlatField constants, while applying xray_gain

# Correction Booleans
only_offset = False # Apply only Offset correction. if False, Offset is applied by Default. if True, Offset is only applied.
rel_gain = True # do relative gain correction based on PC data
xray_gain = True # do relative gain correction based on xray data
blc_noise = True # if set, baseline correction via noise peak location is attempted
blc_stripes = True # if set, baseline corrected via stripes
blc_hmatch = False # if set, base line correction via histogram matching is attempted, not validated and not in use for current version
match_asics = True # if set, inner ASIC borders are matched to the same signal level, to be reviewed
adjust_mg_baseline = True # adjust medium gain baseline to match highest high gain value
zero_nans = True # set NaN values in corrected data to 0
zero_orange = True # set to 0 very negative and very large values in corrected data
blc_set_min = True # Shift to 0 negative medium gain pixels after offset corr
corr_asic_diag = False # if set, diagonal drop offs on ASICs are correted, to be reviewed
force_hg_if_below = True # set high gain if mg offset subtracted value is below hg_hard_threshold
force_mg_if_below = True # set medium gain if mg offset subtracted value is below mg_hard_threshold
mask_noisy_adc = True # Mask entire ADC if they are noise above a relative threshold, to be reviewed
common_mode = True # Common mode correction
melt_snow = True # Identify (and optionally interpolate) 'snowy' pixels, to be reviewed for adaptive only
mask_zero_std = True # Mask pixels with zero standard deviation across train
low_medium_gap = True # 5 sigma separation in thresholding between low and medium gain

TODO: will update description of testing after running a full calibration job