Feat/agipd add fixed gain mode to dark notebook

Overview

• Some refactoring to make this easier
  • Switch to multiprocessing
  • isort
  • Minor flake8 suggestions (unused imports etc.)
  • Switch default input folder to something from CALLAB
  • Minor formatting
• Detection of gain mode
  • Look at slow data
    • Note use of the enum AgipdGainMode
    • Though within this file, "fixed gain mode" or no is just Boolean
• Toggling some steps based on this
  • Skip OR'ing bad pixels from gain separation if fixed_gain_mode
  • Skip gain switching thresholds if fixed_gain_mode
• Inject appropriate constants into CalCat
  • Use new gain mode @ahmedk added to cal_db_interactive
• Extra parameters for practical reasons
  • Separate thresholds_offset_hard_{hg,mg,lg} for fixed mode
    • Requested by @jsztuk. For infrastructure reasons, changing these sorts of parameters during shifts is a bit of a pain. Including values for fixed gain and adaptive gain mode separately in the calibration configurations seems like the pragmatic solution for now.
  • Manual override of operation mode
    • The user can in MyMDC specify the operation mode, but we don't trust them to and so we currently use slow data to determine the operation mode. @jsztuk has suggested that we additionally allow for an override of this to be used for processing runs where this was not set.

Slow data path used

The operation mode detection code looks something like this:

def get_gain_mode(h5fn):
    h5path_gainmode = f'{h5path_ctrl.replace("/CONTROL/", "/RUN/", 1)}/gainModeIndex/value'
    with h5py.File(h5fn, 'r') as fd:
        if h5path_gainmode in fd:
            return AgipdGainMode(fd[h5path_gainmode][0])
    return AgipdGainMode.ADAPTIVE_GAIN

Hopefully this is consistent - we must assume adaptive gain mode by default, so if we for some reason fail to find this path, things go wrong.

As the notebook is given three runs, it checks that they are indeed using exactly high gain, medium gain, and low gain, respectively. (Future TODO: better warnings, or even stopping execution and shouting.)

Test data

Runs / settings used for testing:

in_folder = "/gpfs/exfel/d/raw/SPB/202130/p900188" # path to input data, required
out_folder = "/gpfs/exfel/data/scratch/hammerd/agipd-fixed-gain" # path to output to, required
sequences = [0] # sequence files to evaluate.
modules = [-1]  # list of modules to evaluate, RANGE ALLOWED
run_high = 80 # run number in which high gain data was recorded, required
run_med = 81 # run number in which medium gain data was recorded, required
run_low = 82 # run number in which low gain data was recorded, required

karabo_id = "SPB_DET_AGIPD1M-1" # karabo karabo_id
karabo_da = ['-1']  # a list of data aggregators names, Default [-1] for selecting all data aggregators
receiver_id = "{}CH0" # inset for receiver devices
path_template = 'RAW-R{:04d}-{}-S{:05d}.h5' # the template to use to access data
h5path = '/INSTRUMENT/{}/DET/{}:xtdf/image' # path in the HDF5 file to images
h5path_idx = '/INDEX/{}/DET/{}:xtdf/image' # path in the HDF5 file to images
h5path_ctrl = '/CONTROL/{}/MDL/FPGA_COMP' # path to control information
karabo_id_control = "SPB_IRU_AGIPD1M1" # karabo-id for control device '
karabo_da_control = "AGIPD1MCTRL00" # karabo DA for control infromation

Planning

CalCat and gain mode condition

@ahmedk has merged https://git.xfel.eu/gitlab/detectors/cal_db_interactive/merge_requests/64 which adds the gain mode condition to cal_db_interactive. We should update the requirement for pycalibration and add gain_mode to the condition used for CalCat interaction.

Thresholds

Some changes need to be made in agipdlib.py to handle fixed gain mode not having gain mode thresholds. Of course, the correction notebook and the online correction pipeline need handling for this, too.

How this has been tested

Output for adaptive runs not changed

With new "default values" in the notebook, I ran a calibration job on one module:

#!/usr/bin/env bash
TIMESTAMP=$(date "+%Y-%m-%d-%H-%M")
MYNAME=$(basename "$0")
xfel-calibrate AGIPD DARK \
    --slurm-name "$MYNAME" \
    --in-folder "/gpfs/exfel/d/raw/CALLAB/202031/p900113" \
    --out-folder "/gpfs/exfel/data/scratch/hammerd/test/agipd-fixed-gain/$MYNAME-$TIMESTAMP-data" \
    --modules 0 \
    --run-high 9985 \
    --run-med 9984 \
    --run-low 9983 \
    --karabo-id "HED_DET_AGIPD500K2G" \
    --karabo-id-control "HED_EXP_AGIPD500K2G" \
    --karabo-da-control "AGIPD500K2G00" \
    --h5path-ctrl "/CONTROL/{}/MDL/FPGA_COMP" \
    --no-db-output

I ran this with the current version and with master of pycalibration. I compared the resulting .h5 files (const_BadPixelsDark..., const_ThresholdsDark_...) using h5diff and they did not differ except for the report paths.

As mentioned in a comment, the bottleneck of characterize_module seems ripe for optimization. See https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/446 for some details.

Fixed gain runs seem to work

Looking at the logs, nothing jumps out at me. Injecting constants into test CalCat seems to work. inject-test.pdf

#!/usr/bin/env bash
TIMESTAMP=$(date "+%Y-%m-%d-%H-%M")
MYNAME=$(basename "$0")
xfel-calibrate AGIPD DARK \
    --slurm-name test-agipd-fixed-gain-mode \
    --in-folder "/gpfs/exfel/d/raw/SPB/202130/p900188" \
    --out-folder "/gpfs/exfel/data/scratch/hammerd/test/agipd-fixed-gain/$MYNAME-$TIMESTAMP-data" \
    --report-to "/gpfs/exfel/data/scratch/hammerd/test/agipd-fixed-gain/$MYNAME-$TIMESTAMP-report" \
    --modules 0 \
    --run-high 80 \
    --run-med 81 \
    --run-low 82 \
    --karabo-id "SPB_DET_AGIPD1M-1" \
    --karabo-id-control "SPB_IRU_AGIPD1M1" \
    --karabo-da-control "AGIPD1MCTRL00"

Reviewers

@danilevc @ahmedk @roscar