Skip to content

[LPD][CORRECT] Add parallel gain support

Philipp Schmidt requested to merge feat/lpd-correct-parallel-gain into master

Description

Part 2 of the parallel gain support involving support for corrections. As discussed with FXE, we expect to offer combined images based on Hazem's thresholding approach with sigma levels. Because it was cheap to do, one may also turn off thresholding to obtain all gain images for each frame in corrected form.

I aimed for minimal changes to the correction kernel logic:

  • Its outer loop now runs over output frames rather than input frames, whether this are combined or not
  • A mapping from output frame indices to the location of each gain image in raw data is built
  • Thresholding and combining is performed if this mapping is passed to the kernel
  • Slightly changed the logic handling incorrect gain values, which technically causes a data change: if pixels are in unusable gain states, offsets are no longer subtracted

Pulse indices and cell IDs are either corrected beforehand (without combining) or the high gain values used.

Open questions:

  • Do we need a manual override? Currently parallel gain is determined from CONTROL data, I'm somewhat hesitant as this is not merely a correction flavour, but makes assumptions about input data.
  • Tuning performance parameters - I'm wondering if the current values for max_workers and num_threads_per_worker are the right ones, but couldn't reliably test with the heavy GPFS load at the moment

Corrected data without combining

image

Corrected data with combining

image

How Has This Been Tested?

in_folder = "/gpfs/exfel/exp/FXE/202405/p006736/raw"
run = 80

Suitable darks have already been injected with the parallel gain support in production.

Types of changes

  • New feature (non-breaking change which adds functionality)

Reviewers

@ahmedk @kluyvert

FYI @bermudei

Merge request reports