[LPD][CORRECT] Add parallel gain support
Description
Part 2 of the parallel gain support involving support for corrections. As discussed with FXE, we expect to offer combined images based on Hazem's thresholding approach with sigma levels. Because it was cheap to do, one may also turn off thresholding to obtain all gain images for each frame in corrected form.
I aimed for minimal changes to the correction kernel logic:
- Its outer loop now runs over output frames rather than input frames, whether this are combined or not
- A mapping from output frame indices to the location of each gain image in raw data is built
- Thresholding and combining is performed if this mapping is passed to the kernel
- Slightly changed the logic handling incorrect gain values, which technically causes a data change: if pixels are in unusable gain states, offsets are no longer subtracted
Pulse indices and cell IDs are either corrected beforehand (without combining) or the high gain values used.
Open questions:
- Do we need a manual override? Currently parallel gain is determined from
CONTROL
data, I'm somewhat hesitant as this is not merely a correction flavour, but makes assumptions about input data. - Tuning performance parameters - I'm wondering if the current values for
max_workers
andnum_threads_per_worker
are the right ones, but couldn't reliably test with the heavy GPFS load at the moment
Corrected data without combining
Corrected data with combining
How Has This Been Tested?
in_folder = "/gpfs/exfel/exp/FXE/202405/p006736/raw"
run = 80
Suitable darks have already been injected with the parallel gain support in production.
Types of changes
- New feature (non-breaking change which adds functionality)
Reviewers
FYI @bermudei