Skip to content

[LPD] [Correct] Make virtual cxi for each worker's chunk

Thomas Kluyver requested to merge lpd-make-virt-cxi-chunkwise into master

Description

Users for proposal 3073 want to process corrected data via Nexus files per sequence. The first step in building these Nexus files is to create virtual CXI files to align the data from the different LPD modules.

This creates one VCXI file per Slurm job. For short runs (up to 8 sequences), that will also be one per sequence. For longer runs, a few sequences will be grouped together. Doing it strictly per sequence number would be more involved, since the code doesn't group files by sequence number at present, so I hope this is good enough. The CXI files are only created if the create_virtual_cxi_in parameter is set - it should be a path to a folder.

If the sequences for different LPD modules don't align perfectly, some frames will end up split between files.

How Has This Been Tested?

Installed and ran:

xfel-calibrate lpd CORRECT --slurm-mem 700 --request-time 2022-05-17T14:20:21 \
    --slurm-name correct_FXE_lpd_202201_p003073_r14 \
    --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' \
    --karabo-da LPD00 LPD01 LPD02 LPD03 LPD04 LPD05 LPD06 LPD07 LPD08 LPD09 LPD10 LPD11 LPD12 LPD13 LPD14 LPD15 \
    --mem-cells 512 --bias-voltage 250.0 --capacitor 5pf --photon-energy 9.2 --category 0 \
    --offset-corr --rel-gain --ff-map --gain-amp-map \
    --in-folder /gpfs/exfel/exp/FXE/202201/p003073/raw --run 14 \
    --karabo-id FXE_DET_LPD1M-1 \
    --out-folder /gpfs/exfel/data/scratch/kluyvert/lpd-corr-p3073-r14 \
    --create-virtual-cxi-in /gpfs/exfel/data/scratch/kluyvert/lpd-corr-p3073-r14/vcxi \
    --sequences 1

Resulting VCXI file in /gpfs/exfel/data/scratch/kluyvert/lpd-corr-p3073-r14/vcxi

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • My code follows the code style of this project.

Reviewers

@calibration

Merge request reports