[LPD] [Correct] Exclude trains with <4 modules from LPD VCXI files

Description

One LPD module (module 11) is generating some bogus train IDs, and its sequence file boundaries get out of sync with the other modules. When we create VCXI files 'by sequence', this means that the first train or two only has data from module 11 (and the last train or two are missing data from module 11). Most of the file is still OK, but users looking at the first 10 images in every file see missing data for 15 of 16 modules and understandably think it's broken. This change excludes trains with <4 modules (an arbitrary number) from the VCXI files, which should hide this issue.

Where this happens, there will still be some images with module 11 missing, because it's in the 'wrong' sequence file. Sequence files are not guaranteed to be aligned between different modules, so the whole concept of making a VCXI file for a batch of sequence files is kind of flawed anyway.

How Has This Been Tested?

Ran:

xfel-calibrate lpd CORRECT --slurm-mem 700 \
    --slurm-name correct_FXE_lpd_202201_p003073_r91 \
    --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' \
    --karabo-da LPD00 LPD01 LPD02 LPD03 LPD04 LPD05 LPD06 LPD07 LPD08 LPD09 LPD10 LPD11 LPD12 LPD13 LPD14 LPD15 \
    --mem-cells 512 --bias-voltage 250.0 --capacitor 5pf --photon-energy 9.2 --category 0 \
    --offset-corr --rel-gain --ff-map --gain-amp-map \
    --in-folder /gpfs/exfel/exp/FXE/202201/p003073/raw --run 91 \
    --karabo-id FXE_DET_LPD1M-1 \
    --out-folder /gpfs/exfel/data/scratch/kluyvert/lpd-corr-p3073-r91-b \
    --create-virtual-cxi-in /gpfs/exfel/data/scratch/kluyvert/lpd-corr-p3073-r91-b/vcxi

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the code style of this project.

Reviewers

@dallanto @schmidtp

It appears this can cause an exception to be raised if there are no appropriate trains in any sequence. This would mark the job as failed, is this intended?

added 1 commit

• cf4de1f3 - Exclude trains with flag=0 from output file for LPD

Compare with previous version

No, that's not really intended, and it could come up if the 'extra' trains for one module caused it to have one more sequence file than the others. I think the best option there is to catch the exception.

I also just excluded 'suspect' train IDs from processing, so the bogus train IDs we see from LPD are discarded entirely. Without excluding these, I'm not sure whether the file reading & writing code currently do the right thing with out-of-order trains - My best guess at present is that in this scenario they do do something sensible, but it's not something to rely on..

added 1 commit

• c83fc0a4 - Catch exception from creating LPD1M object

Compare with previous version

changed the description

I will be deploying this now on production. So, LGTM?

merged

mentioned in commit 9ddc981d

changed milestone to %3.5.4