[AGIPD][LFF] Improve lit frame selection

This improves lit frame selection, including:

1. fix the misalignment of selection by trains
2. introduce super pattern selection
3. guess missed patterns for patterns which repeat with constant step
4. shows patterns which repeat with constant step as one line in the report

Description

The code which transforms the LitFrameFinder data into selection is moved to EXtra-redu package. This code actually identifies pattern repetition with constant step, fills the gaps in LFF data, extracts the selection for requested list of trains and so on.

The integration with the EXtra-redu is left here.

The misalignment of selection by trains is a bug, which discovered recently. It happens when some trains missed in the LFF data. Current code doesn't generate the any selection for missed trains and all consequent selection applies with a shift by the number of missed trains. It spoils processing if the bunch pattern varies during the run.

Super pattern selection allows to apply selection which is an union of all patterns in the run. It solves the issue with common mode correction in the case of varying pattern. It's also requested by MID for rolling bunch pattern.

Guess of missed patterns allows to fill the gaps in LFF data and prevent the drop of good detector trains.

Report improvement allows to present the pattern list in more readable way and prevent flooding (but not for every possible case)

How Has This Been Tested?

The changes were tested using a few runs from the last proposal (3025) with fixed bunch pattern and a few runs from the proposal (3118) with rolling bunch pattern.

See some reports: check_r0130.pdf check_r0140.pdf

The check of trains alignment (with disabled filling of the gaps in LFF data): p3025_r72_notrain_guess

Reviewers

@schmidtp @ahmedk

added 1 commit

• b907f45f - Merge branch 'master' into 'feat/litframe-common-pattern'.

Compare with previous version

changed the description

The misalignment of selection by trains is a bug

Is there a specific run that helped you realize this bug? Was this reported somewhere?

Are there any runs and configs that we should include in our tests? same for this MR https://git.xfel.eu/detectors/pycalibration/-/merge_requests/724 as @schmidtp already mentioned.

changed milestone to %3.8.0

added Waiting for review label

changed milestone to %3.7.5

mentioned in commit 262b51de

added 1 commit

• 262b51de - Apply review suggestions from !766 (merged)

Compare with previous version

resolved all threads

LGTM

LGTM.

changed milestone to %3.7.6

mentioned in commit 8a0e7cdc

added 11 commits

• 262b51de...11aed53d - 9 commits from branch master
• a12c73ce - Improve lit frame finder selection
• 8a0e7cdc - Apply review suggestions from !766 (merged)

Compare with previous version

approved this merge request

enabled an automatic merge when the pipeline for 8a0e7cdc succeeds

merged

mentioned in commit 4203e5f9