[AGIPD][DARK] FIX: Make sure to process darks out of sequence file with the highest trainIds

A hacky temporary solution to the bug of processing darks out of one of the available sequences in raw folder.

I though about the fastest fix to deploy without giving any assumptions for a specific sequence number to always process. This was the result.

Processing darks out of the sequences with highest number of trainIds.

Description

• The main problem:

AGIPD dark notebook is designed to parallelize the processing of the 3 gains, based on processing only a file for each gain.

Not realizing this I switched before the value of sequences from [0] to [-1]. This lead to having N files processed in parallel. But still 3 values would only be stored for each gain.

The issue is that the values of the last sequence file could be stored (but not always). The last sequence can have about 30 or 70 trains only compared to about 256 in sequence number 1 S00001.

• The 1st proposed solution:

Is to use S00001 in the first cell. (Currently this is hotfixed in production until we decide with this MR.)

• This MR solution:

Is to not assume always that there will be S00001 available, use [-1] to read all sequences and process only the file with the highest number of images.

• The ultimate solution: Is to not care about any sequences and process all available trains in run folder. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535

How Has This Been Tested?

Relevant Documents (optional)

Tested by running the notebook with the available parameters in the first cell.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

Reviewers

@kluyvert @schmidtp @danilevc

changed the description

added 1 commit

• 6fb83ea6 - [AGIPD][DARK] FIX: Make sure to process darks out of sequence file with the highest trainIds

Compare with previous version

added 1 commit

• 48d28055 - [AGIPD][DARK] FIX: Make sure to process darks out of sequence file with the highest trainIds

Compare with previous version

This establishes the maximum trains per sequence and filters empty sequences from what I understand. It seems like a solution for this issue indeed. Still, could we talk about it after the stand-up?

resolved all threads

resolved all threads

changed the description

added Waiting for review label

Without having looked into https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535 much yet, I'll assume it's already fixed there?

changed milestone to %3.5.0

resolved all threads

resolved all threads

mentioned in commit aee1ae3e

merged

removed Waiting for review label

mentioned in merge request !616 (merged)