[AGIPD][DARK] FIX: Make sure to process darks out of sequence file with the highest trainIds

A hacky temporary solution to the bug of processing darks out of one of the available sequences in raw folder.

I though about the fastest fix to deploy without giving any assumptions for a specific sequence number to always process. This was the result.

Processing darks out of the sequences with highest number of trainIds.

Description

• The main problem:

AGIPD dark notebook is designed to parallelize the processing of the 3 gains, based on processing only a file for each gain.

Not realizing this I switched before the value of sequences from [0] to [-1]. This lead to having N files processed in parallel. But still 3 values would only be stored for each gain.

The issue is that the values of the last sequence file could be stored (but not always). The last sequence can have about 30 or 70 trains only compared to about 256 in sequence number 1 S00001.

• The 1st proposed solution:

Is to use S00001 in the first cell. (Currently this is hotfixed in production until we decide with this MR.)

• This MR solution:

Is to not assume always that there will be S00001 available, use [-1] to read all sequences and process only the file with the highest number of images.

• The ultimate solution: Is to not care about any sequences and process all available trains in run folder. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535

How Has This Been Tested?

Relevant Documents (optional)

Tested by running the notebook with the available parameters in the first cell.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

Reviewers

@kluyvert @schmidtp @danilevc

notebooks/AGIPD/Characterize_AGIPD_Gain_Darks_NBC.ipynb

 ``` python
 # set everything up filewise
 os.makedirs(out_folder, exist_ok=True)
-gain_mapped_files, total_files, total_file_size = map_gain_stages(
+gain_mapped_files, _, total_file_size = map_gain_stages(
     in_folder, offset_runs, path_template, karabo_da, sequences
 )
-print(f"Will process a total of {total_files} files ({total_file_size:.02f} GB).")
+# TODO: Keep this commented out to use it later again, this is false information at the moment.
+# print(f"Will process a total of {total_files} files ({total_file_size:.02f} GB).")
 
-inp = []
+# TODO: Remove all of this nonsense with Extra-data.
+inp = [None]*3*len(modules)
 for gain_index, (gain, qm_file_map) in enumerate(gain_mapped_files.items()):
     gain_input = []
     for module_index in modules:
+        max_trainIds = 0
+        inp_idx = gain_index  * len(modules) + module_index
         qm = module_index_to_qm(module_index)
         if qm not in qm_file_map:
             print(f"Did not find files for {qm}")
             # TODO: remove after using EXtra-data to read files
             # and skip empty trains.
             with h5py.File(filename, "r") as fin:
-                if fin[h5path.format(module_index)+"/trainId"].shape[0] != 0:
-                    print(f"Process {filename} for {qm}")
-                    gain_input.append((filename, module_index, gain_index))
-                else:
-                    print(f"Do not process {filename} because it is empty.")
-    if not gain_input:
+                trainIds = fin[h5path.format(module_index)+"/trainId"].shape[0]
+                if trainIds != 0 and trainIds > max_trainIds:
+                    inp[inp_idx] = (filename, module_index, gain_index)
+                    max_trainIds= trainIds
+
+        print(f"Process {inp[inp_idx][0]} for {qm}")
+    if inp[inp_idx] is None:
         raise ValueError(
             "No images to process for run: "
             f"{[v for v in offset_runs.values()][gain_index]}"
         )
-    inp += gain_input
+inp = [x for x in inp if x is not None]
+total_files = len(inp)
 %% Cell type:markdown id: tags: