[pnCCD] [CORRECT][DARK] EXtra-data and pasha

1. Add Extra data for reading raw data for CORRECT and DARK notebooks.
2. Using pasha in parallelizing data processing instead of pyDetLib.
3. Removing all pyDetlib function, instead for common-mode and plotting.
4. Use balance_sequences for pnccd correction.

• Performance wise. The PnCCD gained a lot from these changes. The correct notebook was not doing any parallelization over images or sequence files.

It went from 3 hours for correcting 12 sequences at /gpfs/exfel/exp/SQS/202031/p900166/raw/r0347 to about 7 mins, after using parallelization over sequence files and pasha over images.

Description

How Has This Been Tested?

Correct

Production case comparison (unfair one)

MASTER run correction

Extra-data run correction

Fair comparison of correcting one sequence only.

MASTER one sequences correction

Extra-data one sequences correction

DARK [x5 speed up]

Master dark

Extra-data dark

NOTE: data quality wise. Noise generated constants are the same after specifying relative tolerance parameter to 1e-4

Relevant Documents (optional)

Types of changes

• New feature (non-breaking change which adds functionality)
• Refactor (refactoring code with no functionality changes)

Checklist:

Reviewers

@schmidtp @kluyvert

src/cal_tools/tools.py

@@ -57,7 +57,7 @@ def run_prop_seq_from_path(filename):
 
 
 def map_modules_from_folder(in_folder, run, path_template, karabo_da,
-                            sequences=None, qm_naming=True):
+                            sequences=None):
     """
     Prepare queues of files to process.
     Queues are stored in dictionary with module name Q{}M{} as a key
@@ -67,9 +67,7 @@ def map_modules_from_folder(in_folder, run, path_template, karabo_da,
     :param path_template: Template for file name
                           e.g. `RAW-R{:04d}-{}-S{:05d}.h5`
     :param karabo_da: List of data aggregators e.g. [AGIPD00, AGIPD01]
-    :param sequences: List of sequences to be considered.
-    :param qm_naming: Flag to use Q{}M{} naming convention for dict keys,
-                      instead of the karabo-da.
+    :param sequences: List of sequences to be considered
     :return: Dictionary of queues of files, dictionary of module indexes,
     total number of sequences, dictionary of number of sequences per module
     """
@@ -80,10 +78,7 @@ def map_modules_from_folder(in_folder, run, path_template, karabo_da,
     sequences_qm = {}
     for inset in karabo_da:
         module_idx = int(inset[-2:])
-        if qm_naming:
-            name = module_index_to_qm(module_idx)
-        else:
-            name = inset
+        name = module_index_to_qm(module_idx)
         module_files[name] = Queue()
         sequences_qm[name] = 0
         mod_ids[name] = module_idx