[Generic][Shimadzu] Dynamic flat-field characterization and correction for MHz microscopy

This implements:

1. The characterization of dark and flat fields of Shimadzu HPVX2 images
2. The correction for Shimadzu HPVX2 images by applying offsets subtraction and dynamic flat-field correction.

Depends on the https://github.com/sarlotabirnsteinova/OnlineVisualization

This is needed for MHz microscopy at SPB

Features:

• Split on sequence files by 200 trains, ignoring original splitting
• Creates new source name SPB_EHD_MIC/CAM/HPVX2_1:daqOutput -> SPB_EHD_MIC/CORR/HPVX2_1:output

Corrected files structure

CORR-R0003-HPVX01-S00000.h5 ├INDEX │ ├SPB_EHD_MIC │ │ └CORR │ │ ├HPVX2_1:output │ │ │ └index_group │ │ │ ├count [uint64: 15] │ │ │ └first [uint64: 15] │ │ └HPVX2_2:output │ │ └index_group │ │ ├count [uint64: 15] │ │ └first [uint64: 15] │ ├flag [int32: 15] │ ├origin [int32: 15] │ ├timestamp [uint64: 15] │ └trainId [uint64: 15] ├INSTRUMENT │ └SPB_EHD_MIC │ └CORR │ ├HPVX2_1:output │ │ └data │ │ └image │ │ ├binning [uint64: 9 × 3] │ │ ├dimTypes [int32: 9 × 3] │ │ ├dims [uint64: 9 × 3] │ │ ├flipX [uint8: 9] │ │ ├flipY [uint8: 9] │ │ ├pixels [float32: 9 × 128 × 250 × 400] │ │ ├roiOffsets [uint64: 9 × 3] │ │ └rotation [int32: 9] │ └HPVX2_2:output │ └data │ └image │ ├binning [uint64: 10 × 3] │ ├dimTypes [int32: 10 × 3] │ ├dims [uint64: 10 × 3] │ ├flipX [uint8: 10] │ ├flipY [uint8: 10] │ ├pixels [float32: 10 × 128 × 250 × 400] │ ├roiOffsets [uint64: 10 × 3] │ └rotation [int32: 10] └METADATA ├creationDate [ASCII string: 1] ├daqLibrary [ASCII string: 1] ├dataFormatVersion [ASCII string: 1] ├dataSources │ ├dataSourceId [ASCII string: 2] │ ├deviceId [ASCII string: 2] │ └root [ASCII string: 2] ├karaboFramework [ASCII string: 1] ├proposalNumber [uint32: 1] ├runNumber [uint32: 1] ├sequenceNumber [uint32: 1] └updateDate [ASCII string: 1]

Reports:

• characterization report: dark_runs_1_2_v0.pdf
• correction report: r0003.pdf

cc @schmidtp

src/cal_tools/shimadzu.py

@@ -4,7 +4,6 @@ from cal_tools.calcat_interface2 import ShimadzuHPVX2Conditions
 
 
 class ShimadzuHPVX2:
-    channel = "daqOutput"
     image_key = "data.image.pixels"
     copy_keys = [
         "data.image.binning",
@@ -27,8 +26,7 @@ class ShimadzuHPVX2:
 
     def conditions(self, dc: "DataCollection", module=None):  # noqa: F821
         if module is None:
-            source_pattern = self.source_name_pattern.format(
-                f"*:{self.channel}")
+            source_pattern = self.source_name_pattern.format(modno='*')
             det_dc = dc.select(source_pattern)
             if not det_dc.instrument_sources:
                 raise ValueError("No detector sources are found")
@@ -41,12 +39,17 @@ class ShimadzuHPVX2:
         return ShimadzuHPVX2Conditions(burst_frame_count=float(num_frames))
 
     def instrument_source(self, module: int):
-        source_name = self.source_name_pattern.format(module)
-        return f"{source_name}:{self.channel}"
+        return self.source_name_pattern.format(modno=module)
 
     def corrected_source(self, module: int):
-        source_name = self.source_name_pattern.format(module)
+        source_name = self.source_name_pattern.format(modno=module)
+
+        # Replace type with CORR.
         parts = source_name.split('/')
         parts[1] = "CORR"
         source_name = '/'.join(parts)
-        return f"{source_name}:output"
+
+        # Replace channel with output.
+        source_name = source_name[:source_name.index(':')] + ':output'
+
+        return source_name