[Generic][Shimadzu] Dynamic flat-field characterization and correction for MHz microscopy

This implements:

1. The characterization of dark and flat fields of Shimadzu HPVX2 images
2. The correction for Shimadzu HPVX2 images by applying offsets subtraction and dynamic flat-field correction.

Depends on the https://github.com/sarlotabirnsteinova/OnlineVisualization

This is needed for MHz microscopy at SPB

Features:

• Split on sequence files by 200 trains, ignoring original splitting
• Creates new source name SPB_EHD_MIC/CAM/HPVX2_1:daqOutput -> SPB_EHD_MIC/CORR/HPVX2_1:output

Corrected files structure

CORR-R0003-HPVX01-S00000.h5 ├INDEX │ ├SPB_EHD_MIC │ │ └CORR │ │ ├HPVX2_1:output │ │ │ └index_group │ │ │ ├count [uint64: 15] │ │ │ └first [uint64: 15] │ │ └HPVX2_2:output │ │ └index_group │ │ ├count [uint64: 15] │ │ └first [uint64: 15] │ ├flag [int32: 15] │ ├origin [int32: 15] │ ├timestamp [uint64: 15] │ └trainId [uint64: 15] ├INSTRUMENT │ └SPB_EHD_MIC │ └CORR │ ├HPVX2_1:output │ │ └data │ │ └image │ │ ├binning [uint64: 9 × 3] │ │ ├dimTypes [int32: 9 × 3] │ │ ├dims [uint64: 9 × 3] │ │ ├flipX [uint8: 9] │ │ ├flipY [uint8: 9] │ │ ├pixels [float32: 9 × 128 × 250 × 400] │ │ ├roiOffsets [uint64: 9 × 3] │ │ └rotation [int32: 9] │ └HPVX2_2:output │ └data │ └image │ ├binning [uint64: 10 × 3] │ ├dimTypes [int32: 10 × 3] │ ├dims [uint64: 10 × 3] │ ├flipX [uint8: 10] │ ├flipY [uint8: 10] │ ├pixels [float32: 10 × 128 × 250 × 400] │ ├roiOffsets [uint64: 10 × 3] │ └rotation [int32: 10] └METADATA ├creationDate [ASCII string: 1] ├daqLibrary [ASCII string: 1] ├dataFormatVersion [ASCII string: 1] ├dataSources │ ├dataSourceId [ASCII string: 2] │ ├deviceId [ASCII string: 2] │ └root [ASCII string: 2] ├karaboFramework [ASCII string: 1] ├proposalNumber [uint32: 1] ├runNumber [uint32: 1] ├sequenceNumber [uint32: 1] └updateDate [ASCII string: 1]

Reports:

• characterization report: dark_runs_1_2_v0.pdf
• correction report: r0003.pdf

cc @schmidtp

src/cal_tools/shimadzu.py

+from dataclasses import dataclass
+from cal_tools.calcat_interface2 import ConditionsBase
+
+
+@dataclass
+class ShimadzuHPVX2Conditions(ConditionsBase):
+    frame_size: float
+
+    calibration_types = {
+        "Offset": ["Frame Size"],
+        "DynamicFF": ["Frame Size"],
+    }
+
+
+class ShimadzuHPVX2:
+    channel = "daqOutput"
+    image_key = "data.image.pixels"
+
+    def __init__(self, source_name_pattern: str, channel=None, image_key=None):
+        self.source_name_pattern = source_name_pattern
+        if channel is not None:
+            self.channel = channel
+        if image_key is not None:
+            self.image_key = image_key
+        self.image_index_group = self.image_key.partition('.')[0]
+        self.instrument = source_name_pattern.split('_')[0]
+
+    def conditions(self, dc: "DataCollection", module=None):  # noqa: F821
+        if module is None:
+            source_pattern = self.source_name_pattern.format(
+                f"*:{self.channel}")
+            det_dc = dc.select(source_pattern)
+            if not det_dc.instrument_sources:
+                raise ValueError("No detector sources are found")
+
+            source_name = list(det_dc.instrument_sources)[0]
+        else:
+            source_name = self.instrument_source(module)
+        keydata = dc[source_name, self.image_key]
+        num_frames = keydata.shape[-3]
+        return ShimadzuHPVX2Conditions(frame_size=num_frames / 256)
+
+    def instrument_source(self, module: int):
+        source_name = self.source_name_pattern.format(module)
+        return f"{source_name}:{self.channel}"
+
+    def corrected_source(self, module: int):
+        source_name = self.source_name_pattern.format(module)
+        parts = source_name.split('/')
+        parts[1] = "CORR"
+        source_name = '/'.join(parts)
+        return f"{source_name}:output"