[Generic][Shimadzu] Dynamic flat-field characterization and correction for MHz microscopy

This implements:

1. The characterization of dark and flat fields of Shimadzu HPVX2 images
2. The correction for Shimadzu HPVX2 images by applying offsets subtraction and dynamic flat-field correction.

Depends on the https://github.com/sarlotabirnsteinova/OnlineVisualization

This is needed for MHz microscopy at SPB

Features:

• Split on sequence files by 200 trains, ignoring original splitting
• Creates new source name SPB_EHD_MIC/CAM/HPVX2_1:daqOutput -> SPB_EHD_MIC/CORR/HPVX2_1:output

Corrected files structure

CORR-R0003-HPVX01-S00000.h5 ├INDEX │ ├SPB_EHD_MIC │ │ └CORR │ │ ├HPVX2_1:output │ │ │ └index_group │ │ │ ├count [uint64: 15] │ │ │ └first [uint64: 15] │ │ └HPVX2_2:output │ │ └index_group │ │ ├count [uint64: 15] │ │ └first [uint64: 15] │ ├flag [int32: 15] │ ├origin [int32: 15] │ ├timestamp [uint64: 15] │ └trainId [uint64: 15] ├INSTRUMENT │ └SPB_EHD_MIC │ └CORR │ ├HPVX2_1:output │ │ └data │ │ └image │ │ ├binning [uint64: 9 × 3] │ │ ├dimTypes [int32: 9 × 3] │ │ ├dims [uint64: 9 × 3] │ │ ├flipX [uint8: 9] │ │ ├flipY [uint8: 9] │ │ ├pixels [float32: 9 × 128 × 250 × 400] │ │ ├roiOffsets [uint64: 9 × 3] │ │ └rotation [int32: 9] │ └HPVX2_2:output │ └data │ └image │ ├binning [uint64: 10 × 3] │ ├dimTypes [int32: 10 × 3] │ ├dims [uint64: 10 × 3] │ ├flipX [uint8: 10] │ ├flipY [uint8: 10] │ ├pixels [float32: 10 × 128 × 250 × 400] │ ├roiOffsets [uint64: 10 × 3] │ └rotation [int32: 10] └METADATA ├creationDate [ASCII string: 1] ├daqLibrary [ASCII string: 1] ├dataFormatVersion [ASCII string: 1] ├dataSources │ ├dataSourceId [ASCII string: 2] │ ├deviceId [ASCII string: 2] │ └root [ASCII string: 2] ├karaboFramework [ASCII string: 1] ├proposalNumber [uint32: 1] ├runNumber [uint32: 1] ├sequenceNumber [uint32: 1] └updateDate [ASCII string: 1]

Reports:

• characterization report: dark_runs_1_2_v0.pdf
• correction report: r0003.pdf

cc @schmidtp

src/cal_tools/shimadzu.py

 from dataclasses import dataclass
+
 from cal_tools.calcat_interface2 import ConditionsBase
 
 
 @dataclass
 class ShimadzuHPVX2Conditions(ConditionsBase):
-    frame_size: float
+    burst_frame_count: float
 
     calibration_types = {
-        "Offset": ["Frame Size"],
-        "DynamicFF": ["Frame Size"],
+        'Offset': ['Burst Frame Count'],
+        'DynamicFF': ['Burst Frame Count'],
     }
 
 
 class ShimadzuHPVX2:
     channel = "daqOutput"
     image_key = "data.image.pixels"
+    copy_keys = [
+        "data.image.binning",
+        "data.image.dimTypes",
+        "data.image.dims",
+        "data.image.flipX",
+        "data.image.flipY",
+        "data.image.roiOffsets",
+        "data.image.rotation",
+    ]
 
     def __init__(self, source_name_pattern: str, channel=None, image_key=None):
         self.source_name_pattern = source_name_pattern
@@ -38,7 +48,7 @@ class ShimadzuHPVX2:
             source_name = self.instrument_source(module)
         keydata = dc[source_name, self.image_key]
         num_frames = keydata.shape[-3]
-        return ShimadzuHPVX2Conditions(frame_size=num_frames / 256)
+        return ShimadzuHPVX2Conditions(burst_frame_count=float(num_frames))
 
     def instrument_source(self, module: int):
         source_name = self.source_name_pattern.format(module)