[Generic][Shimadzu] Dynamic flat-field characterization and correction for MHz microscopy

This implements:

1. The characterization of dark and flat fields of Shimadzu HPVX2 images
2. The correction for Shimadzu HPVX2 images by applying offsets subtraction and dynamic flat-field correction.

Depends on the https://github.com/sarlotabirnsteinova/OnlineVisualization

This is needed for MHz microscopy at SPB

Features:

• Split on sequence files by 200 trains, ignoring original splitting
• Creates new source name SPB_EHD_MIC/CAM/HPVX2_1:daqOutput -> SPB_EHD_MIC/CORR/HPVX2_1:output

Corrected files structure

CORR-R0003-HPVX01-S00000.h5 ├INDEX │ ├SPB_EHD_MIC │ │ └CORR │ │ ├HPVX2_1:output │ │ │ └index_group │ │ │ ├count [uint64: 15] │ │ │ └first [uint64: 15] │ │ └HPVX2_2:output │ │ └index_group │ │ ├count [uint64: 15] │ │ └first [uint64: 15] │ ├flag [int32: 15] │ ├origin [int32: 15] │ ├timestamp [uint64: 15] │ └trainId [uint64: 15] ├INSTRUMENT │ └SPB_EHD_MIC │ └CORR │ ├HPVX2_1:output │ │ └data │ │ └image │ │ ├binning [uint64: 9 × 3] │ │ ├dimTypes [int32: 9 × 3] │ │ ├dims [uint64: 9 × 3] │ │ ├flipX [uint8: 9] │ │ ├flipY [uint8: 9] │ │ ├pixels [float32: 9 × 128 × 250 × 400] │ │ ├roiOffsets [uint64: 9 × 3] │ │ └rotation [int32: 9] │ └HPVX2_2:output │ └data │ └image │ ├binning [uint64: 10 × 3] │ ├dimTypes [int32: 10 × 3] │ ├dims [uint64: 10 × 3] │ ├flipX [uint8: 10] │ ├flipY [uint8: 10] │ ├pixels [float32: 10 × 128 × 250 × 400] │ ├roiOffsets [uint64: 10 × 3] │ └rotation [int32: 10] └METADATA ├creationDate [ASCII string: 1] ├daqLibrary [ASCII string: 1] ├dataFormatVersion [ASCII string: 1] ├dataSources │ ├dataSourceId [ASCII string: 2] │ ├deviceId [ASCII string: 2] │ └root [ASCII string: 2] ├karaboFramework [ASCII string: 1] ├proposalNumber [uint32: 1] ├runNumber [uint32: 1] ├sequenceNumber [uint32: 1] └updateDate [ASCII string: 1]

Reports:

• characterization report: dark_runs_1_2_v0.pdf
• correction report: r0003.pdf

cc @schmidtp

When is the next microscopy beamtime?

Let's nail down some technical details...

As I understand, these notebooks implement offset and dynamic FF corrections for the Shimadzu HPVX2.

• Detector

  • Is this detector only used for MHz microscopy? If not, are these corrections relevant in other settings?
  • Which parts of the notebooks are at all specific to the Shimadzu HPVX2? Could they work for any Camera-like Karabo source?
  • I assume this detector will never consist of more than one module (=PDU)? Do you have multiple at all, and if so are they in different Karabo domains?

• Constants & condition

  • ComponentsFF is fairly unspecific in the grand scheme of things. How about DynamicFF?
  • Do you really need Pixels X, Pixels Y in the condition? If you only added it because all legacy 2D detector constants have it, you can get rid of it.
  • Do you really need Memory cells in the condition? It's hardcoded right now, do you expect to ever run it with a different number of cells? Is it relevant for the constant?
  • What meaning is the value of FF components? It seems like a return value from the characterization and not so much a condition the detector was in. Do you expect to select constants based on this value?
  • In any of the above holds true for ComponentsFF / DynamicFF, does it also hold true for Offset? Different calibrations for the same detector do not need to share conditions for no reason.

Just noting my understanding of having two different types of constants generated from the same notebook.

Usually, we have a notebook per calibration action. Also, a notebook works to generate data from the same run type.

From what I understand in combining both in a characterization notebook, is that it is necessary to have both corrections (dark and flat) in the transmission imaging correction and it's not possible to go with only dark for example. I hope this is correct otherwise I don't see a benefit other than running it now from myMDC in one process.

added 339 commits

• 283a2237...4786d0dc - 334 commits from branch master
• e0f12cb9 - Add dark and flat field characterization notebook for Shimadzu HPVX2
• 513376f4 - Add correction notebook for Shimadzu HPVX2 camera
• 035e0ffa - Rename notebooks
• 91e05cb3 - Support multiple module detector in Characterize_DynamicFF_NBC.ipynb
• 7daba04f - Support multiple module detector in Correct_DynamicFF_NBC.ipynb

Compare with previous version

added 1 commit

• ffcdffb9 - Change name of dynamic flat-field correction package

Compare with previous version

Documenting what we did in CalCat (only test for now):

• Created new detector type ShimadzuHPVX2-Type
• Created new operation mode Transmission Imaging
  • Requires two runs
  • Available for ShimadzuHPVX2-Type
• Created new detector SPB_MIC_HPVX2
  • ShimadzuHPVX2-Type and available at SPB
  • Source name pattern SPB_EHD_MIC/CAM/HPVX2_{} with 5 modules starting at 1
• Created new PDU Shimadzu_HPVX2_I778061F0213
  • SPB_MIC_HPVS2 at module number 1 bound to HPVX01/1
  • Uses property SensorSerial from Karabo device
• Created new PDU Shimadzu_HPVX2_I778055B0076
  • SPB_MIC_HPVS2 at module number 2 bound to HPVX01/2

added 2 commits

• bba593d9 - Add ShimadzuHVX2 conditions and detector abstraction
• 769cbbb5 - Query PDUs, use Shimadzu abstraction, add dummy ccv injection

Compare with previous version

added 2 commits

• 0bdffdb4 - Add DynamicFF notebooks to xfel_calibrate
• 00102c92 - Fix parameters and inject_cvv signature according to the webserver call

Compare with previous version

mentioned in merge request !993 (merged)

added 3 commits

• 6b200f08 - Update ShimadzuHPVX2 conditions, add copy_keys in ShimadzuHPVX2 detector
• a5ffccc1 - Fix dark data for flat-field part, add checks of data
• 9d897ede - Add retrieving calibraion constants from DB in Correct_DynamicFF_NBC.ipynb

Compare with previous version

changed the description

added 2 commits

• 93260448 - Add retrieving calibraion constants from DB in Correct_DynamicFF_NBC.ipynb
• 3a8e6141 - Add dynflatfield package in dependencies

Compare with previous version

marked this merge request as ready

added Waiting for review label

changed title from Shimadzu characterization and correction for MHz microscopy to [Generic][Shimadzu] Dynamic flat-field characterization and correction for MHz microscopy

I'm taking over this MR in Egor's absence and will deliver the true inject_ccv via its own branch against this.

Please review the notebooks apart from this @ahmedk @kluyvert