[AGIPD] Feat: SPI hit-finder

Description

This implements the SPI hit finding using the method implemented in dataAnalysis/data-reduction!21 (merged). The method splits images into two classes (hits and misses). Hits are images where the number of lit-pixels exceeds the threshold. The threshold can be set as fixed or calculated by analyzing the distribution of the number of lit-pixels in neighboring images.

Additionally, this selects the images for the future analysis, including all hits and random fraction of misses.

This writes the results in a EXDF file. They can be used in the future analysis or for data reduction.

This is needed as a part of automation the data analysis and data reduction.

This depends on !986 (merged) and dataAnalysis/data-reduction!21 (merged)

The main hit-finder algorithm is in extra-redu (dataAnalysis/data-reduction!21 (merged))

How Has This Been Tested?

It was tested by processing run 56 in proposal 7076

% extra-data-validate .          
Checking run directory: .
Progress: 209/209 [##################################################]
0 problems
No problems found

The file structure

CORR-R0056-REDU00-S00000.h5
├CONTROL
│ └SPB_DET_AGIPD1M-1
│   └REDU
│     └SPI_HITFINDER
│       ├fixedThreshold
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int64: 3010]
│       ├hitrateWindowSize
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int64: 3010]
│       ├minScores
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int64: 3010]
│       ├missFraction
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int64: 3010]
│       ├missFractionBase
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [4-byte ASCII string: 3010]
│       ├mode
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [8-byte ASCII string: 3010]
│       ├modules
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int8: 3010 × 16]
│       ├numModules
│       │ ├timestamp      [uint64: 3010]
│       │ └value  [int64: 3010]
│       └snr
│         ├timestamp      [uint64: 3010]
│         └value  [float64: 3010]
├INDEX
│ ├SPB_DET_AGIPD1M-1
│ │ └REDU
│ │   ├SPI_HITFINDER
│ │   │ ├count    [uint64: 3010]
│ │   │ └first    [uint64: 3010]
│ │   └SPI_HITFINDER:output
│ │     ├data
│ │     │ ├count  [uint64: 3010]
│ │     │ └first  [uint64: 3010]
│ │     └threshold
│ │       ├count  [uint64: 3010]
│ │       └first  [uint64: 3010]
│ ├flag   [int32: 3010]
│ ├origin [int32: 3010]
│ ├timestamp      [uint64: 3010]
│ └trainId        [uint64: 3010]
├INSTRUMENT
│ └SPB_DET_AGIPD1M-1
│   └REDU
│     └SPI_HITFINDER:output
│       ├data
│       │ ├dataFlag       [uint8: 1056510]
│       │ ├hitFlag        [uint8: 1056510]
│       │ ├hitscore       [int64: 1056510]
│       │ ├missFlag       [uint8: 1056510]
│       │ ├pulseId        [uint64: 1056510]
│       │ └trainId        [uint64: 1056510]
│       └threshold
│         ├mu     [float64: 3010]
│         ├sig    [float64: 3010]
│         └value  [float64: 3010]
├METADATA
│ ├creationDate   [ASCII string: 1]
│ ├daqLibrary     [ASCII string: 1]
│ ├dataFormatVersion      [ASCII string: 1]
│ ├dataSources
│ │ ├dataSourceId [ASCII string: 3]
│ │ ├deviceId     [ASCII string: 3]
│ │ └root [ASCII string: 3]
│ ├karaboFramework        [ASCII string: 1]
│ ├proposalNumber [uint32: 1]
│ ├runNumber      [uint32: 1]
│ ├sequenceNumber [uint32: 1]
│ └updateDate     [ASCII string: 1]
└RUN
  └SPB_DET_AGIPD1M-1
    └REDU
      └SPI_HITFINDER
        ├fixedThreshold
        │ ├timestamp      [uint64: 1]
        │ └value  [int64: 1]
        ├hitrateWindowSize
        │ ├timestamp      [uint64: 1]
        │ └value  [int64: 1]
        ├minScores
        │ ├timestamp      [uint64: 1]
        │ └value  [int64: 1]
        ├missFraction
        │ ├timestamp      [uint64: 1]
        │ └value  [int64: 1]
        ├missFractionBase
        │ ├timestamp      [uint64: 1]
        │ └value  [4-byte ASCII string: 1]
        ├mode
        │ ├timestamp      [uint64: 1]
        │ └value  [8-byte ASCII string: 1]
        ├modules
        │ ├timestamp      [uint64: 1]
        │ └value  [int8: 1 × 16]
        ├numModules
        │ ├timestamp      [uint64: 1]
        │ └value  [int64: 1]
        └snr
          ├timestamp      [uint64: 1]
          └value  [float64: 1]

Relevant Documents (optional)

Calibration report SPB_DET_AGIPD1M-1_correct_007076_r56_240802_111313.pdf

Types of changes

• New feature (non-breaking change which adds functionality)

Reviewers

@schmidtp @ahmedk