[AGIPD] Refactor write_file on AgipdCorrections to create indexes using the DataFile

This refactors write_file of AgipdCorrections in agipdlib.py to create indexes using the DataFile.

Description

This refactor is aimed to create the corrected files independently from the structure and indices of the corresponded raw file. In details, this:

• uses DataFile to create metadata, main index and source indices using DataFile interface;
• creates the new source name for corrected detector data;
• creates link in to the image group in source for corrected data in original detector source, for backward compatibility

The aim:

• to prevent copying of the additional sources and section in XTDF files generated by the new DAQ
• to allow use different source name for corrected data
• to prepare to store additional sources, e.g. number of lit-pixels or peaks

ATTENTION! Breaking change!

It does not copy data from raw files anymore. The others XTDF groups except image are not generating.

How Has This Been Tested?

It was tested by reprocessing run 56 in proposal 7076

• no errors in the report
• pass validation

% extra-data-validate .
Checking run directory: .
Progress: 208/208 [##################################################]
0 problems
No problems found

• test of extra-data test_extra_data.py

% pytest test_extra_data.py
=================================================== test session starts ===================================================
platform linux -- Python 3.8.11, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /gpfs/exfel/u/scratch/SPB/202421/p007076/egor
plugins: subprocess-1.3.2, asyncio-0.16.0, anyio-3.6.2, cov-3.0.0, nbval-0.9.6
collected 1 item

test_extra_data.py .                                                                                                [100%]

=================================================== 1 passed in 45.80s ====================================================

File structure

CORR-R0056-AGIPD00-S00000.h5
├INDEX
│ ├SPB_DET_AGIPD1M-1
│ │ ├CORR
│ │ │ └0CH0:output
│ │ │   └image
│ │ │     ├count        [uint64: 250]
│ │ │     └first        [uint64: 250]
│ │ └DET
│ │   └0CH0:xtdf        -> /INDEX/SPB_DET_AGIPD1M-1/CORR/0CH0:output
│ ├flag [int32: 250]
│ ├origin       [int32: 250]
│ ├timestamp    [uint64: 250]
│ └trainId      [uint64: 250]
├INSTRUMENT
│ └SPB_DET_AGIPD1M-1
│   ├CORR
│   │ └0CH0:output
│   │   └image
│   │     ├blShift      [float32: 87750]
│   │     ├cellId       [uint16: 87750]
│   │     ├data [int16: 87750 × 512 × 128]
│   │     ├gain [uint8: 87750 × 512 × 128]
│   │     ├mask [uint32: 87750 × 512 × 128]
│   │     ├pulseId      [uint64: 87750]
│   │     └trainId      [uint64: 87750]
│   └DET
│     └0CH0:xtdf        -> /INSTRUMENT/SPB_DET_AGIPD1M-1/CORR/0CH0:output
└METADATA
  ├creationDate [ASCII string: 1]
  ├daqLibrary   [ASCII string: 1]
  ├dataFormatVersion    [ASCII string: 1]
  ├dataSources
  │ ├dataSourceId       [ASCII string: 2]
  │ ├deviceId   [ASCII string: 2]
  │ └root       [ASCII string: 2]
  ├karaboFramework      [ASCII string: 1]
  ├proposalNumber       [uint32: 1]
  ├runNumber    [uint32: 1]
  ├sequenceNumber       [uint32: 1]
  └updateDate   [ASCII string: 1]

Source lists in METADATA

CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/dataSourceId
      dtype: ASCII string
      shape: 2
   maxshape: 2
     layout: Contiguous

sample data:
[b'INSTRUMENT/SPB_DET_AGIPD1M-1/CORR/0CH0:output/image' b'INSTRUMENT/SPB_DET_AGIPD1M-1/DET/0CH0:xtdf/image']

0 attributes:

CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/deviceId
      dtype: ASCII string
      shape: 2
   maxshape: 2
     layout: Contiguous

sample data:
[b'SPB_DET_AGIPD1M-1/CORR/0CH0:output/image' b'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf/image']

0 attributes:

CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/root
      dtype: ASCII string
      shape: 2
   maxshape: 2
     layout: Contiguous

sample data:
[b'INSTRUMENT' b'INSTRUMENT']

0 attributes:

Types of changes

• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactor (refactoring code with no functionality changes)

Reviewers

@schmidtp @ahmedk