This refactors write_file of AgipdCorrections in agipdlib.py to create indexes using the DataFile.
This refactor is aimed to create the corrected files independently from the structure and indices of the corresponded raw file. In details, this:
DataFile to create metadata, main index and source indices using DataFile interface;image group in source for corrected data in original detector source, for backward compatibilityThe aim:
ATTENTION! Breaking change!
It does not copy data from raw files anymore. The others XTDF groups except image are not generating.
It was tested by reprocessing run 56 in proposal 7076
% extra-data-validate .
Checking run directory: .
Progress: 208/208 [##################################################]
0 problems
No problems foundextra-data test_extra_data.py
% pytest test_extra_data.py
=================================================== test session starts ===================================================
platform linux -- Python 3.8.11, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /gpfs/exfel/u/scratch/SPB/202421/p007076/egor
plugins: subprocess-1.3.2, asyncio-0.16.0, anyio-3.6.2, cov-3.0.0, nbval-0.9.6
collected 1 item
test_extra_data.py . [100%]
=================================================== 1 passed in 45.80s ====================================================CORR-R0056-AGIPD00-S00000.h5
├INDEX
│ ├SPB_DET_AGIPD1M-1
│ │ ├CORR
│ │ │ └0CH0:output
│ │ │ └image
│ │ │ ├count [uint64: 250]
│ │ │ └first [uint64: 250]
│ │ └DET
│ │ └0CH0:xtdf -> /INDEX/SPB_DET_AGIPD1M-1/CORR/0CH0:output
│ ├flag [int32: 250]
│ ├origin [int32: 250]
│ ├timestamp [uint64: 250]
│ └trainId [uint64: 250]
├INSTRUMENT
│ └SPB_DET_AGIPD1M-1
│ ├CORR
│ │ └0CH0:output
│ │ └image
│ │ ├blShift [float32: 87750]
│ │ ├cellId [uint16: 87750]
│ │ ├data [int16: 87750 × 512 × 128]
│ │ ├gain [uint8: 87750 × 512 × 128]
│ │ ├mask [uint32: 87750 × 512 × 128]
│ │ ├pulseId [uint64: 87750]
│ │ └trainId [uint64: 87750]
│ └DET
│ └0CH0:xtdf -> /INSTRUMENT/SPB_DET_AGIPD1M-1/CORR/0CH0:output
└METADATA
├creationDate [ASCII string: 1]
├daqLibrary [ASCII string: 1]
├dataFormatVersion [ASCII string: 1]
├dataSources
│ ├dataSourceId [ASCII string: 2]
│ ├deviceId [ASCII string: 2]
│ └root [ASCII string: 2]
├karaboFramework [ASCII string: 1]
├proposalNumber [uint32: 1]
├runNumber [uint32: 1]
├sequenceNumber [uint32: 1]
└updateDate [ASCII string: 1]CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/dataSourceId
dtype: ASCII string
shape: 2
maxshape: 2
layout: Contiguous
sample data:
[b'INSTRUMENT/SPB_DET_AGIPD1M-1/CORR/0CH0:output/image' b'INSTRUMENT/SPB_DET_AGIPD1M-1/DET/0CH0:xtdf/image']
0 attributes:
CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/deviceId
dtype: ASCII string
shape: 2
maxshape: 2
layout: Contiguous
sample data:
[b'SPB_DET_AGIPD1M-1/CORR/0CH0:output/image' b'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf/image']
0 attributes:
CORR-R0056-AGIPD00-S00000.h5/METADATA/dataSources/root
dtype: ASCII string
shape: 2
maxshape: 2
layout: Contiguous
sample data:
[b'INSTRUMENT' b'INSTRUMENT']
0 attributes: