[JUNGFRAU][CORRECT] Refactor correction cell and only store corrected datasets
This MR is related to https://git.xfel.eu/calibration/planning/-/issues/135
Description
By default, all correction notebooks in pycalibration (except for LPD lately) copy all datasets and groups from RAW files to CORR files. Except for the dataset for RAW images. This dataset is overwritten in corrected files with the corrected images.
- The number of trains for the corrected data is not the same as RAW data. We exclude trains without data. As a result, there are misalignments between the corrected data and the trains in the corrected file.
This MR fixes this issue by:
- Using the same concept as the new LPD correction notebook.
- Not copying all RAW datasets into CORR files.
- Store actual correct trainIds in INDEX.
- Store METATDATA
- Create INSTRUMENT source and store:
- data.adc (corrected images)
- data.mask (The bad-pixel mask)
- data.frameNumber, data.memoryCell, data.gain: same RAW values are stored, but only for the corrected trains.
- Updated DataFile.create_metadata() to create METADTAD/dataSources with the ability to be resized. Which is done in the case of adding ROI metadata.
- Update Creating INDEX dataset, INSTRUMENT dataset, and CONTROL dataset for ROI.
How Has This Been Tested?
-
Test with changing roi. -
Test with reference runs. -
Test with mentioned DOC ticket run.
Relevant Documents (optional)
limit_trains = 0
Current Master results MR results
limit_trains = 100
Current Master results MR results
Both screenshots are of using h5glance on JNGFR02 corrected files of sequence 17.
-
INSTRUMENT source
- Number of trains is consistent for
memorycell
,gain
,frameNumber
datasets. - I didn't create
timestamp
andtrainId
datasets. They are already available in INDEX source.
- Number of trains is consistent for
-
RUN source
- It is not available any more after this MR. (No more copied.)
Types of changes
- Bug fix (non-breaking change which fixes an issue)
- Style (formatting changes only, no code changes)
Checklist:
-
Add screenshot for before and after this MR
Reviewers
Edited by Karim Ahmed