By default, all correction notebooks in pycalibration (except for LPD lately) copy all datasets and groups from RAW files to CORR files. Except for the dataset for RAW images. This dataset is overwritten in corrected files with the corrected images.
The number of trains for the corrected data is not the same as RAW data. We exclude trains without data.
As a result, there are misalignments between the corrected data and the trains in the corrected file.
This MR fixes this issue by:
Using the same concept as the new LPD correction notebook.
Not copying all RAW datasets into CORR files.
Store actual correct trainIds in INDEX.
Store METATDATA
Create INSTRUMENT source and store:
data.adc (corrected images)
data.mask (The bad-pixel mask)
data.frameNumber, data.memoryCell, data.gain: same RAW values are stored, but only for the corrected trains.
Updated DataFile.create_metadata() to create METADTAD/dataSources with the ability to be resized. Which is done in the case of adding ROI metadata.
Update Creating INDEX dataset, INSTRUMENT dataset, and CONTROL dataset for ROI.
How Has This Been Tested?
Test with changing roi.
Test with reference runs.
Test with mentioned DOC ticket run.
Relevant Documents (optional)
limit_trains = 0
Current Master results
MR results
limit_trains = 100
Current Master results
MR results
Both screenshots are of using h5glance on JNGFR02 corrected files of sequence 17.
INSTRUMENT source
Number of trains is consistent for memorycell, gain, frameNumber datasets.
I didn't create timestamp and trainId datasets. They are already available in INDEX source.
RUN source
It is not available any more after this MR. (No more copied.)
Types of changes
Bug fix (non-breaking change which fixes an issue)
