[JUNGFRAU][CORRECT] Refactor correction cell and only store corrected datasets

This MR is related to https://git.xfel.eu/calibration/planning/-/issues/135

Description

By default, all correction notebooks in pycalibration (except for LPD lately) copy all datasets and groups from RAW files to CORR files. Except for the dataset for RAW images. This dataset is overwritten in corrected files with the corrected images.

1. The number of trains for the corrected data is not the same as RAW data. We exclude trains without data. As a result, there are misalignments between the corrected data and the trains in the corrected file.

This MR fixes this issue by:

• Using the same concept as the new LPD correction notebook.
• Not copying all RAW datasets into CORR files.
• Store actual correct trainIds in INDEX.
• Store METATDATA
• Create INSTRUMENT source and store:
  • data.adc (corrected images)
  • data.mask (The bad-pixel mask)
  • data.frameNumber, data.memoryCell, data.gain: same RAW values are stored, but only for the corrected trains.

2. Updated DataFile.create_metadata() to create METADTAD/dataSources with the ability to be resized. Which is done in the case of adding ROI metadata.
3. Update Creating INDEX dataset, INSTRUMENT dataset, and CONTROL dataset for ROI.

How Has This Been Tested?

• Test with changing roi.
• Test with reference runs.
• Test with mentioned DOC ticket run.

Relevant Documents (optional)

• limit_trains = 0

Current Master results MR results

• limit_trains = 100

Current Master results MR results

Both screenshots are of using h5glance on JNGFR02 corrected files of sequence 17.

• INSTRUMENT source

  • Number of trains is consistent for memorycell, gain, frameNumber datasets.
  • I didn't create timestamp and trainId datasets. They are already available in INDEX source.

• RUN source

  • It is not available any more after this MR. (No more copied.)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• Style (formatting changes only, no code changes)

Checklist:

• Add screenshot for before and after this MR

Reviewers

@kluyvert @schmidtp

changed the description

changed the description

marked the checklist item Test with changing roi. as completed

marked the checklist item Test with reference runs. as completed

marked the checklist item Test with changing roi. as incomplete

marked the checklist item Test with mentioned DOC ticket run. as completed

added 1 commit

• e65abc8c - Enable extending METADATA/dataSources datasets

Compare with previous version

marked this merge request as ready

changed the description

marked the checklist item Test with changing roi. as completed

marked the checklist item Add screenshot for before and after this MR as completed

changed milestone to %3.8.0

changed title from [JUNGFRAU][CORRECT] Refactorcorrection cell and only store corrected datasets to [JUNGFRAU][CORRECT] Refactor correction cell and only store corrected datasets

changed the description

Nice, thanks. I might also change the ROI saving code to use the DataFile APIs, which should simplify things a bit, but that doesn't need to be done in this MR.

added 1 commit

• 0a64f5a3 - MR comments and update ROI part

Compare with previous version

changed the description

Thanks a lot, we've been way too lucky to get away with this for so long. I don't fully understand the changes to DataFile, but notebook looks really good!

added 1 commit

• ebfd3860 - style change and replace data_path

Compare with previous version

@ahmedk @kluyvert Can we get this PR ready today? We need https://git.xfel.eu/detectors/pycalibration/-/merge_requests/721 in by end of next week, and either MR requires a refactor of the other. I would probably prefer merging this one first.

It would be good to have these 2 MRs as well. (https://git.xfel.eu/detectors/pycalibration/-/merge_requests/742, https://git.xfel.eu/detectors/pycalibration/-/merge_requests/730)

This one is currently hotfixed as FXE couldn't operate without it in burst mode. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/742

This is for the assembled dark images. A lot of the issues arising lately are caught really late because of how hard it is to check the current report. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/730

I don't see any major blocks for those three MRs, only a single open question left. I will rebase the strixel MR tomorrow then.

added 1 commit

• fa42674c - remove METATDATA extension for ROI and timestamp shape (1,)

Compare with previous version

added 1 commit

• 40b927f6 - remove METATDATA extension for ROI and timestamp shape (1,)

Compare with previous version

approved this merge request

changed milestone to %3.7.2

I see this MR has been approved already and I have resolved all discussions. I will be merging in this afternoon if there are no further comments.

Thank you for the review!

merged

mentioned in commit 7e4e0281

Thanks for your patience with all of my changes of direction.

LGTM

It was a pleasure, Thank you!

(That comment was meant for !713 (merged) - I clicked on the wrong email to reply to)

mentioned in merge request !762 (merged)

mentioned in merge request !764 (merged)