Skip to content

[JUNGFRAU][CORRECT] Refactor correction cell and only store corrected datasets

Karim Ahmed requested to merge fix/write_consistent_dataset into master

This MR is related to https://git.xfel.eu/calibration/planning/-/issues/135

Description

By default, all correction notebooks in pycalibration (except for LPD lately) copy all datasets and groups from RAW files to CORR files. Except for the dataset for RAW images. This dataset is overwritten in corrected files with the corrected images.

  1. The number of trains for the corrected data is not the same as RAW data. We exclude trains without data. As a result, there are misalignments between the corrected data and the trains in the corrected file.

This MR fixes this issue by:

  • Using the same concept as the new LPD correction notebook.
  • Not copying all RAW datasets into CORR files.
  • Store actual correct trainIds in INDEX.
  • Store METATDATA
  • Create INSTRUMENT source and store:
    • data.adc (corrected images)
    • data.mask (The bad-pixel mask)
    • data.frameNumber, data.memoryCell, data.gain: same RAW values are stored, but only for the corrected trains.
  1. Updated DataFile.create_metadata() to create METADTAD/dataSources with the ability to be resized. Which is done in the case of adding ROI metadata.
  2. Update Creating INDEX dataset, INSTRUMENT dataset, and CONTROL dataset for ROI.

How Has This Been Tested?

  • Test with changing roi.
  • Test with reference runs.
  • Test with mentioned DOC ticket run.

Relevant Documents (optional)

  • limit_trains = 0

Current Master results drawing MR results drawing

  • limit_trains = 100

Current Master results drawing MR results drawing

Both screenshots are of using h5glance on JNGFR02 corrected files of sequence 17.

  • INSTRUMENT source

    • Number of trains is consistent for memorycell, gain, frameNumber datasets.
    • I didn't create timestamp and trainId datasets. They are already available in INDEX source.
  • RUN source

    • It is not available any more after this MR. (No more copied.)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Style (formatting changes only, no code changes)

Checklist:

  • Add screenshot for before and after this MR

Reviewers

@kluyvert @schmidtp

Edited by Karim Ahmed

Merge request reports