[Jungfrau][DARK][CORRECT] FEAT / Use EXtra-data for Jungfrau's dark and correction notebooks

Reading raw data with Extra-data and replacing correction using chunking with pasha.

Description

How Has This Been Tested?

Corrected files were compared with the corrected files from the master using np.allclose(.

xfel-calibrate jungfrau correct \
--out-folder /gpfs/exfel/data/scratch/ahmedk/test/jfCorrect/psh1CorrFile \
--in-folder /gpfs/exfel/exp/SPB/202130/p900204/raw --run 112 \
--karabo-da JNGFR01 JNGFR02 JNGFR03 JNGFR04 JNGFR05 JNGFR06 JNGFR07 JNGFR08 \
--karabo-id SPB_IRDA_JF4M --receiver-id JNGFR{:02d} \
--receiver-control-id CONTROL --sequences-per-node 1

Relevant Documents (optional)

Proposal: 900204, run 112.

Master report: JungfrauOfflineCorrection.pdf

Extra-data + pasha multiple correction sequences(current MR): JungfrauOfflineCorrection.pdf

Extra-data + pasha 1 correction file:JUNGFRAUCORRECT_1CORRCalibration.pdf[Jungfrau_Gain_Correct_and_Verify_NBC_1corr_file.ipynb]

Plotting for 3 different correction versions:

Master (using chunking correction), Extra-data + pasha multiple correction sequences, and Extra-data + pasha 1 correction file.

As can be seen after using pasha and use all ProcessContext with all available CPUs of a node, processing time is lower for both versions compared to master. The slowest part though is saving the data. This is clearly visible and expected for writing to one correction file.

• With Extra-data and pasha, there is a speedup of ~2X compared to the master.

Second iteration of tests.

Dark:

Master: jf_master_dark.pdf

Extra-data + no pyDetLib chunking: jf_ext_dark.pdf

Correct:

Master: jf_ext_correct.pdf

Extra-data + Pasha + balance_sequences: jf_ext_correct.pdf

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code follows the code style of this project.
• Add functionality for copying h5data into out-file.
• Add write_compressed_frames to write mask array for JF correction.
• Remove db-module and rename receiver-id in calibration_configurations repo. https://git.xfel.eu/detectors/calibration_configurations/-/merge_requests/18

Reviewers

@calibration