Skip to content

[pnCCD] [CORRECT][DARK] EXtra-data and pasha

Karim Ahmed requested to merge feat/Extra_data_pnCCD into master
  1. Add Extra data for reading raw data for CORRECT and DARK notebooks.
  2. Using pasha in parallelizing data processing instead of pyDetLib.
  3. Removing all pyDetlib function, instead for common-mode and plotting.
  4. Use balance_sequences for pnccd correction.
  • Performance wise. The PnCCD gained a lot from these changes. The correct notebook was not doing any parallelization over images or sequence files.

It went from 3 hours for correcting 12 sequences at /gpfs/exfel/exp/SQS/202031/p900166/raw/r0347 to about 7 mins, after using parallelization over sequence files and pasha over images.

Description

How Has This Been Tested?

Correct

Production case comparison (unfair one)

MASTER run correction

Extra-data run correction

Fair comparison of correcting one sequence only.

MASTER one sequences correction

Extra-data one sequences correction

DARK [x5 speed up]

Master dark

Extra-data dark

NOTE: data quality wise. Noise generated constants are the same after specifying relative tolerance parameter to 1e-4

Relevant Documents (optional)

Types of changes

  • New feature (non-breaking change which adds functionality)
  • Refactor (refactoring code with no functionality changes)

Checklist:

Reviewers

@schmidtp @kluyvert

Edited by Karim Ahmed

Merge request reports