[pnCCD] [CORRECT][DARK] EXtra-data and pasha

1. Add Extra data for reading raw data for CORRECT and DARK notebooks.
2. Using pasha in parallelizing data processing instead of pyDetLib.
3. Removing all pyDetlib function, instead for common-mode and plotting.
4. Use balance_sequences for pnccd correction.

• Performance wise. The PnCCD gained a lot from these changes. The correct notebook was not doing any parallelization over images or sequence files.

It went from 3 hours for correcting 12 sequences at /gpfs/exfel/exp/SQS/202031/p900166/raw/r0347 to about 7 mins, after using parallelization over sequence files and pasha over images.

Description

How Has This Been Tested?

Correct

Production case comparison (unfair one)

MASTER run correction

Extra-data run correction

Fair comparison of correcting one sequence only.

MASTER one sequences correction

Extra-data one sequences correction

DARK [x5 speed up]

Master dark

Extra-data dark

NOTE: data quality wise. Noise generated constants are the same after specifying relative tolerance parameter to 1e-4

Relevant Documents (optional)

Types of changes

• New feature (non-breaking change which adds functionality)
• Refactor (refactoring code with no functionality changes)

Checklist:

Reviewers

@schmidtp @kluyvert

added 1 commit

• b599c2d4 - Verify same data and plots output for correction

Compare with previous version

added 1 commit

• 264b4c0e - pnCCD dark notebook

Compare with previous version

added 1 commit

• 2385904e - update map_modules_from_folder

Compare with previous version

added 119 commits

• 2385904e...61f827ad - 118 commits from branch master
• 1e51abe7 - Fix Dark pnccd notebook and create PnccdCtrl class

Compare with previous version

marked this merge request as ready

changed title from WIP: EXtra-data for [pnCCD] [CORRECT] and [DARK] to [pnCCD] [CORRECT][DARK] EXtra-data and pasha

changed the description

assigned to @ahmedk

added 1 commit

• 97585df0 - strip notebook output

Compare with previous version

added 1 commit

• c14b0a73 - flake8 for pnccdlib

Compare with previous version

added 2 commits

• b93555e9 - revert cal_tools.tools changes
• 4b9648bd - use extra-data to find available files to correct

Compare with previous version

resolved all threads

added 1 commit

• 453bf1ac - separate correction processes

Compare with previous version

changed milestone to %3.5.0

added 1 commit

• da91659b - separate correction processes

Compare with previous version