[JUNGFRAU][DARK][BURST] Convert to float only within each process to save memory for > 4800 trains

Fixing error with memory allocation for dark processing of adaptive burst mode runs.

http://max-exfl016.desy.de:8008//gpfs/exfel/d/cal/caldb_store/xfel/reports/SPB/SPB_IRDA_JF4M/dark/dark_900258_r46_r47_r48_220303_134704.pdf

Description

@mramilli contacted me regarding dark processing failure for SPB adaptive burst mode dark processing.

For normal operation in adaptive burst mode, medium and low runs are of 300 trains X 16 memory cells.

I am not sure why I didn't face this issue while developing this part, but I assume that these runs were not as big as we normally operate.

For the FXE reference run it had the same issue and it failed in the automated tests but I mistakenly overlooked it as all jungfraus failed due to h5file validation as expected.

To avoid this issue I am basically converting the images type inside the memory cells parallel processes. This leads to using half of the previously used memory.

FXE_XAD_JF1M-DARK-BURST_220303_151642.pdf

How Has This Been Tested?

Test with automated test against for all reference runs at: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/610

Relevant Documents (optional)

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

Reviewers

@calibration

added 1 commit

• 177c196b - convert to float only within each process to save memory

Compare with previous version

changed the description

changed title from convert to float only within each process to save memory to [JUNGFRAU][DARK][BURST] Convert to float only within each process to save memory for > 4800 trains

Looking at the code, I think maybe we don't need the conversion at all - the docs for np.mean() and np.std() say they'll use float64 by default to process integer data. I don't know how that would compare on performance, though.

But in terms of a hotfix, this LGTM!

added 1 commit

• fe3579c6 - convert to float only within each process to save memory

Compare with previous version

Thanks for the fast review!

resolved all threads

enabled an automatic merge when the pipeline for fe3579c6 succeeds

merged

mentioned in commit 22023f3f