Improvements to darks summary notebook
Description
FXE pointed out during the bring-up day that when they process LPD darks, the main processing notebooks take about 2 minutes each, but then the summary notebook takes about 5 minutes.
The primary aim of this MR is to speed this notebook up. It was doing a lot of operations in serial, including some avoidable memory copies. I have changed some code to make certain operations more efficient, and used pasha for parallelism - first for loading the constants, then for various reductions. This takes the time from ~5 to ~2 minutes. It's still not using all the available cores, so it may be possible to speed it up more, but that would probably make it more complex, so I stopped here for now.
I also made some other changes while I was working on it:
- If there's no difference from the previous constant found, skip plotting the differences. This can happen when you take the first dark with a new condition, or if you reprocess the same darks (so the 'previous' constant is identical).
- When we do plot the differences, use a diverging colour scale, to better distinguish increases and decreases.
- Various code cleanup - this has made the notebook longer, but I hope it's easier to follow the logic now that some loops are trying to do less.
Other than the difference plots, I haven't tried to change the output at all. I think it might make sense to push some of this inspection and comparison to Caliphant, but that's a separate discussion.
How Has This Been Tested?
Run with FXE darks, output in /gpfs/exfel/data/scratch/kluyvert/lpd-dark-p900320-r26_27_28. Here's the report: LPDOffset_NoiseandDeadPixelsCharacterization.pdf. Compare with this report in CalCat.
Types of changes
- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Refactor (refactoring code with no functionality changes)
Checklist:
- My code follows the code style of this project.