[AGIPD] Add support to round intensity to absolute photon numbers
Description
MID requested some time ago to round the result of AGIPD corrections to absolute number of photons. Assuming the intensity is in units of keV after proper corrections, this is simply a division by the actual photon energy. In certain experimental situations this can improve SNR, reduce the data volume and is done at MID on the analysis-side regularly.
The photon energy can be taken from a DoocsXGM device via the use_xgm_device parameter. It is checked for a 1% tolerance and photon rounding is disabled if this fails. If the XGM device is not available, the photon-energy operating condition is used instead with a warning. After rounding, any values below 0 are set to 0 and masked as BadPixels.VALUE_OUT_OF_RANGE.
To exploit the potential data reduction opportunities, this MR contains two additional changes:
- Make the compressed fields configurable via parameter
- Add code to cast the image data in-place to a different (smaller or equal) type
These two changes allow for pretty impressive results with almost zero cost, assuming you want rounded photons:
| treatment | type | compressed | size | read* |
|---|---|---|---|---|
| standard | f4 |
4.2G (100%) | 4.45s (100%) | |
| standard | f4 |
X | 3.6G (86%) | 13.5s (303%) |
| rounded | f4 |
4.2G (100%) | 4.46s (100%) | |
| rounded | f4 |
X | 403M (10%) | 12.2s (274%) |
| rounded | u2 |
2.2G (52%) | 2.6s (58%) | |
| rounded | u2 |
X | 153M (4%) | 4.33s (97%) |
* Time required for xd.H5File(<one sequence>)['MID_DET_AGIPD1M-1/DET/15CH0:xtdf', 'image.data'].ndarray(), 7 runs warm-up, 7 runs average measurement
When data is casted to an integer-type, any value exceeding its bounds are set to the respectively min/max value and marked as BadPixels.VALUE_OUT_OF_RANGE.
How Has This Been Tested?
xfel-calibrate agipd CORRECT --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' --ctrl-source-template '{}/MDL/FPGA_COMP' --karabo-da AGIPD15 --sequences 0 --karabo-id-control MID_EXP_AGIPD1M1 --receiver-template '{}CH0' --adjust-mg-baseline --bias-voltage 300 --blc-set-min --blc-stripes --cm-dark-fraction 0.15 --cm-dark-range -30 30 --cm-n-itr 4 --common-mode --ff-gain 1.0 --force-hg-if-below --force-mg-if-below --hg-hard-threshold 1000 --low-medium-gap --mg-hard-threshold 1000 --overwrite --rel-gain --sequences-per-node 1 --xray-gain --in-folder /gpfs/exfel/exp/MID/202221/p003217/raw --out-folder <please-use-your-own> --karabo-id MID_DET_AGIPD1M-1 --run 722 --round-photons
Optional flags: --compress-fields mask gain data --recast-image-data u2
Relevant Documents (optional)
Example for p3217, r722, average frame for first sequence of Q4M4:
Types of changes
- New feature (non-breaking change which adds functionality)