[AGIPD] Add support to round intensity to absolute photon numbers
Description
MID requested some time ago to round the result of AGIPD corrections to absolute number of photons. Assuming the intensity is in units of keV after proper corrections, this is simply a division by the actual photon energy. In certain experimental situations this can improve SNR, reduce the data volume and is done at MID on the analysis-side regularly.
The photon energy can be taken from a DoocsXGM device via the use_xgm_device parameter. It is checked for a 1% tolerance and photon rounding is disabled if this fails. If the XGM device is not available, the photon-energy operating condition is used instead with a warning. After rounding, any values below 0 are set to 0 and masked as BadPixels.VALUE_OUT_OF_RANGE.
To exploit the potential data reduction opportunities, this MR contains two additional changes:
- Make the compressed fields configurable via parameter
- Add code to cast the image data in-place to a different (smaller or equal) type
These two changes allow for pretty impressive results with almost zero cost, assuming you want rounded photons:
| treatment | type | compressed | size | read* |
|---|---|---|---|---|
| standard | f4 |
4.2G (100%) | 4.45s (100%) | |
| standard | f4 |
X | 3.6G (86%) | 13.5s (303%) |
| rounded | f4 |
4.2G (100%) | 4.46s (100%) | |
| rounded | f4 |
X | 403M (10%) | 12.2s (274%) |
| rounded | u2 |
2.2G (52%) | 2.6s (58%) | |
| rounded | u2 |
X | 153M (4%) | 4.33s (97%) |
* Time required for xd.H5File(<one sequence>)['MID_DET_AGIPD1M-1/DET/15CH0:xtdf', 'image.data'].ndarray(), 7 runs warm-up, 7 runs average measurement
When data is casted to an integer-type, any value exceeding its bounds are set to the respectively min/max value and marked as BadPixels.VALUE_OUT_OF_RANGE.
How Has This Been Tested?
xfel-calibrate agipd CORRECT --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' --ctrl-source-template '{}/MDL/FPGA_COMP' --karabo-da AGIPD15 --sequences 0 --karabo-id-control MID_EXP_AGIPD1M1 --receiver-template '{}CH0' --adjust-mg-baseline --bias-voltage 300 --blc-set-min --blc-stripes --cm-dark-fraction 0.15 --cm-dark-range -30 30 --cm-n-itr 4 --common-mode --ff-gain 1.0 --force-hg-if-below --force-mg-if-below --hg-hard-threshold 1000 --low-medium-gap --mg-hard-threshold 1000 --overwrite --rel-gain --sequences-per-node 1 --xray-gain --in-folder /gpfs/exfel/exp/MID/202221/p003217/raw --out-folder <please-use-your-own> --karabo-id MID_DET_AGIPD1M-1 --run 722 --round-photons
Optional flags: --compress-fields mask gain data --recast-image-data u2
Relevant Documents (optional)
Example for p3217, r722, average frame for first sequence of Q4M4:
Types of changes
- New feature (non-breaking change which adds functionality)
Reviewers
Merge request reports
Activity
changed milestone to %3.5.2
added Waiting for review label
As this changes the meaning of the output data (when the new option is enabled), would it make sense to change the source & key name in the output files, something like
'MID_DET_AGIPD1M-1/CALIB/15CH0:xtdf', 'image.photonCounts'? As this stands, it would be easy to load the data assuming it's photon counts (or energy) and get the wrong answer because the calibration is not using the option you expect. A different name would avoid that ambiguity. It would also make it possible to save both if necessary, although of course the files would be even bigger than before.
added 1 commit
- 6c10238e - (fixup) Clarify comment on slopes_ff_from_files
added 1 commit
- 55ff13f7 - (Fixup) Mask any out-of-bounds values in image data when downcasting
added 1 commit
- 25f64d61 - (fixup) Make sure in-place cast can be done without copy
537 549 538 550 # Write the corrected data 539 551 for field in image_fields: 540 if field in compress_fields: 552 if field in self.compress_fields: 541 553 self._write_compressed_frames( 542 554 image_grp[field], data_dict[field][:n_img], 543 555 ) 544 556 else: 545 557 image_grp[field][:] = data_dict[field][:n_img] 546 558 559 # Restore the original arrays with their proper dtype. 560 for field, dtype in self.recast_image_fields.items(): 561 data_dict[field] = orig_image_arrays[field] 562 - Comment on lines +559 to +562
Modifying
data_dict(a reference to an object inself.shared_dict) and then restoring the values afterwards doesn't seem great. If there's an error during the method, the restore step may not happen.If we do
data_dict = self.shared_dict[i_proc].copy(), then we can replace values indata_dictand not worry about restoring them afterwards. dict.copy() is a shallow copy, so it doesn't copy the arrays inside the dictionary. changed this line in version 9 of the diff
removed milestone %3.5.2
changed milestone to %3.6.0
removed milestone %3.6.0
changed milestone to %3.6.2
added 1 commit
- 86adc245 - (fixup) Make changes to local copy in AgipdCorrections.write_file rather than...
- Resolved by Thomas Kluyver
I'd still ideally like this to have a different name in the output files. The data means something different with this option set, so I'd like it to be clear in the naming when you're accessing photon counts. But I appreciate that throws up logistical issues around using the data which we haven't really worked out.
- Last reply by Thomas Kluyver
changed milestone to %3.6.1
mentioned in commit 835586a1
removed Waiting for review label
mentioned in merge request !746 (merged)
