[AGIPD] Add support to round intensity to absolute photon numbers

Description

MID requested some time ago to round the result of AGIPD corrections to absolute number of photons. Assuming the intensity is in units of keV after proper corrections, this is simply a division by the actual photon energy. In certain experimental situations this can improve SNR, reduce the data volume and is done at MID on the analysis-side regularly.

The photon energy can be taken from a DoocsXGM device via the use_xgm_device parameter. It is checked for a 1% tolerance and photon rounding is disabled if this fails. If the XGM device is not available, the photon-energy operating condition is used instead with a warning. After rounding, any values below 0 are set to 0 and masked as BadPixels.VALUE_OUT_OF_RANGE.

To exploit the potential data reduction opportunities, this MR contains two additional changes:

• Make the compressed fields configurable via parameter
• Add code to cast the image data in-place to a different (smaller or equal) type

These two changes allow for pretty impressive results with almost zero cost, assuming you want rounded photons:

treatment	type	compressed	size	read*
standard	f4		4.2G (100%)	4.45s (100%)
standard	f4	X	3.6G (86%)	13.5s (303%)
rounded	f4		4.2G (100%)	4.46s (100%)
rounded	f4	X	403M (10%)	12.2s (274%)
rounded	u2		2.2G (52%)	2.6s (58%)
rounded	u2	X	153M (4%)	4.33s (97%)

* Time required for xd.H5File(<one sequence>)['MID_DET_AGIPD1M-1/DET/15CH0:xtdf', 'image.data'].ndarray(), 7 runs warm-up, 7 runs average measurement

When data is casted to an integer-type, any value exceeding its bounds are set to the respectively min/max value and marked as BadPixels.VALUE_OUT_OF_RANGE.

How Has This Been Tested?

xfel-calibrate agipd CORRECT --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' --ctrl-source-template '{}/MDL/FPGA_COMP' --karabo-da AGIPD15 --sequences 0 --karabo-id-control MID_EXP_AGIPD1M1 --receiver-template '{}CH0' --adjust-mg-baseline --bias-voltage 300 --blc-set-min --blc-stripes --cm-dark-fraction 0.15 --cm-dark-range -30 30 --cm-n-itr 4 --common-mode --ff-gain 1.0 --force-hg-if-below --force-mg-if-below --hg-hard-threshold 1000 --low-medium-gap --mg-hard-threshold 1000 --overwrite --rel-gain --sequences-per-node 1 --xray-gain --in-folder /gpfs/exfel/exp/MID/202221/p003217/raw --out-folder <please-use-your-own> --karabo-id MID_DET_AGIPD1M-1 --run 722 --round-photons

Optional flags: --compress-fields mask gain data --recast-image-data u2

Relevant Documents (optional)

Example for p3217, r722, average frame for first sequence of Q4M4:

Types of changes

• New feature (non-breaking change which adds functionality)

Reviewers

@ahmedk @kluyvert @moellerj

src/cal_tools/agipdlib.py

@@ -475,7 +475,10 @@ class AgipdCorrections:
         agipd_base = f'INSTRUMENT/{self.h5_data_path}/'.format(module_idx)
         idx_base = self.h5_index_path.format(module_idx)
         data_path = f'{agipd_base}/image'
-        data_dict = self.shared_dict[i_proc]
+
+        # Obtain a shallow copy of the pointer map to allow for local
+        # changes in this method.
+        data_dict = self.shared_dict[i_proc].copy()
 
         image_fields = [
             'trainId', 'pulseId', 'cellId', 'data', 'gain', 'mask', 'blShift',
@@ -486,11 +489,8 @@ class AgipdCorrections:
             return
         trains = data_dict['trainId'][:n_img]
 
-        # Create new views with a different dtype for arrays to be
-        # recast and store the original array temporarily.
-        orig_image_arrays = {}
+        # Re-cast fields in-place, i.e. using the same memory region.
         for field, dtype in self.recast_image_fields.items():
-            orig_image_arrays[field] = data_dict[field]
             data_dict[field] = cast_array_inplace(data_dict[field], dtype)
 
         with h5py.File(ofile_name, "w") as outfile:
@@ -536,10 +536,6 @@ class AgipdCorrections:
                 else:
                     image_grp[field][:] = data_dict[field][:n_img]
 
-        # Restore the original arrays with their proper dtype.
-        for field, dtype in self.recast_image_fields.items():
-            data_dict[field] = orig_image_arrays[field]
-
     def _write_compressed_frames(self, dataset, arr):
         """Compress gain/mask frames in multiple threads, and save their data