[AGIPD][CORRECT] Fixes for correcting AGIPD HIBEF data

SUMMARY:

Changes are done by Jola and I, for running the correction notebook for AGIPD HIBEF.

1. New plot changes.
2. Bugfix for default fillvalue of np.nan from extra-data was not converted to 0.
3. Changing the method of reading data from h5 file for fixing very slow data reading.

Tests:

• Raw data calibrated for testing directly running the notebook:

1. /gpfs/exfel/exp/HED/202031/p900174/raw/r0155
2. /gpfs/exfel/exp/MID/201901/p002542/raw/r0229
3. /gpfs/exfel/exp/SPB/202030/p900119/raw/r0098

Testing using xfel-calibrate CLI:

xfel-calibrate AGIPD CORRECT \
--slurm-mem 750 \
--slurm-name cor_HED_900174_r155 \
--karabo-da -1 \
--receiver-id {}CH0 \
--karabo-id-control HED_EXP_AGIPD500K2G \
--karabo-da-control AGIPD500K2G00 \
--h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
--overwrite \
--sequences-per-node 1 \
--in-folder /gpfs/exfel/exp/HED/202031/p900174/raw/ \
--out-folder /gpfs/exfel/data/scratch/ahmedk/test/AGIPD_HIBEF \
--karabo-id HED_DET_AGIPD500K2G \
--run 155 \
--force-hg-if-below \
--force-mg-if-below \
--low-medium-gap \
--zero-nans \
--modules 0-7 \
--acq-rate 4.5 \
--bias-voltage 200 \
--gain-setting 0 \
--only-offset \
--sequences 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15\

Details related to the 3rd point:

There was an issue that reading image data for the HIBEF AGIPD was taking too long (> 2000 seconds)

It was found out there are restrictions when reading data using indexing (parameter:firange) as we do in agipdlib.read_file function.

I tested with all three mentioned raw data, which had more than 1000 list of indices while load data. But the issue of reading performance was only with HIBEF. Reading the whole image then indexing later was faster by a big factor.

What differs between the HIBEF and the SPB runs is not the number of list of indices used (SPB is bigger) but the fact that the list of indices used while reading data is not increasing because of the many empty pulses present for HIBEF raw data.

Also, the mentioned run for HIBEF was tested for sequence 16 which had zeros counts of 7 and more that is why 2 was used.

Here https://docs.h5py.org/en/latest/high/dataset.html#fancy-indexing it was mentioned regarding fancy indexing that

The following restrictions exist:

Selection coordinates must be given in increasing order
Duplicate selections are ignored
Very long lists (> 1000 elements) may produce poor performance

Reviewers:

@kluyvert @danilevc

cal_tools/cal_tools/agipdlib.py

 from pathlib import Path
+import os
 from typing import Optional, Tuple
 
 import h5py
@@ -264,15 +265,15 @@ class AgipdCorrections:
         module_idx = int(file_name.split('/')[-1].split('-')[2][-2:])
         agipd_base = self.h5_data_path.format(module_idx)
         idx_base = self.h5_index_path.format(module_idx)
-        data_path = f'{agipd_base}/image'
+        data_path = os.path.join(agipd_base, 'image')
         data_dict = self.shared_dict[i_proc]
 
         try:
             f = h5py.File(file_name, 'r')
             group = f[data_path]
 
-            valid, first_index, last_index, valid_trains, valid_indices = \
-                self.get_valid_image_idx(idx_base, f)
+            valid, first_index, last_index, valid_trains, valid_indices, \
+                zero_count = self.get_valid_image_idx(idx_base, f)
             firange = self.gen_valid_range(first_index, last_index,
                                            self.max_cells, agipd_base, f,
                                            valid_indices)
@@ -286,7 +287,15 @@ class AgipdCorrections:
             data_dict['moduleIdx'][0] = module_idx
             data_dict['nImg'][0] = n_img
 
-            raw_data = group['data'][firange]
+            # if zero_count is high reading performance through
+            # indexing will be slower compared to reading the whole
+            # image.
+            # 50 was chosen based on tested data with 100 zero_count
+            if zero_count > 50:
+                read_raw_data = group['data'][:]
+                raw_data = read_raw_data[firange]
+            else:
+                raw_data = group['data'][firange]
             data_dict['data'][:n_img] = raw_data[:, 0]
             data_dict['rawgain'][:n_img] = raw_data[:, 1]
 
@@ -598,0+607,0 @@
     def get_valid_image_idx(self, idx_base, infile, index_v=2):
         """ Return the indices of valid data
         """
-
+        zero_count = 0
         if index_v == 2:
             count = np.squeeze(infile[idx_base + "image/count"])
             first = np.squeeze(infile[idx_base + "image/first"])
-            if np.count_nonzero(count != 0) == 0:
+            nonzero_count = np.count_nonzero(count != 0)
+            zero_count = count.shape[0] - nonzero_count
+            zero_indices = np.where(count == 0)[0]
+            if nonzero_count == 0:
                 raise IOError("File has no valid counts")
             valid = count != 0
             idxtrains = np.squeeze(infile["/INDEX/trainId"])
@@ -624,9 +636,10 @@ class AgipdCorrections:
 
         elif index_v == 1:
             status = np.squeeze(infile[idx_base + "image/status"])
-            if np.count_nonzero(status != 0) == 0:
-                raise IOError("File {} has no valid counts".format(
-                    infile))
+            nonzero_count = np.count_nonzero(status != 0)
+            zero_count = count.shape[0] - nonzero_count
+            if nonzero_count == 0:
+                raise IOError(f"File {infile} has no valid counts")
             last = np.squeeze(infile[idx_base + "image/last"])
             first = np.squeeze(infile[idx_base + "image/first"])
             valid = status != 0
@@ -641,9 +654,10 @@ class AgipdCorrections:
             valid_indices = None
         else:
             raise AttributeError(
-                "Not a known raw format version: {}".format(index_v))
+                f"Not a known raw format version: {index_v}")
 
-        return valid, first_index, last_index, idxtrains, valid_indices
+        return (valid, first_index, last_index, idxtrains,
+                valid_indices, zero_count)
 
     def gen_valid_range(self, first_index, last_index, max_cells, agipd_base,
                         infile, valid_indices=None):