" line = ['{}, {} gain'.format(l_data_name[i], gain_names[gain]),\n",
...
...
%% Cell type:markdown id: tags:
# Characterize Dark Images #
Author: S. Hauf, Version: 0.1
The following code analyzes a set of dark images taken with the AGIPD detector to deduce detector offsets , noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell. Data for the detector's three gain stages needs to be present, separated into separate runs.
The evaluated calibration constants are stored locally and injected in the calibration data base.
%% Cell type:code id: tags:
``` python
cluster_profile="noDB"# The ipcluster profile to use
in_folder="/gpfs/exfel/d/raw/SPB/202030/p900138/"# path to input data, required
out_folder="/gpfs/exfel/data/scratch/ahmedk/test/AGIPD3"# path to output to, required
out_folder="/gpfs/exfel/data/scratch/ahmedk/test/AGIPDb1"# path to output to, required
sequences=[0]# sequence files to evaluate.
modules=[-1]# list of modules to evaluate, RANGE ALLOWED
run_high=264# run number in which high gain data was recorded, required
run_med=265# run number in which medium gain data was recorded, required
run_low=266# run number in which low gain data was recorded, required
karabo_id="SPB_DET_AGIPD1M-1"# karabo karabo_id
karabo_da=[-1]# data aggregators
receiver_id="{}CH0"# inset for receiver devices
path_template='RAW-R{:04d}-{}-S{:05d}.h5'# the template to use to access data
h5path='/INSTRUMENT/{}/DET/{}:xtdf/image'# path in the HDF5 file to images
h5path_idx='/INDEX/{}/DET/{}:xtdf/image'# path in the HDF5 file to images
h5path_ctrl='/CONTROL/{}/MDL/FPGA_COMP_TEST'# path to control information
karabo_id_control="SPB_IRU_AGIPD1M1"# karabo-id for control device
karabo_da_control='DA02'# karabo DA for control infromation
use_dir_creation_date=True# use dir creation date as data production reference date
cal_db_interface="tcp://max-exfl016:8020"# the database interface to use
cal_db_timeout=3000000# timeout on caldb requests"
local_output=True# output constants locally
db_output=False# output constants to database
mem_cells=0# number of memory cells used, set to 0 to automatically infer
bias_voltage=300# detector bias voltage
gain_setting=0.1# the gain setting, use 0.1 to try to auto-determine
acq_rate=0.# the detector acquisition rate, use 0 to try to auto-determine
interlaced=False# assume interlaced data format, for data prior to Dec. 2017
rawversion=2# RAW file format version
thresholds_offset_sigma=3.# thresholds in terms of n sigma noise for offset deduced bad pixels
thresholds_offset_hard=[4000,8500]# thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_offset_hard=[0,0]# This offset threshold is left for back compatability or for defining threshold for the 3 gains
thresholds_offset_hard_hg=[4000,8500]# High-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_offset_hard_mg=[4000,8500]# Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_offset_hard_lg=[4000,8500]# Low-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_sigma=5.# thresholds in terms of n sigma noise for offset deduced bad pixels
thresholds_noise_hard=[4,20]# thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_hard=[0,0]# This noise threshold is left for back compatability or for defining threshold for the 3 gains
thresholds_noise_hard_hg=[4,20]# High-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_hard_mg=[4,20]# Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_hard_lg=[4,20]# Low-gain thresholds in absolute ADU terms for offset deduced bad pixels
gain_sep_sigma=5.# gain separation sigma
high_res_badpix_3d=False# set this to True if you need high-resolution 3d bad pixel plots. Runtime: ~ 1h
```
%% Cell type:code id: tags:
``` python
# imports and things that do not usually need to be changed
print(f"Will process a total of {total_sequences} sequences.")
```
%% Cell type:markdown id: tags:
## Calculate Offsets, Noise and Thresholds ##
The calculation is performed per-pixel and per-memory-cell. Offsets are simply the median value for a set of dark data taken at a given gain, noise the standard deviation, and gain-bit values the medians of the gain array.
The thresholds for gain switching are then defined as the mean value between in individual gain bit levels. Note that these thresholds need to be refined with charge induced thresholds, as the two are not the same.
Single cell overviews allow to identify potential effects on all memory cells, e.g. on sensor level. Additionally, they should serve as a first sanity check on expected behaviour, e.g. if structuring on the ASIC level is visible in the offsets, but otherwise no immediate artifacts are visible.
The following plots show the results of bad pixel evaluation for all evaluated memory cells. Cells are stacked in the Z-dimension, while pixels values in x/y are rebinned with a factor of 2. This excludes single bad pixels present only in disconnected pixels. Hence, any bad pixels spanning at least 4 pixels in the x/y-plane, or across at least two memory cells are indicated. Colors encode the bad pixel type, or mixed type.
The following tables show summary information for the evaluated module. Values for currently evaluated constants are compared with values for pre-existing constants retrieved from the calibration database.
