" if creation_time.replace(tzinfo=None) < dateutil.parser.parse('2020-01-31'):\n",
" print(\"Set gain-setting to None for runs taken before 2020-01-31\")\n",
...
...
%% Cell type:markdown id: tags:
# Characterize AGIPD Pulse Capacitor Data #
Author: S. Hauf, Version 1.0
The following code characterizes AGIPD gain via data take with the pulse capacitor source (PCS). The PCS allows scanning through the high and medium gains of AGIPD, by subsequently intecreasing the number of charge pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees in a given integration time.
Because induced charge does not originate from X-rays on the sensor, the gains evaluated here will later need to be rescaled with gains deduced from X-ray data.
PCS data is organized into multiple runs, as the on-ASIC current source cannot supply all pixels of a given module with charge at the same time. Hence, only certain pixel rows will have seen charge for a given image. These rows then first need to be combined into single module images again.
We then use a K-means clustering algorithm to identify components in the resulting per-pixel data series, matching to three general regions:
* a high gain slope
* a transition region, where gain switching occurs
* a medium gain slope.
The same regions are present in the gain-bit data and are used to deduce the switching threshold.
The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data.
%% Cell type:code id: tags:
``` python
in_folder='/gpfs/exfel/exp/MID/201931/p900091/raw'# path to input data, required
in_folder='/gpfs/exfel/exp/SPB/202030/p900138/raw/'# path to input data, required
modules=[1,]# modules to work on, required, range allowed
out_folder="/gpfs/exfel/exp/MID/201931/p900091/usr/PC/4.5_250/"# path to output to, required
runs=[30,23,24,25,26,27,28,29]# runs to use, required, range allowed
n_sequences=3# number of sequence files, starting for 0 to evaluate
out_folder="/gpfs/exfel/data/scratch/ahmedk/test/pc"# path to output to, required
runs=[466,467,468,469,470,471,472,473]# runs to use, required, range allowed
n_sequences=1# number of sequence files, starting for 0 to evaluate
cluster_profile="noDB"# The ipcluster profile to use
local_output=True# output constants locally
db_output=False# output constants to database
bias_voltage=300# detector bias voltage
cal_db_interface="tcp://max-exfl016:8019"# the database interface to use
mem_cells=0.# number of memory cells used, use 0 to auto-derive
interlaced=False# assume interlaced data format, for data prior to Dec. 2017
fit_hook=True# fit a hook function to medium gain slope
rawversion=2# RAW file format version
instrument="MID"
instrument="SPB"
high_res_badpix_3d=False# set this to True if you need high-resolution 3d bad pixel plots. Runtime: ~ 1h
acq_rate=0.# the detector acquisition rate, use 0 to try to auto-determine
use_dir_creation_date=True
creation_time=""# To overwrite the measured creation_time. Required Format: YYYY-MM-DD HR:MN:SC.ms e.g. 2019-07-04 11:02:41.00
gain_setting=0.1# gain setting can have value 0 or 1, Default=0.1 for no (None) gain-setting
karabo_da_control="AGIPD1MCTRL00"# karabo DA for control infromation
h5path_ctrl='/CONTROL/{}/MDL/FPGA_COMP_TEST'# path to control information
```
%% Cell type:code id: tags:
``` python
# imports, usually no need to change anything here
fromdatetimeimportdatetime
importdateutil.parser
importh5py
importos
importnumpyasnp
importmatplotlib
matplotlib.use("Qt4Agg")
importmatplotlib.pyplotasplt
%matplotlibinline
# make sure a cluster is running with ipcluster start --n=32, give it a while to start
The following two cells contain the actual algorithm logic as well as a preview of a single pixel and memory cells visualizing the data and the concepts.
We start out with calculating an estimate of the slope in proximity of a given data value. This is done by calculating the slopes of a given value with 15 neighbours and averaging the result. Values are then clustered by these slopes into three regions via a K-means algorithm.
* for the first region a linear function is fitted to the data, determining the gain slope and offset for the high gain mode.
$$y = mx + b$$
* for the second and third region a composite function of the form:
$$y = A*e^{-(x-O)/C}+mx+b$$
is fitted, covering both the transition region and the medium gain slope.
The follwing is an visualization of the clustering and fitting for a subset of pixels. If data significantly mismatches expectations, the clustering and fitting algorithms should fail for this subset:
* the first plot shows the clustering results for pixels which were sucessfully evaluated
* the second plot shows the clustering results for pixels which failed to evaluate
* the third plot shows the fits and fit residuals for the pixel clusters shown in the first plot
Non-smooth behaviour is an indication that you are errorously processing interleaved data that is not, or vice versa, or have the wrong number of memory cells set.
The following plots show the results of bad pixel evaluation for all evaluated memory cells. Cells are stacked in the Z-dimension, while pixels values in x/y are rebinned with a factor of 2. This excludes single bad pixels present only in disconnected pixels. Hence, any bad pixels spanning at least 2 pixels in the x/y-plane, or across at least two memory cells are indicated. Colors encode the bad pixel type, or mixed type.
