Newer
Older
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
"# Summary of AGIPD dark characterization #\n",
"\n",
"The following report shows a set of dark images taken with the AGIPD detector to deduce detector offsets, noise, bad-pixel maps and thresholding. All four types of constants are evaluated per-pixel and per-memory cell.\n",
"\n",
"\n",
"**The offset** ($O$) is defined as the median ($M$) of the dark signal ($Ds$) over trains ($t$) for a given pixel ($x,y$) and memory cell ($c$). \n",
"\n",
"**The noise** $N$ is the standard deviation $\\sigma$ of the dark signal.\n",
"\n",
"$$ O_{x,y,c} = M(Ds)_{t} ,\\,\\,\\,\\,\\,\\, N_{x,y,c} = \\sigma(Ds)_{t}$$\n",
"\n",
"**The bad pixel** mask is encoded as a bit mask.\n",
"\n",
"**\"OFFSET_OUT_OF_THRESHOLD\":**\n",
"\n",
"Offset outside of bounds:\n",
"\n",
"$$M(O)_{x,y} - \\sigma(O)_{x,y} * \\mathrm{thresholds\\_offset\\_sigma} < O < M(O)_{x,y} + \\sigma(O)_{x,y} * \\mathrm{thresholds\\_offset\\_sigma} $$\n",
"\n",
"or offset outside of hard limits\n",
"\n",
"$$ \\mathrm{thresholds\\_offset\\_hard}_\\mathrm{low} < O < \\mathrm{thresholds\\_offset\\_hard}_\\mathrm{high} $$\n",
"\n",
"**\"NOISE_OUT_OF_THRESHOLD\":**\n",
"\n",
"Noise outside of bounds:\n",
"\n",
"$$M(N)_{x,y} - \\sigma(N)_{x,y} * \\mathrm{thresholds\\_noise\\_sigma} < N < M(N)_{x,y} + \\sigma(N)_{x,y} * \\mathrm{thresholds\\_noise\\_sigma} $$\n",
"\n",
"or noise outside of hard limits\n",
"\n",
"$$\\mathrm{thresholds\\_noise\\_hard}_\\mathrm{low} < N < \\mathrm{thresholds\\_noise\\_hard}_\\mathrm{high} $$\n",
"\n",
"**\"OFFSET_NOISE_EVAL_ERROR\":**\n",
"\n",
"Offset and Noise both not $nan$ values \n",
"\n",
"Values: $\\mathrm{thresholds\\_offset\\_sigma}$, $\\mathrm{thresholds\\_offset\\_hard}$, $\\mathrm{thresholds\\_noise\\_sigma}$, $\\mathrm{thresholds\\_noise\\_hard}$ are given as parameters.\n",
"\n",
"**\"GAIN_THRESHOLDING_ERROR\":**\n",
"\n",
"Bad gain separated pixels with sigma separation less than gain_separation_sigma_threshold\n",
"$$ sigma\\_separation = \\frac{\\mathrm{gain\\_offset} - \\mathrm{previous\\_gain\\_offset}}{\\sqrt{\\mathrm{gain\\_offset_{std}}^\\mathrm{2} + \\mathrm{previuos\\_gain\\_offset_{std}}^\\mathrm{2}}}$$ \n",
"$$ Bad\\_separation = sigma\\_separation < \\mathrm{gain\\_separation\\_sigma\\_threshold} $$"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"cluster_profile = \"noDB\" # The ipcluster profile to use\n",
"out_folder = \"/gpfs/exfel/data/scratch/ahmedk/test/AGIPD_bad_gain_separation/\" # path to output to, required\n",
"gain_names = ['High gain', 'Medium gain', 'Low gain'] # a list of gain names to be used in plotting\n",
"threshold_names = ['HG-MG threshold', 'MG_LG threshold'] # a list of gain names to be used in plotting"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from collections import OrderedDict\n",
"import copy\n",
"from datetime import datetime\n",
"import os\n",
"import warnings\n",
"warnings.filterwarnings('ignore')\n",
"\n",
"import glob\n",
"import h5py\n",
"from IPython.display import display, Markdown, Latex\n",
"import numpy as np\n",
"import matplotlib\n",
"matplotlib.use(\"agg\")\n",
"import matplotlib.patches as patches\n",
"import matplotlib.pyplot as plt\n",
"%matplotlib inline\n",
"import tabulate\n",
"from cal_tools.ana_tools import get_range\n",
"from extra_geom import AGIPD_1MGeometry\n",
"from iCalibrationDB import Detectors\n",
"from XFELDetAna.plotting.heatmap import heatmapPlot\n",
"from XFELDetAna.plotting.simpleplot import simplePlot"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# TODO: After changes in the Cal DB interface read files from cal repository\n",
"\n",
"# Load constants from local files\n",
"data = OrderedDict()\n",
"mod_names = []\n",
"# Loop over modules\n",
"for i in range(16):\n",
" qm = \"Q{}M{}\".format(i//4 + 1, i % 4 + 1)\n",
" # loop over constants\n",
" detinst = getattr(Detectors, dinstance)\n",
" for const in ['Offset', 'Noise', 'ThresholdsDark', 'BadPixelsDark']:\n",
" det = getattr(detinst, qm)\n",
" if det is None:\n",
" continue\n",
" det_name = det.device_name\n",
"\n",
" fpath = '{}/const_{}_{}.h5'.format(out_folder, const, det_name)\n",
" oldfpath = '{}/old/const_{}_{}.h5'.format(out_folder, const, det_name)\n",
" if not os.path.isfile(fpath):\n",
" continue\n",
" with h5py.File(fpath, 'r') as f:\n",
" if qm not in data:\n",
" mod_names.append(qm)\n",
" data[qm] = OrderedDict()\n",
"\n",
" data[qm][const] = f[\"data\"][()]\n",
"\n",
" if not os.path.isfile(oldfpath):\n",
" continue\n",
"\n",
" with h5py.File(oldfpath, 'r') as oldf:\n",
" if qm not in old_cons:\n",
" old_cons[qm] = OrderedDict()\n",
" old_cons[qm][const] = oldf[\"data\"][()]\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"modules = np.argsort(mod_names)\n",
"# extracting constant shape.\n",
"for qm, constant in data.items():\n",
" for cname, cons in constant.items():\n",
" shape = data[qm][cname].shape\n",
" if cname not in cons_shape:\n",
" cons_shape[cname] = shape\n",
"constants = {}\n",
"prev_const = {}\n",
"for cname, sh in cons_shape.items():\n",
" constants[cname]= np.zeros((len(mod_names),) + sh[:])\n",
" prev_const[cname]= np.zeros((len(mod_names),) + sh[:])\n",
"for i, mod in enumerate(modules):\n",
" for cname, cval in constants.items():\n",
" cval[i] = data[mod_names[mod]][cname]\n",
" prev_const[cname][i] = old_cons[mod_names[mod]][cname]\n",
"mod_names = np.array(mod_names)[modules]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"display(Markdown('## Processed modules ##'))\n",
"\n",
"fig, ax = plt.subplots(1, figsize=(10, 10))\n",
"ax.set_axis_off()\n",
"\n",
"ax.set_xlim(0, 90)\n",
"ax.set_ylim(0, 75)\n",
"asic_pos = 5\n",
"for iq, q_x in enumerate([[43,66],[45,34],[4,32],[2,64]]):\n",
" for im in range(4):\n",
" color = 'gray'\n",
" if 'Q{}M{}'.format(iq+1, im+1) in mod_names:\n",
" color = 'green'\n",
" if 'Noise' not in constants.keys() or \\\n",
" np.nanmean(constants['Noise'][counter, :, :, :, 0]) == 0:\n",
" x = q_x[0]\n",
" for i_as in range(8):\n",
" ax.add_patch(matplotlib.patches.Rectangle((x,q_x[1]-q_st*im), l_x, l_y, linewidth=2,edgecolor='darkgreen',\n",
" facecolor=color, fill=True))\n",
" x += asic_pos\n",
" ax.text(q_x[0]+14.5, q_x[1]-q_st*im+1.5, 'Q{}M{}'.format(\n",
" iq+1, im+1), fontsize=24, color='gold')\n",
"\n",
"_ = ax.legend(handles=[patches.Patch(facecolor='red', label='No data'),\n",
" patches.Patch(facecolor='gray', label='Not processed'),\n",
" patches.Patch(facecolor='green', label='Processed')],\n",
" loc='outside-top', ncol=3, bbox_to_anchor=(0.1, 0.25, 0.7, 0.8))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary figures across Modules ##\n",
"\n",
"The following plots give an overview of calibration constants averaged across pixels and memory cells. A bad pixel mask is applied."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"#Initializing AGIPD geometry\n",
"geom = AGIPD_1MGeometry.from_quad_positions(quad_pos=[\n",
" (-525, 625),\n",
" (-550, -10),\n",
" (520, -160),\n",
" (542.5, 475),\n",
" ])\n",
"\n",
"Mod_data=OrderedDict()\n",
"for const_name, const in constants.items():\n",
" if const_name == 'BadPixelsDark':\n",
" continue\n",
" \n",
" Mod_data[const_name] = OrderedDict()\n",
" Mod_data['d-{}'.format(const_name)] = OrderedDict()\n",
" Mod_data['dpct-{}'.format(const_name)] = OrderedDict()\n",
" display(Markdown('##### {}'.format(const_name)))\n",
"\n",
" # Avoid plotting for high gain offset in 3 index \n",
" # for Threshold gain dimension.\n",
" if const_name == 'ThresholdsDark':\n",
" if gain == 2:\n",
" for i in range(16):\n",
" qm = \"Q{}M{}\".format(i//4 + 1, i % 4 + 1)\n",
" if const.shape[0] > i:\n",
" values = np.nanmean(const[i, :, :, :, gain], axis=2)\n",
" dval = np.nanmean(prev_const[const_name][i, :, :, :, gain], axis=2)\n",
" values[values == 0] = np.nan\n",
" dval[dval == 0] = np.nan\n",
" dval = values - dval\n",
" values = np.moveaxis(values, 0, -1).reshape(1, values.shape[1], values.shape[0])\n",
" dval = np.moveaxis(dval, 0, -1).reshape(1, dval.shape[1], dval.shape[0])\n",
" dval_pct = np.moveaxis(dval_pct, 0, -1).reshape(1, dval_pct.shape[1], dval_pct.shape[0])\n",
" else:\n",
" # if module not available fill arrays with nan\n",
" values = np.zeros((1, 512, 128),dtype=np.float64)\n",
" values[values == 0] = np.nan\n",
" dval = values \n",
" dval_pct = dval\n",
" try:\n",
" Mod_data[const_name][gain_names[gain]] = np.concatenate((Mod_data[const_name][gain_names[gain]],\n",
" Mod_data['d-{}'.format(const_name)][gain_names[gain]] = \\\n",
" np.concatenate((Mod_data['d-{}'.format(const_name)][gain_names[gain]],\n",
" dval), axis=0)\n",
" Mod_data['dpct-{}'.format(const_name)][gain_names[gain]] = \\\n",
" np.concatenate((Mod_data['dpct-{}'.format(const_name)][gain_names[gain]],\n",
" dval_pct), axis=0)\n",
" Mod_data[const_name][gain_names[gain]] = values\n",
" Mod_data['d-{}'.format(const_name)][gain_names[gain]] = dval\n",
" Mod_data['dpct-{}'.format(const_name)][gain_names[gain]] = dval_pct\n",
"\n",
" vmin, vmax = get_range(Mod_data[const_name][gain_names[gain]], 2)\n",
" dvmin, dvmax = get_range(Mod_data['d-{}'.format(const_name)][gain_names[gain]], 2)\n",
" dpctvmin, dpctvmax = get_range(Mod_data['dpct-{}'.format(const_name)][gain_names[gain]], 2)\n",
" display(Markdown('###### {} ######'.format(glabel[gain])))\n",
" \n",
" gs = gridspec.GridSpec(2, 2)\n",
" fig = plt.figure(figsize=(24, 32))\n",
" \n",
" ax0 = fig.add_subplot(gs[0, :])\n",
" ax1 = fig.add_subplot(gs[1, 0])\n",
" ax2 = fig.add_subplot(gs[1, 1])\n",
" geom.plot_data_fast(Mod_data[const_name][gain_names[gain]], vmin=vmin, vmax=vmax,\n",
" ax=ax0, colorbar={\n",
" 'shrink': 0.9,\n",
" 'pad': 0.05})\n",
"\n",
" colorbar = ax0.images[0].colorbar\n",
" if const_name != 'BadPixelsDark':\n",
" colorbar.set_label('ADUs')\n",
" ax0.set_title('{} {}'.format(const_name, glabel[gain]), fontsize=15)\n",
" ax0.set_xlabel('Columns', fontsize=15)\n",
" ax0.set_ylabel('Rows', fontsize=15)\n",
" # Plotting the difference with previous constants in ADUs and %\n",
" geom.plot_data_fast(Mod_data['d-{}'.format(const_name)][gain_names[gain]],\n",
" vmin=dvmin, vmax=dvmax, ax=ax1,colorbar={\n",
" 'shrink': 0.6,\n",
" 'pad': 0.1,\n",
" 'orientation': 'horizontal'})\n",
" colorbar = ax1.images[0].colorbar\n",
" if const_name != 'BadPixelsDark':\n",
" colorbar.set_label('ADUs')\n",
" geom.plot_data_fast(Mod_data['dpct-{}'.format(const_name)][gain_names[gain]], \n",
" 'shrink': 0.6,\n",
" 'pad': 0.1,\n",
" 'orientation': 'horizontal'})\n",
" colorbar = ax2.images[0].colorbar\n",
" colorbar.set_label('%')\n",
" ax1.set_title('Difference with previous {} {}'.format(const_name, \n",
" glabel[gain]), fontsize=15)\n",
" ax1.set_xlabel('Columns', fontsize=15)\n",
" ax1.set_ylabel('Rows', fontsize=15)\n",
" ax2.set_title('Difference with previous {} {} (%)'.format(const_name, glabel[gain]), fontsize=15)\n",
" ax2.set_xlabel('Columns', fontsize=15)\n",
" ax2.set_ylabel('Rows', fontsize=15)\n",
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
},
"outputs": [],
"source": [
"# Loop over capacitor settings, modules, constants\n",
"for const_name, const in constants.items():\n",
"\n",
" display(Markdown('### Summary across Modules - {}'.format(const_name)))\n",
" for gain in range(3):\n",
"\n",
" if const_name == 'ThresholdsDark':\n",
" if gain == 2:\n",
" continue\n",
" data = np.copy(const[:, :, :, :, gain])\n",
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
" if const_name != 'BadPixelsDark':\n",
" label = '{} value [ADU], good pixels only'.format(const_name)\n",
" data[constants['BadPixelsDark'][:, :, :, :, gain] > 0] = np.nan\n",
" datamean = np.nanmean(data, axis=(1, 2))\n",
"\n",
" fig = plt.figure(figsize=(15, 6), tight_layout={\n",
" 'pad': 0.2, 'w_pad': 1.3, 'h_pad': 1.3})\n",
" ax = fig.add_subplot(121)\n",
" else:\n",
" label = 'Fraction of bad pixels'\n",
" data[data > 0] = 1.0\n",
" datamean = np.nanmean(data, axis=(1, 2))\n",
" datamean[datamean == 1.0] = np.nan\n",
"\n",
" fig = plt.figure(figsize=(15, 6), tight_layout={\n",
" 'pad': 0.2, 'w_pad': 1.3, 'h_pad': 1.3})\n",
" ax = fig.add_subplot(111)\n",
"\n",
" d = []\n",
" for im, mod in enumerate(datamean):\n",
" d.append({'x': np.arange(mod.shape[0]),\n",
" 'y': mod,\n",
" 'drawstyle': 'steps-pre',\n",
" 'label': mod_names[im],\n",
" })\n",
"\n",
" _ = simplePlot(d, figsize=(10, 10), xrange=(-12, 510),\n",
" x_label='Memory Cell ID',\n",
" y_label=label,\n",
" use_axis=ax,\n",
" title_position=[0.5, 1.18],\n",
" legend='outside-top-ncol6-frame', legend_size='18%',\n",
" legend_pad=0.00)\n",
"\n",
" if const_name != 'BadPixelsDark':\n",
" ax = fig.add_subplot(122)\n",
" label = '## $\\sigma$ {} [ADU], good pixels only'.format(const_name)\n",
" d = []\n",
" for im, mod in enumerate(np.nanstd(data, axis=(1, 2))):\n",
" d.append({'x': np.arange(mod.shape[0]),\n",
" 'y': mod,\n",
" 'drawstyle': 'steps-pre',\n",
" 'label': mod_names[im],\n",
" })\n",
"\n",
" _ = simplePlot(d, figsize=(10, 10), xrange=(-12, 510),\n",
" x_label='Memory Cell ID',\n",
" y_label=label,\n",
" use_axis=ax,\n",
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
" title_position=[0.5, 1.18],\n",
" legend='outside-top-ncol6-frame', legend_size='18%',\n",
" legend_pad=0.00)\n",
"\n",
" plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary tables across Modules ##\n",
"\n",
"Tables show values averaged across all pixels and memory cells of a given detector module."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"if u'$' in tabulate.LATEX_ESCAPE_RULES:\n",
" del(tabulate.LATEX_ESCAPE_RULES[u'$'])\n",
" \n",
"if u'\\\\' in tabulate.LATEX_ESCAPE_RULES:\n",
" del(tabulate.LATEX_ESCAPE_RULES[u'\\\\'])"
]
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
"source": [
"head = ['Module', 'High gain', 'Medium gain', 'Low gain']\n",
"head_th = ['Module', 'High threshold', 'Medium threshold']\n",
"for const_name, const in constants.items():\n",
" table = []\n",
"\n",
" for i_mod, mod in enumerate(mod_names):\n",
"\n",
" t_line = [mod]\n",
" for gain in range(3):\n",
" \n",
" if const_name == 'ThresholdsDark': \n",
" if gain == 2:\n",
" continue\n",
" header = head_th\n",
" else:\n",
" header = head\n",
"\n",
" data = np.copy(const[i_mod, :, :, :, gain])\n",
" if const_name == 'BadPixelsDark':\n",
" data[data > 0] = 1.0\n",
" datasum = np.nansum(data)\n",
" datamean = np.nanmean(data)\n",
" if datamean == 1.0:\n",
" datamean = np.nan\n",
" datasum = np.nan\n",
"\n",
" t_line.append(f'{datasum:6.0f} ({datamean:6.3f}) ')\n",
" label = '## Number(fraction) of bad pixels'\n",
" else:\n",
"\n",
" data[constants['BadPixelsDark']\n",
" [i_mod, :, :, :, gain] > 0] = np.nan\n",
"\n",
" t_line.append(f'{np.nanmean(data):6.1f} $\\\\pm$ {np.nanstd(data):6.1f}')\n",
" label = f'## Average {const_name} [ADU], good pixels only'\n",
"\n",
" table.append(t_line)\n",
"\n",
" display(Markdown(label))\n",
" md = display(Latex(tabulate.tabulate(\n",
" table, tablefmt='latex', headers=header)))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}