Skip to content
Snippets Groups Projects

Improved BOZ analysis

Merged Improved BOZ analysis
boz-improved into master
Merged Loïc Le Guyader requested to merge boz-improved into master
Compare
and
8 files
+ 5046
− 4865
Compare changes
  • Side-by-side
  • Inline
Files
8
doc/BOZ analysis part I parameters determination.ipynb deleted 100644 → 0
+ 0
− 538
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"%matplotlib notebook\n",
"import matplotlib.pyplot as plt\n",
"plt.rcParams['figure.constrained_layout.use'] = True\n",
"\n",
"import dask\n",
"print(f'dask: {dask.__version__}')\n",
"\n",
"from psutil import virtual_memory\n",
"\n",
"mem = virtual_memory()\n",
"print(f'Physical memory: {mem.total/1024/1024/1024:.0f} Gb') # total physical memory available\n",
"\n",
"import logging\n",
"logging.basicConfig(filename='example.log', level=logging.DEBUG)\n",
"\n",
"import toolbox_scs as tb\n",
"print(tb.__file__)\n",
"import toolbox_scs.routines.boz as boz"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Create parameters object"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"tags": [
"parameters"
]
},
"outputs": [],
"source": [
"proposal = 2937\n",
"darkrun = 478\n",
"run = 477\n",
"module = 15\n",
"gain = 0.5\n",
"sat_level = 500\n",
"rois_th = 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params = boz.parameters(proposal=proposal, darkrun=darkrun, run=run, module=module, gain=gain)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from extra_data.read_machinery import find_proposal\n",
"\n",
"root = find_proposal(f'p{proposal:06d}')\n",
"path = root + '/usr/processed_runs/' + f'r{params.run}/'\n",
"print(path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### load data persistently"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.dask_load()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dark run inspection"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The aim is to check dark level and extract bad pixel map."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"dark = boz.average_module(params.arr_dark).compute()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"pedestal = np.mean(dark)\n",
"pedestal"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mean_th = (pedestal-25, pedestal+30)\n",
"f = boz.inspect_dark(params.arr_dark, mean_th=mean_th)\n",
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-dark.png', dpi=300)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.mean_th = mean_th\n",
"params.set_mask(boz.bad_pixel_map(params))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(params)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Histogram"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"h, f = boz.inspect_histogram(params.arr,\n",
" params.arr_dark,\n",
" mask=params.get_mask() #, extra_lines=True\n",
" )\n",
"f.suptitle(f'p:{params.proposal} r:{params.run} d:{params.darkrun}')\n",
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-histogram.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"adding guide to the eye"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ROIs extraction"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.rois_th = rois_th\n",
"params.rois = boz.find_rois_from_params(params)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(params)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"data = boz.average_module(params.arr, dark=dark).compute()\n",
"dataM = data.mean(axis=0) # mean over pulseId"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = boz.inspect_rois(dataM, params.rois, params.rois_th)\n",
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-rois.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Flat field extraction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first step is to compute a good average image, this mean remove saturated shots and ignoring bad pixels"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.sat_level = sat_level\n",
"res = boz.average_module(params.arr, dark=dark,\n",
" ret='mean', mask=params.get_mask(), sat_roi=params.rois['sat'],\n",
" sat_level=params.sat_level)\n",
"avg = res.compute().mean(axis=0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The second step is from that good average image to fit the plane field on n/0 and p/0 rois. We have to make sure that the rois have same width."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = boz.inspect_plane_fitting(avg, params.rois)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-noflatfield.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"fit the plane field correction"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plane = boz.plane_fitting(params)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plane"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"compute the correction and inspect the result of its application"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.set_flat_field(plane.x)\n",
"ff = boz.compute_flat_field_correction(params.rois, params.get_flat_field())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"f = boz.inspect_plane_fitting(avg/ff, params.rois)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-withflatfield.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Non-linearities correction extraction"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To speed up online analysis, we save the corrections with a dummy non-linearity correction. The saved file can be used for online analysis as soon as it appears."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.set_Fnl(np.arange(2**9))\n",
"params.save(path=path)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"N = 80\n",
"domain = boz.nl_domain(N, 40, 511)\n",
"params.alpha = 0.5\n",
"params.max_iter = 25"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## minimizing"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"res, fit_res = boz.nl_fit(params, domain)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.set_Fnl(boz.nl_lut(domain, res.x))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(params)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"f = boz.inspect_correction(params, gain=params.gain)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-correction.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### plotting the fitted correction"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = boz.inspect_Fnl(params.get_Fnl())"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-Fnl.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### plotting the fit progresion"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f = boz.inspect_nl_fit(fit_res)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"f.savefig(path+f'p{params.proposal}-r{params.run}-d{params.darkrun}-inspect-nl-fit.png', dpi=300)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Save the analysis parameters"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"params.save(path=path)"
]
}
],
"metadata": {
"celltoolbar": "Tags",
"kernelspec": {
"display_name": "xfel",
"language": "python",
"name": "xfel"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
},
"nbsphinx": {
"execute": "never"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

Legal Notice | Contacts