@@ -5,21 +5,6 @@ The following notebooks are currently integrated into European XFEL Offline Cali
## AGIPD
### Combine Constants
This notebook combines constants from various evaluations
- Dark image analysis, yielding offset and noise
- Flat field analysis, yielding X-ray gain
- Pulse capacitor analysis, yielding medium gain stage slopes and
thresholds
- Charge injection analysis, yielding low gain stage slopes and thresholds into a single set of calibration constants.
These constants do not include offset and noise as they need to be reevaluated more frequently.
Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages.
### AGIPD Correction
Offline Correction for AGIPD Detector
...
...
@@ -31,16 +16,6 @@ This notebook analyzes a set of dark images taken with AGIPD detector to deduce
The evaluated calibration constants are stored locally and injected in the calibration database.
### Gain Characterization
This notebook is used to produce AGIPD Flat-Field constants.
### Histogramming of AGIPD FF data
!!! warning
TODO: Add description for this notebook
### Characterize AGIPD Pulse Capacitor Data
...
...
@@ -63,6 +38,31 @@ The same regions are present in the gain-bit data and are used to deduce the swi
The resulting slopes are then fitted with a linear function and a combination of a linear and exponential decay function to determine the relative gains of the pixels with respect to the module. Additionally, we deduce masks for bad pixels form the data.
### Gain Characterization
This notebook is used to produce AGIPD Flat-Field constants.
### Histogramming of AGIPD FF data
??? warning "No description"
TODO: Add description for this notebook
### Combine Constants
This notebook combines constants from various evaluations
- Dark image analysis, yielding offset and noise
- Flat field analysis, yielding X-ray gain
- Pulse capacitor analysis, yielding medium gain stage slopes and
thresholds
- Charge injection analysis, yielding low gain stage slopes and thresholds into a single set of calibration constants.
These constants do not include offset and noise as they need to be reevaluated more frequently.
Additionally, a bad pixel mask for all gain stages is deduced from the input. The mask contains dedicated entries for all pixels and memory cells as well as all three gains stages.
The sequence of correction applied are: Offset –> Common Mode Noise –> Relative Gain –> Charge Sharing –> Absolute Gain.
Offset, common mode and gain corrected data is saved to
/data/image/pixels in the CORR files.
`/data/image/pixels` in the CORR files.
If pattern classification is applied (charge sharing correction), this data will be saved to /data/image/pixels_classified, while the
corresponding patterns will be saved to /data/image/patterns in the CORR
files.
If pattern classification is applied (charge sharing correction), this data will be saved to /data/image/pixels_classified, while the corresponding patterns will be saved to /data/image/patterns in the CORR files.
### ePix100 Dark Characterization
...
...
@@ -105,51 +105,13 @@ Noise and bad pixels maps are calculated independently for each of the 4 ASICs o
Common mode correction can be applied to increase sensitivity to noise related bad pixels. Common mode correction is achieved by subtracting the median of all pixels that are read out at the same time along a row/column. This is done in an iterative process, by which a new bad pixels map is calculated and used to mask data as the common mode values
across the rows/columns is updated.
Resulting maps are saved as `.h5` files for a later use and injected to calibration DB.
Resulting maps are saved as HDF5 files for a later use and injected to calibration DB.
The following notebook provides Offset correction of images acquired with the ePix10K detector.
### ePix10K Dark Characterization
The following notebook provides dark image analysis of the ePix10K detector.
Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as `.h5` files for a latter use and injected to the calibration DB.
## FASTCCD
### FastCCD Data Correction
The following notebook provides correction of images acquired with the FastCCD.
### FastCCD Dark Characterization
The following notebook provides dark image analysis of the FastCCD detector.
Dark characterization evaluates offset and noise of the FastCCD
detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database.
## GENERIC
### Constants from DB to HDF5
Currently, available instances are LPD1M1 and AGIPD1M1
### Statistical analysis of calibration factors
Plot calibration constants retrieved from the cal. DB.
To be visualized, calibration constants are averaged per group of pixels. Plot shows calibration constant over time for each constant.
Offline Correction for Jungfrau detector. For single and burst mode and for detector operated in adaptive or fixed gain.
This notebook expects 4 different calibration constants. `Offset`, `BadPixelsDark`, `RelativeGain`, and `BadPixelsFF`. The raw data is by default offset subtracted then divided by the gain parameter. Both bad pixels parameters are stored as a masked array `data/mask` along with the corrected data (same data path as RAW `data/adc`).
The notebook accounts for the raw gain values (0[00], 1[01], 3[11]) during correction. But the raw gain values are stored exactly the same in the corrected files.
- Save reduced ROIs
This notebook supports writing reduce ROI data into the corrected files. For example FXE uses a spectrometer which sends a spectrum onto a set region of the detector. This can be selected and reduced to 1D array, along with a comparable background region. This reduced data can be read rather than the full images.
- Correct strixel sensors.
This notebook is able to offline correct strixel JUNGFRAU as well. Using a cython function a the sensor is decoded and pixels are reordered for offline correction.
* The sensor instead has rectangular pixels: 25 um x 225 um, i.e. 1/3 of the pixel pitch along the x-axis and three times along the y-axis (to maintain the total number of pixel unchanged);
* The readout however is not 'aware' of these changes, so it treats it like a 'normal' JUNGFRAU module, hence the output must be re-shuffled (in the row and column dimensions) in order for the image to make sense.
### Jungfrau Dark Image Characterization
Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad pixel maps
Analyzes Jungfrau dark image data to deduce offset, noise and bad pixel maps from three dark runs of each gain. For data of single or burst mode and detector operated in adaptive or fixed gain.
`Offset` and `Noise` calibration parameters are computed using mean and standard deviation across pixels per memory cells and gain, respectively.
`BadPixelsDark` calibration parameter consists of pixels with wrong gain values, empty cell images (cells with 0 pixel values), pixels for offset and noise maps evaluated with values above bad pixel threshold sigmas, and infinite values in offset or noise maps.
This notebook applies Offset correction then gain correction by default on LPD1M.
Additionally, bad-pixels are masked and stored in a mask array along with the corrected data.
The notebook expects 6 different constants. `Offset`, `RelativeGain`, `FFMap`, `GainAmpMap` and two badpixel maps (`BadPixelsDark` and `BadPixelsFF`)
Offset –> RelativeGain * FFMap * GainAmpMap.
TODO: Add description for this notebook
### LPD Offset, Noise and Dead Pixels Characterization
This notebook performs re-characterize of dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell.
This notebook process dark images to derive offset, noise and bad-pixel maps. All three types of constants are evaluated per-pixel and per-memory cell.
The notebook will correctly handle veto settings, but note that if you veto cells you will not be able to use these offsets for runs with different veto settings - vetoed cells will have zero offset.
...
...
@@ -212,10 +204,10 @@ The evaluated calibration constants are stored locally and injected in the calib
### LPD Radial X-ray Gain Evaluation
Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce :math:`1/r` effects.
Taking proper flat field for LPD can be difficult, as air scattering will always be present. Additionally, the large detector mandates a large distance to the source, in order to reduce $1/r$ effects.
Because of this a radial evaluation method is used, which assumes that pixels are the same radial distance :math:`r` should on average have the
same signal :math:`S(r)`.
Because of this a radial evaluation method is used, which assumes that pixels are the same radial distance $r$ should on average have the
same signal $S(r)$.
#### Injecting calibration constant data to the database
...
...
@@ -236,11 +228,7 @@ The following code characterizes the gain of the LPD detector from charge inject
The data is then analyzed by calculating the per-pixel, per memory cell mean of the samples for each setting. These means are then normalized to the median peak position of all means of the first module. Overlapping settings in neighboring gain ranges are used to deduce the slopes of the different gains with respect to the high gain setting.
@@ -248,22 +236,26 @@ The data is then analyzed by calculating the per-pixel, per memory cell mean of
The following notebook provides offset, common mode, relative gain, split events and pattern classification corrections of images acquired with the pnCCD. This notebook *does not* yet correct for charge transfer inefficiency.
Offset –> Common Mode –> Gain Correction –> Split pattern classification
The notebook stores the offset corrected data at `/data/pixels`, the common mode corrected data at `/data/pixels_cm`, the gain constant used is stored at `/data/gain`, the pattern classification has two arrays stored `/data/pixels_classified` and `/data/patterns`, and the bad pixels are masked and stored at `data/mask`.
### pnCCD Dark Characterization
The following notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.
This notebook provides dark image analysis of the pnCCD detector. Dark characterization evaluates offset and noise of the detector and gives information about bad pixels.
On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps.
Edge pixels are also added to the bad pixels map.
On the first iteration, the offset and noise maps are generated. Initial bad pixels map is obtained based on the offset and initial noise maps. Edge pixels are also added to the bad pixels map.
On the second iteration, the noise map is corrected for common mode. A second bad pixel map is generated based on the offset map and offset-and-common-mode-corrected noise map. Then, the hole in the center of the CCD is added to the second bad pixel map.
On the third and final iteration, the pixels that show up on the above-mentioned bad pixels map are masked. Possible events due to cosmic rays are found and masked. The data are then again offset and common mode corrected and a new final noise and bad pixels maps are generated.
These latter resulting maps together with the offset map are saved as .h5 files to a local path for a later use. These dark constants are not automatically sent to the database.
These latter resulting maps together with the offset map are saved as HDF5 files to a local path for a later use. These dark constants are not automatically sent to the database.
### pnCCD Gain Characterization
The following notebook provides gain characterization for the pnCCD. It relies on data which are previously corrected using the Meta Data Catalog web service interface or by running the Correct_pnCCD_NBC.ipynb notebook. Prior to running this notebook, the corrections which should be applied by the web service or the aforementioned notebook are as follows:
The following notebook provides gain characterization for the pnCCD. It relies on EuXFEL offline corrected data. Prior to running this notebook, the expected applied corrections are:
- Offset correction
- Common mode correction
...
...
@@ -273,15 +265,63 @@ The following notebook provides gain characterization for the pnCCD. It relies o
Offset correction of images acquired with the ePix10K detector.
### ePix10K Dark Characterization
Dark image analysis of the ePix10K detector.
Dark characterization evaluates offset and noise of the detector and gives information about bad pixels. Resulting maps are saved as HDF5 files for a latter use and injected to the calibration DB.
The following notebook provides correction of images acquired with the FastCCD.
### FastCCD Dark Characterization
The following notebook provides dark image analysis of the FastCCD detector.
Dark characterization evaluates offset and noise of the FastCCD
detector, corrects the noise for Common Mode (CM), and defines bad pixels relative to offset and CM corrected noise. Bad pixels are then excluded and CM corrected noise is recalculated excluding the bad pixels. Resulting offset and CM corrected noise maps, as well as the bad pixel map are sent to the calibration database.
