Offline Calibration

The offline calibration is a package that consists of different services, responsible for applying most of the offline calibration and characterization for the detectors.

Offline calibration installation

It's recommended to install the offline calibration (pycalibration) package over maxwell, using anaconda/3 environment.

Installation using Anaconda

First you need to load the anaconda/3 environment through:

1. module load anaconda/3

If installing into other python enviroments, this step can be skipped.

Then the package for the offline calibration can be obtained from the git repository:

2. git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git

You can then install all requirements of this tool chain in your home directory by running:

3. pip install -r requirements.txt . --user

in pycalibration's root directory.

After installation, you should make sure that the home directory is in the PATH environment variable:

4. export PATH=$HOME/.local/bin:$PATH

Installation using virtual python environment

Create virtual environment:

module load anaconda/3
python -m venv /path/to/new/virtual/environment
source /path/to/new/virtual/environment/bin/activate

Clone from git:

cd /path/to/packages
git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
cd pycalibration

Edit path to the python environment in the bin/activate.sh file. Change:

export PATH=/home/${USER}/.local/bin:$PATH

to:

export PATH=/path/to/new/virtual/environment/bin:$PATH

Install the package:

pip install -r requirements.txt

In additional install pyDetLib package, which is required for many notebooks:

cd /path/to/packages
git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git
cd pyDetLib/lib
pip install -r requirements.txt
pip install .

Setting an ipython kernel for virtual environments

To set a kernel for your virtual environment:

source /path/to/new/virtual/environment/bin/activate
pip install ipykernel
python -m ipykernel install --user --name <virtenv-name> --display-name "virtenv-display-name"

This can be useful for Jupyter notebook tools as "max-jhub.desy.de".

Development Installation

For a development installation, which automatically picks up (most) changes, first install the dependencies as above, but then install the tool-chain separately in development mode (install in home directory using --user, in case of using Anaconda/3):

pip install -e .

Activate Offline calibration

For using pycalibration package one needs to activate it through:

source activate

from inside of the pycalibration directory. This will automatically load all needed modules and export the $PATH for the home directory.

Python Scripted Calibration

First: do not run this on the Maxwell gateway. Rather, salloc a node for yourself first:

salloc -p exfel/upex -t 01:00:00

where -p gives the partition to use: exfel or upex and -t the duration the node should be allocated. Then ssh onto that node.

(optionally) Set up the environment:

module load python3
pip install --user ipython --upgrade
pip install --user ipyparallel --upgrade
pip install --user dill

If running headless (i.e. without X forwarding), be sure to set MPLBACKEND=Agg, via:

export MPLBACKEND=Agg

Then start an ipcluster. If you followed the steps above this can be done via:

~/.local/bin/ipcluster start --n=32

Run the script:

python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
   --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1

Here --input should point to a directory of RAW files for the detector you are calibrating. They will be output into the folder specified by --output, which will have the run number or the last folder in the hiearchy of the input appended. Additionally, you need to specify the number of --mem-cells used for the run, as well as the --detector. Finally, you can optionally specify to only process certain --sequences of files, matching the sequence numbers of the RAW input. These should be given as a comma-separated list.

Finally, there is a --no-relgain option, which disables relative gain correction. This can be useful while we still further characterize the detectors to provid accurate relative gain correction constants.

You'll get a series of plots in the output directory as well.