README.rst

Offline Calibration
===================

The offline calibration is a package that consists of different services,
responsible for applying most of the offline calibration and characterization
for the detectors.

Offline calibration installation
================================

It's recommended to install the offline calibration (pycalibration) package
over maxwell, using anaconda/3 environment.

Installation using Anaconda
---------------------------

First you need to load the anaconda/3 environment through::

    1. module load anaconda/3

If installing into other python enviroments, this step can be skipped.

Then the package for the offline calibration can be obtained from the git repository::

    2. git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git


You can then install all requirements of this tool chain in your home directory by running::

    3. pip install -r requirements.txt . --user

in pycalibration's root directory.

After installation, you should make sure that the home directory is in the PATH environment variable::

    4. export PATH=$HOME/.local/bin:$PATH

Installation using virtual python environment
---------------------------------------------

Create virtual environment::

    module load anaconda/3
    python -m venv /path/to/new/virtual/environment
    source /path/to/new/virtual/environment/bin/activate

Clone from git::

    cd /path/to/packages
    git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
    cd pycalibration

Install the package::

    pip install -r requirements.txt

In additional install pyDetLib package, which is required for many notebooks::

    cd /path/to/packages
    git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git
    cd pyDetLib/lib
    pip install -r requirements.txt
    pip install .

++++++++++++++++++++++++++++++++++++++++++++++++++
Setting an ipython kernel for virtual environments
++++++++++++++++++++++++++++++++++++++++++++++++++

To set a kernel for your virtual environment::

    source /path/to/new/virtual/environment/bin/activate
    pip install ipykernel
    python -m ipykernel install --user --name <virtenv-name> --display-name "virtenv-display-name"

This can be useful for Jupyter notebook tools as "max-jhub.desy.de".

Development Installation
------------------------

For a development installation, which automatically
picks up (most) changes, first install the dependencies as above,
but then install the tool-chain separately in development mode (install in home directory using --user, in case of using Anaconda/3)::

   pip install -e .


Activate Offline calibration
============================

For using pycalibration package one needs to activate it through::

    source activate

from inside of the pycalibration directory. This will automatically load 
all needed modules and export the $PATH for the home directory.


Python Scripted Calibration
===========================

First: do not run this on the Maxwell gateway. Rather, `salloc`
a node for yourself first::

   salloc -p exfel/upex -t 01:00:00

where `-p` gives the partition to use: exfel or upex and `-t`
the duration the node should be allocated. Then `ssh` onto 
that node.

(optionally) Set up the environment::

   module load python3
   pip install --user ipython --upgrade
   pip install --user ipyparallel --upgrade
   pip install --user dill
   
If running headless (i.e. without X forwarding), be sure to set 
`MPLBACKEND=Agg`, via::

   export MPLBACKEND=Agg

Then start an `ipcluster`. If you followed the steps above this can be done
via::

   ~/.local/bin/ipcluster start --n=32


Run the script::

    python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
       --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1   

Here `--input` should point to a directory of `RAW` files for the detector you
are calibrating. They will be output into the folder specified by `--output`, 
which will have the run number or the last folder in the hiearchy of the input
appended. Additionally, you need to specify the number of `--mem-cells` used
for the run, as well as the `--detector`. Finally, you can optionally 
specify to only process certain `--sequences` of files, matching the sequence
numbers of the `RAW` input. These should be given as a comma-separated list.

Finally, there is a `--no-relgain` option, which disables relative gain 
correction. This can be useful while we still further characterize the detectors
to provid accurate relative gain correction constants.

You'll get a series of plots in the output directory as well.