README.rst

Offline Calibration

The offline calibration is a package that consists of different services, responsible for applying most of the offline calibration and characterization for the detectors.

Contents

• Offline Calibration Installation
  • Installation using python virtual environment - recommended
  • Installation into user home directory
  • Creating an ipython kernel for virtual environments
• Contributing
  • Guidelines
  • Basics
  • Pre-Commit Hooks
    • Run Checks Only On Diffs
    • Skipping Checks
• Python Scripted Calibration

Offline Calibration Installation

It's recommended to install the offline calibration (pycalibration) package over maxwell, using anaconda/3 environment.

Installation using python virtual environment - recommended

1. git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration - clone the offline calibration package from EuXFEL GitLab
2. module load anaconda/3 - load the anaconda/3 environment
3. python3 -m venv .venv - create the virtual environment
4. source .venv/bin/activate - activate the virtual environment
5. python3 -m pip install --upgrade pip - upgrade version of pip
6. python3 -m pip install -r requirements.txt - install dependencies
7. python3 -m pip install . - install the pycalibration package (add -e flag for editable development installation)
8. pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib"

Copy/paste script:

git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
module load anaconda/3
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
python3 -m pip install .  # `-e` flag for editable install
python3 -m pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib/"

Installation into user home directory

1. git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration - clone the offline calibration package from EuXFEL GitLab
2. module load anaconda/3 - load the anaconda/3 environment. If installing into other python environments, this step can be skipped
3. pip install -r requirements.txt - install all requirements of this tool chain in your home directory
4. pip install . - install the pycalibration package (add -e flag for editable development installation)
5. export PATH=$HOME/.local/bin:$PATH - make sure that the home directory is in the PATH environment variable

Copy/paste script:

git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
module load anaconda/3
pip install -r requirements.txt --user
pip install .  # `-e` flag for editable install, e.g. `pip install -e .`
export PATH=$HOME/.local/bin:$PATH

Creating an ipython kernel for virtual environments

To create an ipython kernel with pycalibration available you should (if using a venv) activate the virtual environment first, and then run:

python3 -m pip install ipykernel  # If not using a venv add `--user` flag
python3 -m ipykernel install --user --name pycalibration --display-name "pycalibration"  # If not using a venv pick different name

This can be useful for Jupyter notebook tools as https://max-jhub.desy.de/hub/login

Contributing

Guidelines

Development guidelines can be found on the GitLab Wiki page here: https://git.xfel.eu/gitlab/detectors/pycalibration/wikis/GitLab-Guidelines

Basics

The installation instructions above assume that you have set up SSH keys for use with GitLab to allow for passwordless clones from GitLab, this way it's possible to run pip install git+ssh... commands and install packages directly from GitLab.

To do this check the settings page here: https://git.xfel.eu/gitlab/profile/keys

Pre-Commit Hooks

This repository uses pre-commit hooks automatically run some code quality and standard checks, this includes the following:

1. identity - The 'identity' meta hook prints off a list of files that the hooks will execute on
2. 'Standard' file checks
   1. check-added-large-files - Ensures no large files are committed to repo
   2. check-ast - Checks that the python AST is parseable
   3. check-json - Checks json file formatting is parseable
   4. check-yaml - Checks yaml file formatting is parseable
   5. check-toml - Checks toml file formatting is parseable
   6. rstcheck - Checks rst file formatting is parseable
   7. end-of-file-fixer - Fixes EoF to be consistent
   8. trailing-whitespace - Removes trailing whitespaces from lines
   9. check-merge-conflict - Checks no merge conflicts remain in the commit
   10. mixed-line-ending - Fixes mixed line endings
3. Code checks
   1. flake8 - Code style checks
   2. isort - Sorts imports in python files
   3. check-docstring-first - Ensures docstrings are in the correct place
4. Notebook checks
   1. nbqa-flake8 - Runs flake8 on notebook cells
   2. nbqa-isort - Runs isort on notebook cells
   3. nbstripoutput - Strips output from ipynb files

To install these checks, set up you environment as mentioned above and then run the command:

pre-commit install-hooks

This will set up the hooks in git locally, so that each time you run the command git commit the hooks get executed on the staged files only, beware that if the pre-commit hooks find required changes some of them will modify your files, however they only modify the current working files, not the ones you have already staged. This means that you can look at the diff between your staged files and the ones that were modified to see what changes are suggested.

Run Checks Only On Diffs

Typically pre-commit is ran on --all-files within a CI, however as this is being set up on an existing codebase these checks will always fail with a substantial number of issues. Using some creative workarounds, the CI has been set up to only run on files which have changed between a PR and the target branch.

If you want to run the pre-commit checks as they would run on the CI, then you can use the bin/pre-commit-diff.sh to execute the checks as on the CI pipeline.

A side effect of this is that the checks will run on all of the differences between the 'local' and target branch. This means that if changes have recently been merged into the target branch, and there is divergence between the two, then the tests will run on all the differences.

If this happens and the hooks in the CI (or via the script) run on the wrong files then you should rebase onto the target branch to prevent the checks from running on the wrong files/diffs.

Skipping Checks

If the checks are failing and you want to ignore them on purpose then you have two options:

• use the --no-verify flag on your git commit command to skip them, e.g. git commit -m "Commit skipping hooks" --no-verify
• use the variable SKIP=hooks,to,skip before the git commit command to list hooks to skip, e.g. SKIP=flake8,isort git commit -m "Commit skipping only flake8 and isort hooks"

In the CI pipeline the pre-commit check stage has allow_failure: true set so that it is possible to ignore errors in the checks, and so that subsequent stages will still run even if the checks have failed. However there should be a good reason for allowing the checks to fail, e.g. checks failing due to unmodified sections of code being looked at.

Python Scripted Calibration

Do not run this on the Maxwell gateway. Rather, salloc a node for yourself first:

salloc -p exfel/upex -t 01:00:00

where -p gives the partition to use: exfel or upex and -t the duration the node should be allocated. Then ssh onto that node.

Then activate your environment as described above (or just continue if you are not using a venv).

If running headless (i.e. without X forwarding), be sure to set MPLBACKEND=Agg, via:

export MPLBACKEND=Agg

Then start an ipcluster. If you followed the steps above this can be done via:

ipcluster start --n=32

Finally run the script:

python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
  --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1

Here --input should point to a directory of RAW files for the detector you are calibrating. They will be output into the folder specified by --output, which will have the run number or the last folder in the hierarchy of the input appended. Additionally, you need to specify the number of --mem-cells used for the run, as well as the --detector. Finally, you can optionally specify to only process certain --sequences of files, matching the sequence numbers of the RAW input. These should be given as a comma-separated list.

Finally, there is a --no-relgain option, which disables relative gain correction. This can be useful while we still further characterize the detectors to provide accurate relative gain correction constants.

You'll get a series of plots in the output directory as well.