###################
Offline Calibration
###################

The offline calibration is a package that consists of different services,
responsible for applying most of the offline calibration and characterization
for the detectors.

.. contents::


Offline Calibration Installation
********************************

It's recommended to install the offline calibration (pycalibration) package on
maxwell, using the anaconda/3 environment.

The following instructions clone from the EuXFEL GitLab instance using SSH
remote URLs, this assumes that you have set up SSH keys for use with GitLab
already. If you have not then read the appendix section on `SSH Key Setup for
GitLab`_ for instructions on how to do this .


Installation using python virtual environment - recommended
===========================================================

1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
2. ``module load anaconda/3`` - load the anaconda/3 environment
3. ``python3 -m venv .venv`` - create the virtual environment
4. ``source .venv/bin/activate`` - activate the virtual environment
5. ``python3 -m pip install --upgrade pip`` - upgrade version of pip
6. ``python3 -m pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)

Copy/paste script:

.. code::

  git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
  cd pycalibration
  module load anaconda/3
  python3 -m venv .venv
  source .venv/bin/activate
  python3 -m pip install --upgrade pip
  python3 -m pip install .  # `-e` flag for editable install, e.g. `pip install -e .`


Installation into user home directory
=====================================

This is not recommended as `pycalibration` has pinned dependencies for
stability, if you install it directly into you users home environment then it
will down/upgrade your local packages, which may cause major issues and may
**break your local environment**, it is highly recommended to use the venv
installation method instead.

1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
2. ``module load anaconda/3`` - load the anaconda/3 environment. If installing into other python environments, this step can be skipped
3. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
4. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable

Copy/paste script:

.. code::

  git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
  cd pycalibration
  module load anaconda/3
  pip install --user .  # `-e` flag for editable install, e.g. `pip install -e .`
  export PATH=$HOME/.local/bin:$PATH


Creating an ipython kernel for virtual environments
===================================================

To create an ipython kernel with pycalibration available you should (if using a
venv) activate the virtual environment first, and then run:

.. code::

  python3 -m pip install ipykernel  # If not using a venv add `--user` flag
  python3 -m ipykernel install --user --name pycalibration --display-name "pycalibration"  # If not using a venv pick different name

This can be useful for Jupyter notebook tools as https://max-jhub.desy.de/hub/login


Contributing
************


Guidelines
==========

Development guidelines can be found on the GitLab Wiki page here: https://git.xfel.eu/gitlab/detectors/pycalibration/wikis/GitLab-Guidelines


Basics
======

If you are installing the package for development purposes then you should
install the optional dependencies as well. Follow the instructions as above, but
instead of ``pip install .`` use ``pip install ".[test,dev]"`` to install both
the extras.

The installation instructions above assume that you have set up SSH keys for use
with GitLab to allow for passwordless clones from GitLab, this way it's possible
to run ``pip install git+ssh...`` commands and install packages directly from
GitLab.

To do this check the settings page here: https://git.xfel.eu/gitlab/profile/keys


Pre-Commit Hooks
================

This repository uses pre-commit hooks automatically run some code quality and
standard checks, this includes the following:

a. ``identity`` - The 'identity' meta hook prints off a list of files that the hooks will execute on
b. 'Standard' file checks

   1. ``check-added-large-files`` - Ensures no large files are committed to repo
   2. ``check-ast`` - Checks that the python AST is parseable
   3. ``check-json`` - Checks json file formatting is parseable
   4. ``check-yaml`` - Checks yaml file formatting is parseable
   5. ``check-toml`` - Checks toml file formatting is parseable
   6. ``rstcheck`` - Checks rst file formatting is parseable
   7. ``end-of-file-fixer`` - Fixes EoF to be consistent
   8. ``trailing-whitespace`` - Removes trailing whitespaces from lines
   9. ``check-merge-conflict`` - Checks no merge conflicts remain in the commit
   10. ``mixed-line-ending`` - Fixes mixed line endings

c. Code checks

   1. ``flake8`` - Code style checks
   2. ``isort`` - Sorts imports in python files
   3. ``check-docstring-first`` - Ensures docstrings are in the correct place

d. Notebook checks

   1. ``nbqa-flake8`` - Runs flake8 on notebook cells
   2. ``nbqa-isort`` - Runs isort on notebook cells
   3. ``nbstripoutput`` - Strips output from ipynb files

To install these checks, set up you environment as mentioned above and then run
the command:

.. code::

  pre-commit install-hooks

This will set up the hooks in git locally, so that each time you run the command
``git commit`` the hooks get executed on the **staged files only**, beware that
if the pre-commit hooks find required changes some of them will **modify your
files**, however they only modify the current working files, not the ones you
have already staged. This means that you can look at the diff between your
staged files and the ones that were modified to see what changes are suggested.


Run Checks Only On Diffs
------------------------

Typically ``pre-commit`` is ran on ``--all-files`` within a CI, however as this
is being set up on an existing codebase these checks will always fail with a
substantial number of issues. Using some creative workarounds, the CI has been
set up to only run on files which have changed between a PR and the target
branch.

If you want to run the pre-commit checks as they would run on the CI, then you
can use the ``bin/pre-commit-diff.sh`` to execute the checks as on the CI
pipeline.

A side effect of this is that the checks will run on **all** of the differences
between the 'local' and target branch. This means that if changes have recently
been merged into the target branch, and there is divergence between the two,
then the tests will run on all the differences.

If this happens and the hooks in the CI (or via the script) run on the wrong
files then you should **rebase onto the target branch** to prevent the checks
from running on the wrong files/diffs.


Skipping Checks
---------------

If the checks are failing and you want to ignore them on purpose then you have two options:

- use the ``--no-verify`` flag on your ``git commit`` command to skip them, e.g. ``git commit -m "Commit skipping hooks" --no-verify``
- use the variable ``SKIP=hooks,to,skip`` before the git commit command to list hooks to skip, e.g. ``SKIP=flake8,isort git commit -m "Commit skipping only flake8 and isort hooks"``

In the CI pipeline the pre-commit check stage has ``allow_failure: true`` set so
that it is possible to ignore errors in the checks, and so that subsequent
stages will still run even if the checks have failed. However there should be a
good reason for allowing the checks to fail, e.g. checks failing due to
unmodified sections of code being looked at.


Python Scripted Calibration
***************************

**Do not run this on the Maxwell gateway**. Rather, ``salloc`` a node for
yourself first:

.. code::

  salloc -p exfel/upex -t 01:00:00

where `-p` gives the partition to use: exfel **or** upex and `-t` the duration
the node should be allocated. Then ``ssh`` onto that node.

Then activate your environment as described above (or just continue if you are
not using a venv).

If running headless (i.e. without X forwarding), be sure to set
``MPLBACKEND=Agg``, via:

.. code::

  export MPLBACKEND=Agg

Then start an ``ipcluster``. If you followed the steps above this can be done
via:

.. code::

  ipcluster start --n=32


Finally run the script:

.. code::

  python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
    --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1

Here ``--input`` should point to a directory of ``RAW`` files for the detector
you are calibrating. They will be output into the folder specified by
``--output``, which will have the run number or the last folder in the hierarchy
of the input appended. Additionally, you need to specify the number of
``--mem-cells`` used for the run, as well as the ``--detector``. Finally, you
can optionally specify to only process certain ``--sequences`` of files,
matching the sequence numbers of the `RAW` input. These should be given as a
comma-separated list.

Finally, there is a ``--no-relgain`` option, which disables relative gain
correction. This can be useful while we still further characterize the detectors
to provide accurate relative gain correction constants.

You'll get a series of plots in the output directory as well.


Appendix
********

Important information that doesn't really fit in as part of the readme.

TODO: Place this into the docs? Also, improve docs (out of scope for PR !437)


SSH Key Setup for GitLab
========================

It is highly recommended to set up SSH keys for access to GitLab as this
simplifies the setup process for all of our internal software present on GitLab.

To set up the keys:

1. Connect to Maxwell
2. Generate a new keypair with ``ssh-keygen -o -a 100 -t ed25519``, you can
   either leave this in the default location (``~/.ssh/id_ed25519``) or place it
   into a separate directory to make management of keys easier if you already
   have multiple ones. If you are using a password for your keys please check
   this page to learn how to manage them: https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent
3. Add the public key (``id_ed25519.pub``) to your account on GitLab: https://git.xfel.eu/gitlab/profile/keys
4. Add the following to your ``~/.ssh/config`` file

.. code::

  # Special flags for gitlab over SSH
  Host git.xfel.eu
      User git
      Port 10022
      ForwardX11 no
      IdentityFile ~/.ssh/gitlab/id_ed25519

Once this is done you can clone repositories you have access to from GitLab
without having to enter your password each time. As ``pycalibration``
requirements are installed from SSH remote URLs having SSH keys set up is a
requirement for installing pycalibration.


GitLab Access for ``xcaltst`` and ``xcal``
==========================================

To make it easier to work with and deploy software via ``xcaltst``/``xcal``, we
have created an xcal account for gitlab with the following details:

- Full Name: ReadOnly Gitlab Calibration External
- User ID: 423
- Username: ``xcalgitlab``
- Password: ask Robert Rosca

This account is intended to be used as a read only account which can be given
access to certain repos to make it easier to clone them when using our
functional accounts on Maxwell.

The ``xcaltst`` account has an ed25519 keypair under ``~/.ssh/gitlab/``, the
public key has been added to the ``xcalgitlab``'s approved SSH keys.

Additionally this block has been added to ``~/.ssh/config``:

.. code::

  # Special flags for gitlab over SSH
  Host git.xfel.eu
      User git
      Port 10022
      ForwardX11 no
      IdentityFile ~/.ssh/gitlab/id_ed25519

Now any repository that ``xcalgitlab`` has read access to, e.g. if it is added as
a reporter, can be cloned on Maxwell without having to enter a password.

For example, ``xcalgitlab`` is a reporter on the pycalibration
https://git.xfel.eu/gitlab/detectors/pycalibration repository, so now
``xcalgitlab`` can do passwordless clones with SSH:

.. code::

  [xcaltst@max-exfl017 tmp]$ git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
  Cloning into 'pycalibration'...
  remote: Enumerating objects: 9414, done.
  remote: Counting objects: 100% (9414/9414), done.
  remote: Compressing objects: 100% (2858/2858), done.
  remote: Total 9414 (delta 6510), reused 9408 (delta 6504)
  Receiving objects: 100% (9414/9414), 611.81 MiB | 54.87 MiB/s, done.
  Resolving deltas: 100% (6510/6510), done.

References:

- Redmine ticket: https://in.xfel.eu/redmine/issues/83954
- Original issue: https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/121