README.rst

Offline Calibration

The offline calibration is a package that consists of different services, responsible for applying most of the offline calibration and characterization for the detectors.

Contents

• Offline Calibration Installation
  • Installation using python virtual environment - recommended
  • Installation into user home directory
  • Creating an ipython kernel for virtual environments
• Offline Calibration Configuration
  • Examples
  • Checking Configurations
• Contributing
  • Guidelines
  • Basics
  • Pre-Commit Hooks
    • Run Checks Only On Diffs
    • Skipping Checks
• Python Scripted Calibration
  • Reproducing calibration
• Appendix
  • SSH Key Setup for GitLab
  • GitLab Access for xcaltst and xcal

Offline Calibration Installation

It's recommended to install the offline calibration (pycalibration) package on maxwell, using the anaconda/3 environment.

The following instructions clone from the EuXFEL GitLab instance using SSH remote URLs, this assumes that you have set up SSH keys for use with GitLab already. If you have not then read the appendix section on SSH Key Setup for GitLab for instructions on how to do this .

Installation using python virtual environment - recommended

pycalibration uses the same version of Python as Karabo, which in June 2021 updated to use Python 3.8. Currently the default python installation on Maxwell is still Python 3.6.8, so Python 3.8 needs to be loaded from a different location.

One option is to use the Maxwell Spack installation, running module load maxwell will activate the test Spack instance from DESY, then you can use module load python-3.8.6-gcc-10.2.0-622qtxd to Python 3.8. Note that this Spack instance is currently a trial phase and may not be stable.

Another option is to use pyenv, we provide a pyenv installation at /gpfs/exfel/sw/calsoft/.pyenv which we use to manage different versions of python. This can be activated with source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate

A quick setup would be:

1. source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
2. git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration - clone the offline calibration package from EuXFEL GitLab
3. pyenv shell 3.8.11 - load required version of python
4. python3 -m venv .venv - create the virtual environment
5. source .venv/bin/activate - activate the virtual environment
6. python3 -m pip install --upgrade pip - upgrade version of pip
7. python3 -m pip install . - install the pycalibration package (add -e flag for editable development installation)

Copy/paste script:

source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
pyenv shell 3.8.11
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install .  # `-e` flag for editable install, e.g. `pip install -e .`

Installation into user home directory

This is not recommended as pycalibration has pinned dependencies for stability, if you install it directly into you users home environment then it will downgrade/upgrade your local packages, which may cause major issues and may break your local environment, it is highly recommended to use the venv installation method instead.

1. source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
2. git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration - clone the offline calibration package from EuXFEL GitLab
3. pyenv shell 3.8.11 - load required version of python
4. pip install . - install the pycalibration package (add -e flag for editable development installation)
5. export PATH=$HOME/.local/bin:$PATH - make sure that the home directory is in the PATH environment variable

Copy/paste script:

source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
pyenv shell 3.8.11
cd pycalibration
pip install --user .  # `-e` flag for editable install, e.g. `pip install -e .`
export PATH=$HOME/.local/bin:$PATH

Creating an ipython kernel for virtual environments

To create an ipython kernel with pycalibration available you should (if using a venv) activate the virtual environment first, and then run:

python3 -m pip install ipykernel  # If not using a venv add `--user` flag
python3 -m ipykernel install --user --name pycalibration --display-name "pycalibration"  # If not using a venv pick different name

This can be useful for Jupyter notebook tools as https://max-jhub.desy.de/hub/login

Offline Calibration Configuration

The offline calibration package is configured with three configuration files:

• webservice/config/webservice.yaml - configuration for the web service
• webservice/config/serve_overview.yaml - configuration for the overview page
• src/cal_tools/restful_config.yaml - configuration for RESTful access to myMDC and CalCat by cal tools

These configuration files should not be modified directly, instead you should create a file $CONFIG.secrets.yaml (e.g. webservice.secrets.yaml) in the configuration directory, and then add any modifications, such as secrets, to this file.

Alternatively, configurations are also searched for in ~/.config/pycalibration/$MODULE/$CONFIG.yaml (e.g. ~/.config/pycalibration/webservice/serve_overview.yaml), which is a useful place to store configurations like secrets so that they are present even if you delete the pycalibration directory, or if you have multiple pycalibration repos checked out, as you no longer need to copy/paste the configurations each time.

Finally, you can use environment variables to override the configuration without modifying any files, which is useful for one-off changes or if you are running tests in a CI environment. The environment variables should be prefixed with:

• webservice/config/webservice.yaml - CAL_WEBSERVICE
• webservice/config/serve_overview.yaml - CAL_SERVE_OVERVIEW
• src/cal_tools/restful_config.yaml - CAL_CAL_TOOLS

Followed by an underscore and the configuration key you wish to change. Nested keys can be accessed with two underscores, e.g. CAL_WEBSERVICE_CONFIG_REPO__URL would modify the `config-repo: url: ` value.

Note that the order of priority is:

• default configuration - e.g. webservice/config/webservice.yaml
• local configuration - e.g. webservice/config/webservice.secrets.yaml
• user configuration - e.g. ~/.config/pycalibration/webservice/webservice.yaml
• environment variables - e.g. export CAL_WEBSERVICE_*=...

Examples

For example, webservice/config/webservice.yaml has:

```yaml config-repo:

url: "@note add this to secrets file" local-path: "@format {env[HOME]}/calibration_config"

... metadata-client:

user-id: "@note add this to secrets file" user-secret: "@note add this to secrets file" user-email: "@note add this to secrets file"

```

So you would create a file webservice/config/webservice.secrets.yaml:

```yaml config-repo:

url: "https://USERNAME:TOKEN@git.xfel.eu/gitlab/detectors/calibration_configurations.git"

metadata-client:

user-id: "id..." user-secret: "secret..." user-email: "calibration@example.com"