Offline Calibration Webservice

The offline calibration webservice interacts with the Metadata Catalogue (MDC), such that migration of data to the offline cluster automatically triggers calibration jobs on relevant files.

Installation

Installation should only be performed by trained personal.

Prerequisites

The service needs to be installed under a functional user account which

• has read permission to the raw data folder on the Maxwell cluster
• has write permission to the proc folders for outputting corrected data
• is allowed to launch SLURM jobs on the cluster

The hosting system needs to be accessible via ZMQ calls from the MDC. This requires appropriate DMZ settings. Additionally, it needs to be able to interact with the MDC via the MDC client interface

Installation of Dependencies

The installation requirements can be found in the requirements.txt file. Additionally, the xfel-calibrate environment needs to be installed:

1. clone the pycalibration repo into a directory of your choice:

   git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git .

2. pick the python environment to install into. On Maxwell the anaconda/3 environment will work:

   module load anaconda/3

3. install the xfel-calibrate environment

   pip install --user -r requirements.txt

4. some correction notebooks require pyDetLib. It requires manual installation in a non-Karabo python environment

   mkdir pydetlib
   cd pydetlib
   git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git .
   pip install --user ./lib/requirements.txt
   pip install --user pycuda
   pip install --user ./lib/
   cd ..
   

5. install the separate requirements for the webservice:

   cd webservice
   pip install --user -r requirements.txt

6. install the metadata_client library, according to instructions at

   https://git.xfel.eu/gitlab/ITDM/metadata_client

You are now good to go.

Configuration

Configuration is done through the webservice.yaml file in the webservice directory. On a new installation you will likely need to change the following parameters.

In the config-repo section, the configuration repository needs to be configured:

config-repo:
    url: https://git.xfel.eu/gitlab/detectors/calibration_configurations.git
    local-path: /home/haufs/calibration_config/

Here you should prepend the url entry with a gitlab access token, that provides access to the calibration_configurations repository.

In the web-service section, the webservice itself is configured:

web-service:
    port: 5555
    bind-to: tcp://*
    allowed-ips: '111.222.222.111', ''111.222.222.112'
    job-db: ./webservice_jobs.sqlite
    job-update-interval: 30
    job-timeout: 3600

In case you want to use authentication, add a list of allowed-ips.

In the metadata-client section, the client interface to the MDC is configured:

metadata-client:
    user-id:
    user-secret:
    user-email:
    metadata-web-app-url: 'https://in.xfel.eu/metadata'
    metadata-web-app-url: 'https://in.xfel.eu/metadata'
    token-url: 'https://in.xfel.eu/metadata/oauth/token'
    refresh-url: 'https://in.xfel.eu/metadata/oauth/token'
    auth-url: 'https://in.xfel.eu/metadata/oauth/authorize'
    scope: ''
    base-api-url: 'https://in.xfel.eu/metadata/api/'

Here, user-ids, secrets, email etc as provided by ITDM need to be entered.

Finally, sections for the individual actions the service knows can be configured. Currently, the only action is correct:

correct:
    in-folder: /gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/raw
    out-folder: /gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/proc

It expects templates for the input and output file paths to be provided.

Starting the Service

The webservice can be started as a normal python process:

python webservice.py --mode [prod | prod-auth | sim]

The available modes are:

• prod: production mode
• prod-auth: production mode with authentication on ZMQ
• sim: simulation mode, no actual xfel-calibrate jobs are launched.

Use

python webservice.py --help

to display a list of available options.

Testing

There is a test environment on max-exfl-cal002, separate from the production instance.

ssh xcaltst@max-exfl-cal002.desy.de
cd /home/xcaltst/pycalibration

# Get the code you want to test
git pull
git checkout <branch-to-test>

# Stop the already running server, start a new one with your code
ps aux | grep webservice/webservice.py | awk '{ print $2}' | xargs kill
source ~/pycalibration_venv/bin/activate
nohup ~/pycalibration_venv/bin/python ~/pycalibration/webservice/webservice.py \
  --mode prod --logging INFO --config-file ~/pycalibration/webservice/webservice.yaml &

# Follow the logs
tail -f ~/pycalibration/web.log

The test instance of myMdC talks to this test service. Using data in the CALLAB proposal there, request recalibration (runs tab) and processing of dark runs (calibration constants tab) to send the 'correct' and 'dark_request' actions. The webservice logs and the status in myMdC should update as the processing occurs.

The command squeue -u xcaltst will show running & pending Slurm jobs started by this test system.

Manually Submitting Jobs

A script manual_launch.py is provided to manually submit jobs to the service.

usage: manual_launch.py [-h] --proposal PROPOSAL [--delay DELAY] [--noconfirm] [--really] slices [slices ...]

Manually submit calibration jobs.

positional arguments:
  slices               slices (or single numbers) of runs to process, inclusive range, starting at 1 (e.g. 1:3 parsed to {1, 2, 3}, 10 parsed to {10}, :10
                       parsed to {1, 2, ..., 10})

optional arguments:
  -h, --help           show this help message and exit
  --proposal PROPOSAL  proposal number
  --delay DELAY        delay in seconds between submissions
  --noconfirm          skip confirmation
  --really             actually submit jobs instead of just printing them

To run in the background use `nohup PYTHONUNBUFFERED=1 python manual_launch.py ... &` followed by `disown`.

Slices inclusive, so 1:10 would mean runs 1 to 10 inclusive of 1 and 10. The 'slice' can also be a single number.

Example of usage would be python3 ./manual_launch.py 1 10:12 160:-1 --delay 60 --proposal 2222 --really to submit runs 1, 10 to 12, and 160+ for calibration, for proposal 2222, with a 60 second delay between submissions.