README.md

Offline Calibration Webservice

The offline calibration webservice interacts with the Metadata Catalogue (MDC), such that migration of data to the offline cluster automatically triggers calibration jobs on relevant files.

Installation

Installation should only be performed by trained personal.

Prerequisites

The service needs to be installed under a functional user account which

• has read permission to the raw data folder on the Maxwell cluster
• has write permission to the proc folders for outputting corrected data
• is allowed to launch SLURM jobs on the cluster

The hosting system needs to be accessible via ZMQ calls from the MDC. This requires appropriate DMZ settings. Additionally, it needs to be able to interact with the MDC via the MDC client interface

Installation of Dependencies

The installation requirements can be found in the requirements.txt file. Additionally, the xfel-calibrate environment needs to be installed:

1. clone the pycalibration repo into a directory of your choice:

   git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git .

2. pick the python environment to install into. On Maxwell the anaconda/3 environment will work:

   module load anaconda/3

3. install the xfel-calibrate environment

   pip install --user -r requirements.txt

4. some correction notebooks require pyDetLib. It requires manual installation in a non-Karabo python environment

   mkdir pydetlib
   cd pydetlib
   git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git .
   pip install --user ./lib/requirements.txt
   pip install --user pycuda
   pip install --user ./lib/
   cd ..
   

5. install the separate requirements for the webservice:

   cd webservice
   pip install --user -r requirements.txt

6. install the metadata_client library, according to instructions at

   https://git.xfel.eu/gitlab/ITDM/metadata_client

You are now good to go.

Configuration

Configuration is done through the webservice.yaml file in the webservice directory. On a new installation you will likely need to change the following parameters.

In the config-repo section, the configuration repository needs to be configured:

config-repo:
    url: https://git.xfel.eu/gitlab/detectors/calibration_configurations.git
    local-path: /home/haufs/calibration_config/

Here you should prepend the url entry with a gitlab access token, that provides access to the calibration_configurations repository.

In the web-service section, the webservice itself is configured:

web-service:
    port: 5555
    bind-to: tcp://*
    allowed-ips: '111.222.222.111', ''111.222.222.112'
    job-db: ./webservice_jobs.sqlite
    job-update-interval: 30
    job-timeout: 3600

In case you want to use authentication, add a list of allowed-ips.