Offline Calibration Webservice
==============================
The offline calibration webservice interacts with the Metadata Catalogue (MDC),
such that migration of data to the offline cluster automatically triggers
calibration jobs on relevant files.
Installation
------------
Installation should only be performed by trained personal.
**Prerequisites**
The service needs to be installed under a functional user account which
* has read permission to the *raw* data folder on the Maxwell cluster
* has write permission to the *proc* folders for outputting corrected data
* is allowed to launch SLURM jobs on the cluster
The hosting system needs to be accessible via ZMQ calls from the MDC.
This requires appropriate DMZ settings. Additionally, it needs to be able
to interact with the MDC via the MDC client interface
**Installation of Dependencies**
The installation requirements can be found in the *requirements.txt* file.
Additionally, the *xfel-calibrate* environment needs to be installed:
1. clone the *pycalibration* repo into a directory of your choice:
``` bash
git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git .
```
2. pick the python environment to install into. On Maxwell the anaconda/3
environment will work:
``` bash
module load anaconda/3
```
3. install the *xfel-calibrate* environment
``` bash
pip install --user -r requirements.txt
```
4. some correction notebooks require pyDetLib. It requires manual installation in
a non-Karabo python environment
``` bash
mkdir pydetlib
cd pydetlib
git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git .
pip install --user ./lib/requirements.txt
pip install --user pycuda
pip install --user ./lib/
cd ..
5. install the separate requirements for the webservice:
``` bash
cd webservice
pip install --user -r requirements.txt
```
6. install the metadata_client library, according to instructions at
https://git.xfel.eu/gitlab/ITDM/metadata_client
You are now good to go.
Configuration
-------------
Configuration is done through the *webservice.yaml* file in the webservice directory.
On a new installation you will likely need to change the following parameters.
In the **config-repo** section, the configuration repository needs to be configured:
``` YAML
config-repo:
url: https://git.xfel.eu/gitlab/detectors/calibration_configurations.git
local-path: /home/haufs/calibration_config/
```
Here you should prepend the *url* entry with a gitlab access token, that provides access
to the calibration_configurations repository.
In the **web-service** section, the webservice itself is configured:
``` YAML
web-service:
port: 5555
bind-to: tcp://*
allowed-ips: '111.222.222.111', ''111.222.222.112'
job-db: ./webservice_jobs.sqlite
job-update-interval: 30
job-timeout: 3600
```
In case you want to use authentication, add a list of *allowed-ips*.
In the **metadata-client** section, the client interface to the MDC is configured:
``` YAML
metadata-client:
user-id:
user-secret:
user-email:
metadata-web-app-url: 'https://in.xfel.eu/metadata'
metadata-web-app-url: 'https://in.xfel.eu/metadata'
token-url: 'https://in.xfel.eu/metadata/oauth/token'
refresh-url: 'https://in.xfel.eu/metadata/oauth/token'
auth-url: 'https://in.xfel.eu/metadata/oauth/authorize'
scope: ''
base-api-url: 'https://in.xfel.eu/metadata/api/'
```
Here, user-ids, secrets, email etc as provided by ITDM need to be entered.
Finally, sections for the individual *actions* the service knows can be configured.
Currently, the only action is *correct*:
``` YAML
correct:
in-folder: /gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/raw
out-folder: /gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/proc
```
It expects templates for the input and output file paths to be provided.
Starting the Service
--------------------
The webservice can be started as a normal python process:
``` bash
python webservice.py --mode [prod | prod-auth | sim]
```
The available modes are:
* prod: production mode
* prod-auth: production mode with authentication on ZMQ
* sim: simulation mode, no actual *xfel-calibrate* jobs are launched.
Use
``` bash
python webservice.py --help
```
to display a list of available options.
Testing
-------
There is a test environment on ``max-exfl-cal002``, separate from the production
instance.
```bash
ssh xcaltst@max-exfl-cal002.desy.de
cd /home/xcaltst/pycalibration
# Get the code you want to test
git pull
git checkout <branch-to-test>
# Stop the already running server, start a new one with your code
ps aux | grep webservice/webservice.py | awk '{ print $2}' | xargs kill
source ~/pycalibration_venv/bin/activate
nohup ~/pycalibration_venv/bin/python ~/pycalibration/webservice/webservice.py \
--mode prod --logging INFO --config-file ~/pycalibration/webservice/webservice.yaml &
# Follow the logs
tail -f ~/pycalibration/web.log
```
The [test instance of myMdC](https://in.xfel.eu/test_metadata/) talks to this
test service. Using data in the [CALLAB proposal](https://in.xfel.eu/test_metadata/proposals/259)
there, request recalibration ([runs tab](https://in.xfel.eu/test_metadata/proposals/259#proposal-runs))
and processing of dark runs ([calibration constants tab](https://in.xfel.eu/test_metadata/proposals/259#proposal-calibration))
to send the 'correct' and 'dark_request' actions. The webservice logs and the
status in myMdC should update as the processing occurs.
The command ``squeue -u xcaltst`` will show running & pending Slurm jobs started
by this test system.
Manually Submitting Jobs
------------------------
A script `manual_launch.py` is provided to manually submit jobs to the service.
```bash
usage: manual_launch.py [-h] --proposal PROPOSAL [--delay DELAY] [--noconfirm] [--really] slices [slices ...]
Manually submit calibration jobs.
positional arguments:
slices slices (or single numbers) of runs to process, inclusive range, starting at 1 (e.g. 1:3 parsed to {1, 2, 3}, 10 parsed to {10}, :10
parsed to {1, 2, ..., 10})
optional arguments:
-h, --help show this help message and exit
--proposal PROPOSAL proposal number
--delay DELAY delay in seconds between submissions
--noconfirm skip confirmation
--really actually submit jobs instead of just printing them
To run in the background use `nohup PYTHONUNBUFFERED=1 python manual_launch.py ... &` followed by `disown`.
```
Slices inclusive, so `1:10` would mean runs 1 to 10 inclusive of 1 and 10. The
'slice' can also be a single number.
Example of usage would be `python3 ./manual_launch.py 1 10:12 160:-1 --delay 60
--proposal 2222 --really` to submit runs 1, 10 to 12, and 160+ for calibration,
for proposal 2222, with a 60 second delay between submissions.