Skip to content

Add runtime summary to calibration metadata

David Hammer requested to merge feat/metadata_add_runtime_summary into master

Overview

As suggested by @moellerj on https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/202, we could add the "runtime summary" as it appears also to the calibration_metadata.yml. This MR will update finalize.py to do so. It also includes a fair amount of refactoring; some to split out the functionality of getting runtime summary, some to use pathlib in the functions touched.

The runtime summary in the report is not changed. The runtime summary information in the metadata would, with the current version of the MR, look like (containing the same information):

runtime-summary:
  calibration-jobs:
  - Elapsed: 00:00:55
    End: '2021-02-09T21:22:57'
    JobID: '6870244'
    Start: '2021-02-09T21:22:02'
    State: COMPLETED
    Suspended: 00:00:00
  - Elapsed: 00:00:17
    End: '2021-02-09T21:23:19'
    JobID: '6870245'
    Start: '2021-02-09T21:23:02'
    State: COMPLETED
    Suspended: 00:00:00
  - Elapsed: 00:00:05
    End: '2021-02-09T21:23:37'
    JobID: '6870246'
    Start: '2021-02-09T21:23:32'
    State: RUNNING
    Suspended: 00:00:00
  pipeline-steps:
    report-compilation-time: '2021-02-09T21:23:38'
    request-time: '2021-02-09T21:21:40'
    submission-time: '2021-02-09T21:21:40'

Testing

Small calibration run example:

TIMESTAMP=$(date "+%Y-%m-%d-%H-%M")
xfel-calibrate AGIPD CORRECT \
                           --slurm-mem 750 \
                           --slurm-name test-mr-409 \
                           --report-to /gpfs/exfel/data/scratch/hammerd/$TIMESTAMP-report \
                           --receiver-id {}CH0 \
                           --karabo-id-control MID_EXP_AGIPD1M1 \
                           --karabo-da-control AGIPD1MCTRL00 \
                           --h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
                           --sequences-per-node 1 \
                           --blc-stripes \
                           --in-folder /gpfs/exfel/exp/MID/202022/p002693/raw \
                           --out-folder /gpfs/exfel/data/scratch/hammerd/$TIMESTAMP-data \
                           --karabo-id MID_DET_AGIPD1M-1 \
                           --gain-setting 0 \
                           --cm-dark-fraction 0.15 \
                           --modules 0 \
                           --sequences 0 \
                           --run 1424

After the jobs have finished, the report files and such are in the correct locations. Slurm logs don't show any issues related to the new changes (will rebase and retest once the master branch settles).

Reviewers

@ahmedk @danilevc @roscar

Edited by David Hammer

Merge request reports