Add runtime summary to calibration metadata

Overview

As suggested by @moellerj on https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/202, we could add the "runtime summary" as it appears also to the calibration_metadata.yml. This MR will update finalize.py to do so. It also includes a fair amount of refactoring; some to split out the functionality of getting runtime summary, some to use pathlib in the functions touched.

The runtime summary in the report is not changed. The runtime summary information in the metadata would, with the current version of the MR, look like (containing the same information):

runtime-summary:
  calibration-jobs:
  - Elapsed: 00:00:55
    End: '2021-02-09T21:22:57'
    JobID: '6870244'
    Start: '2021-02-09T21:22:02'
    State: COMPLETED
    Suspended: 00:00:00
  - Elapsed: 00:00:17
    End: '2021-02-09T21:23:19'
    JobID: '6870245'
    Start: '2021-02-09T21:23:02'
    State: COMPLETED
    Suspended: 00:00:00
  - Elapsed: 00:00:05
    End: '2021-02-09T21:23:37'
    JobID: '6870246'
    Start: '2021-02-09T21:23:32'
    State: RUNNING
    Suspended: 00:00:00
  pipeline-steps:
    report-compilation-time: '2021-02-09T21:23:38'
    request-time: '2021-02-09T21:21:40'
    submission-time: '2021-02-09T21:21:40'

Testing

Small calibration run example:

TIMESTAMP=$(date "+%Y-%m-%d-%H-%M")
xfel-calibrate AGIPD CORRECT \
                           --slurm-mem 750 \
                           --slurm-name test-mr-409 \
                           --report-to /gpfs/exfel/data/scratch/hammerd/$TIMESTAMP-report \
                           --receiver-id {}CH0 \
                           --karabo-id-control MID_EXP_AGIPD1M1 \
                           --karabo-da-control AGIPD1MCTRL00 \
                           --h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
                           --sequences-per-node 1 \
                           --blc-stripes \
                           --in-folder /gpfs/exfel/exp/MID/202022/p002693/raw \
                           --out-folder /gpfs/exfel/data/scratch/hammerd/$TIMESTAMP-data \
                           --karabo-id MID_DET_AGIPD1M-1 \
                           --gain-setting 0 \
                           --cm-dark-fraction 0.15 \
                           --modules 0 \
                           --sequences 0 \
                           --run 1424

After the jobs have finished, the report files and such are in the correct locations. Slurm logs don't show any issues related to the new changes (will rebase and retest once the master branch settles).

Reviewers

@ahmedk @danilevc @roscar

added 2 commits

• aa061aa8 - Fix: incorrect path name (run path is tmp path)
• 9bb2c91e - Fix too short title underline

Compare with previous version

added 2 commits

• aa061aa8 - Fix: incorrect path name (run path is tmp path)
• 9bb2c91e - Fix too short title underline

Compare with previous version

added 1 commit

• 700a5c3e - isort

Compare with previous version

changed title from Add runtime summary to calibration metadata, start pathlib refactor to Add runtime summary to calibration metadata

changed the description

added Waiting for review label

changed the description

assigned to @roscar and unassigned @hammerd

assigned to @hammerd and unassigned @roscar

added 1 commit

• 9ffdb515 - Apply suggestion to xfel_calibrate/finalize.py

Compare with previous version

added 1 commit

• 1f51f689 - Improve get_job_info interaction with sacct

Compare with previous version

added 1 commit

• b2762682 - Further improvement of get_job_info

Compare with previous version

added 1 commit

• b2762682 - Further improvement of get_job_info

Compare with previous version

added 2 commits

• 8c693a28 - Remove temporary subdirs immediately, don't gather list
• fd8f4f69 - Stringify arguments to shutil.move (until python 3.9)

Compare with previous version