Store calibration_metadata.yml primarily in work directory

Description

Currently, calibration_metadata.yml is saved in the output directory, and copied to the work directory (along with Slurm output etc.) only in the finalize step. This causes issues when there are multiple detectors being processed with the same output directory - they try to use the same file, and overwrite each other's data.

The aim of this MR is to reverse this, so the metadata file is saved in the work directory first, and symlinked to the output directory at the end.

Keeping everything else working requires a number of tweaks:

• In the output directory, the metadata file is linked as both calibration_metadata_MID_DET_AGIPD1M-1.yml (using the karabo_id parameter) and calibration_metadata.yml for compatibility with anything that expects that. If there are multiple detectors, the last one wins the unqualified name.
• Notebooks that need to refer to this file gain a new parameter metadata_folder with special handling - the xfel-calibrate machinery passes this value itself, rather than exposing it as a command-line option.
• The AGIPD correction notebooks, which use the metadata file for passing along constants to use, fall back to putting it in the output directory for interactive use (when metadata_folder is not set).
• Dark characterisation notebooks, which use it to get the path where the PDF report will be placed to tell myMdC, have no fallback (as there's no report when you run them interactively).

How Has This Been Tested?

Running an AGIPD correction:

xfel-calibrate agipd CORRECT --slurm-mem 700 --slurm-name correct_MID_agipd_202201_p002834_r60 \
   --cal-db-timeout 300000 --cal-db-interface 'tcp://max-exfl016:8015#8044' \
   --ctrl-source-template '{}/MDL/FPGA_COMP' \
   --karabo-da AGIPD00 AGIPD01 AGIPD02 AGIPD03 AGIPD04 AGIPD05 AGIPD06 AGIPD07 AGIPD08 AGIPD09 AGIPD10 AGIPD11 AGIPD12 AGIPD13 AGIPD14 AGIPD15 \
   --karabo-id-control MID_EXP_AGIPD1M1 --receiver-template '{}CH0' --adjust-mg-baseline \
   --bias-voltage 300 --blc-set-min --blc-stripes --cm-dark-fraction 0.15 \
   --cm-dark-range -30 30 --cm-n-itr 4 --common-mode --ff-gain 1.0 \
   --force-hg-if-below --force-mg-if-below --hg-hard-threshold 1000 \
   --low-medium-gap --mg-hard-threshold 1000 --overwrite --rel-gain \
   --sequences-per-node 1 --slopes-ff-from-files '' --xray-gain \
   --karabo-id MID_DET_AGIPD1M-1 \
   --in-folder /gpfs/exfel/exp/MID/202201/p002834/raw  --run 60 \
   --out-folder /gpfs/exfel/data/scratch/kluyvert/agipd-calib-2834-60-mddir

Results in /gpfs/exfel/data/scratch/kluyvert/agipd-calib-2834-60-mddir look like I'd expect - there are the two symlinks to the metadata file in the slurm_out_... folder, and the logs show that the main AGIPD notebook got the constants from the YAML file as it should.

Relevant Documents (optional)

AGIPDOfflineCorrection.pdf

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the code style of this project.

Reviewers

@ahmedk @schmidtp