Skip to content

Pre-place directory at 'archive' location

Thomas Kluyver requested to merge feat/work-dir-direct into master

Description

The working directory for calibration jobs (where notebooks, Slurm .out files, etc.) live is currently under a temp folder wherever xfel-calibrate is launched from, and the finalize job eventually moves it to an 'archive' location alongside the PDF report. This is a leftover from when the directory wasn't saved. This MR simplifies that to use the final location directly, getting rid of the move step. This means that if the finalize step fails, the output can be found in the normal location, rather than in xcal's home directory.

I've also tried to standardise what we call this directory in the xfel_calibrate code, to cal_work_dir. Previously it was a mixture of names like run_path, run_tmp_path, temp_path, some of which are also used for other directories, so it was quite confusing. However, it's still called metadata_folder in the notebooks, since that's what they use it for.

How Has This Been Tested?

Running on Maxwell:

xfel-calibrate agipd CORRECT \
  --karabo-da AGIPD00 AGIPD01 AGIPD02 AGIPD03 AGIPD04 AGIPD05 AGIPD06 AGIPD07 AGIPD08 AGIPD09 AGIPD10 AGIPD11 AGIPD12 AGIPD13 AGIPD14 AGIPD15 \
  --receiver-template '{}CH0' --karabo-id-control SPB_IRU_AGIPD1M1 \
  --overwrite --sequences-per-node 1 \
  --bias-voltage 300 --force-hg-if-below --hg-hard-threshold 1000 \
  --force-mg-if-below --mg-hard-threshold 1000 --blc-stripes \
  --cm-dark-range -30 30 --cm-dark-fraction 0.15 --cm-n-itr 4 --rel-gain \
  --adjust-mg-baseline --blc-set-min --low-medium-gap --xray-gain \
  --slopes-ff-from-files '' --ff-gain 7.3 \
  --in-folder /gpfs/exfel/exp/SPB/201801/p002120/raw \
  --out-folder /gpfs/exfel/data/scratch/kluyvert/agiprd-calib-p2120-r97 \
  --karabo-id SPB_DET_AGIPD1M-1 --run 97 --integration-time 12 --gain-mode 0 \
  --sequences 0

Output is in /gpfs/exfel/data/scratch/kluyvert/agiprd-calib-p2120-r97 (including the report and the cal work dir).

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Variable renaming

Checklist:

  • My code follows the code style of this project.

Reviewers

@schmidtp @ahmedk

Merge request reports