Reproducibility, step 1

Description

This is meant to be a first step towards reproducible calibration. Specifically, given the work directory (containing the parameterised ipynb files) from a previous calibration run, we can re-run the notebooks with a specified target Python, optionally directing output to a new location. I think this will also be useful for testing, because it will be easy to prepare the plan of what work is to be done without launching it.

There are two main limitations in terms of reproducibility, which I hope to address in later MRs:

• Package versions are recorded (pip freeze), but for now you have to create a suitable environment manually, and tell the 'repeat' machinery to use it.
• We can run the same notebooks with the same parameters, but they may use changing external resources like the calibration database. This will need to be addressed within the notebooks.

Open questions:

• Naming, of course:
  • The JobArgs, Step, JobGroup classes for describing different levels of work to be done.
  • I've called the module 'repeat'. But if there's an option like --prepare to do everything up to submitting the jobs, then you could be using the same thing to run it for the first time.
• What information should be stored in what files & formats?
  • I added a new steps.json describing how to run the notebooks in the directory.
  • Python packages are listed in requirements.txt
  • It also uses some info from calibration_metadata.yml
    • This has a mixture of information from the machinery (like the parameters) and from the specific pipeline (like calibration constants) - should they be split up?
    • It's saved both in the work directory (where the .ipynb files go) and the output directory - what if they get out of sync?

How Has This Been Tested?

Running commands on Maxwell. Run a correction:

xfel-calibrate AGIPD CORRECT \
    --in-folder /gpfs/exfel/exp/SPB/202131/p900230/raw --run 174 \
    --out-folder /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900230-174-repro \
    --karabo-id SPB_DET_AGIPD1M-1 --karabo-id-control SPB_IRU_AGIPD1M1 --karabo-da-control AGIPD1MCTRL00 \
    --modules 0-4

Repeat it:

python3 -m xfel_calibrate.repeat \
    /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900230-174-repro/slurm_out_AGIPDOfflineCorrection \
    --out-folder /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900230-174-repro2

Types of changes

• New feature (non-breaking change which adds functionality)
• Refactor (refactoring code with no functionality changes)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.

Reviewers

@calibration