Feat/user notebooks

Description

Adds in support for notebooks in user-defined spaces. Idea is that you can, in xfel_calibrate/notebooks.py add in a configuration like:

{
    ...
    "REMI": {
        "CORRECT": {
            "notebook": None,
            "user": {
                "notebook": "/gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/usr/calibration/notebooks/correct.ipynb",
                "venv": "/gpfs/exfel/sw/software/exfel_environments/sqs-remi-preview"
            },
            "concurrency": {
                "parameter": None,
                "use function": None,
                "default concurrency": None,
                "cluster cores": 1
            },
        },
    },
}

If ["notebook"] is set to None then user_notebook gets parsed and templated by CLI arguments, then the "notebook" key is set to this string and the rest continues as normal.

How Has This Been Tested?

• Add in tests
• Deploy and test with REMI CALLAB data on max-exfl17 with mymdc test instance

Manual test setup involved adding a notebook to /gpfs/exfel/exp/CALLAB/202130/p900203/usr/calibration/notebooks/correct.ipynb (this matches the REMI user notebook path pattern), then running xfel-calibrate REMI CORRECT --instrument CALLAB --cycle 202130 --proposal 900203.

This will then execute the test notebook.

The test notebook does... nothing really, it just creates an empty file called touch in the in_folder directory. If the SLURM job finishes without errors, the PDF renders correctly, and the touch file is present, then the user notebook with a user venv executed correctly.

Relevant Documents (optional)

Part of SQS Moonshot, see:

• https://git.xfel.eu/gitlab/moonshot/sqs/issues/13
• https://git.xfel.eu/gitlab/moonshot/sqs/issues/19

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

Notes:

• MR is done onto the https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/464 branch as this'll be merged only after that one is merged

tests/test_xfel_calibrate/conftest.py

@@ -6,11 +6,10 @@ from unittest import mock
 
 import extra_data.tests.make_examples as make_examples
 import pytest
+from pytest_subprocess.core import FakeProcess
 
 from xfel_calibrate import calibrate, settings
 
-from pytest_subprocess.core import FakeProcess
-
 
 @pytest.fixture(autouse=True)
 def fake_process_xfel_calibrate(fake_process: FakeProcess) -> FakeProcess: