Feat/user notebooks

Description

Adds in support for notebooks in user-defined spaces. Idea is that you can, in xfel_calibrate/notebooks.py add in a configuration like:

{
    ...
    "REMI": {
        "CORRECT": {
            "notebook": None,
            "user": {
                "notebook": "/gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/usr/calibration/notebooks/correct.ipynb",
                "venv": "/gpfs/exfel/sw/software/exfel_environments/sqs-remi-preview"
            },
            "concurrency": {
                "parameter": None,
                "use function": None,
                "default concurrency": None,
                "cluster cores": 1
            },
        },
    },
}

If ["notebook"] is set to None then user_notebook gets parsed and templated by CLI arguments, then the "notebook" key is set to this string and the rest continues as normal.

How Has This Been Tested?

• Add in tests
• Deploy and test with REMI CALLAB data on max-exfl17 with mymdc test instance

Manual test setup involved adding a notebook to /gpfs/exfel/exp/CALLAB/202130/p900203/usr/calibration/notebooks/correct.ipynb (this matches the REMI user notebook path pattern), then running xfel-calibrate REMI CORRECT --instrument CALLAB --cycle 202130 --proposal 900203.

This will then execute the test notebook.

The test notebook does... nothing really, it just creates an empty file called touch in the in_folder directory. If the SLURM job finishes without errors, the PDF renders correctly, and the touch file is present, then the user notebook with a user venv executed correctly.

Relevant Documents (optional)

Part of SQS Moonshot, see:

• https://git.xfel.eu/gitlab/moonshot/sqs/issues/13
• https://git.xfel.eu/gitlab/moonshot/sqs/issues/19

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

Notes:

• MR is done onto the https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/464 branch as this'll be merged only after that one is merged

tests/test_xfel_calibrate/test_cli.py

@@ -8,6 +8,7 @@ the current test cases, this should be improved later on.
 
 import ast
 import sys
+import shutil
 from datetime import date
 from pathlib import Path
 from unittest import mock
@@ -243,6 +244,65 @@ class TestIntelliList:
                 assert parameters["concurrency_parameter"][0] == i + 1
 
 
+class TestUserNotebooks:
+    @pytest.fixture(scope="class")
+    def mock_proposal(self, tmpdir_factory):
+        return MockProposal(
+            tmp_path=Path(tmpdir_factory.mktemp("exp")),
+            instrument="AGIPD",
+            cycle="202031",
+            proposal="p900113",
+            runs=1,
+            sequences=1,
+        )
+
+    @pytest.fixture(scope="function")
+    def calibrate_call(self, mock_proposal: MockProposal, capsys, tmp_path: Path):
+        # The user notebook path template for `TEST`/`TEST-USER-NB` is:
+        #    `"/{instrumen}/{cycle}/p{proposal}/test-cli.ipynb"`
+        #
+        # So by setting `instrument` to tmp_path/TEST we set the user notebook
+        # location to one in our tmp dir
+
+        self.user_nb = mock_proposal.path_usr / "test-cli.ipynb"
+
+        shutil.copy(
+            Path("./notebooks/test/test-cli.ipynb").absolute(), str(self.user_nb)
+        )
+
+        return CalibrateCall(
+            tmp_path,
+            capsys,
+            in_folder=mock_proposal.path_raw,
+            out_folder=mock_proposal.path_proc,
+            detector="TEST",
+            cal_type="TEST-USER-NB",
+            extra_args=[
+                "--root",
+                str(mock_proposal.path_usr),
+                "--number",
+                "10",
+            ],
+        )
+
+    def test_user_notebook_is_test_notebook(self, calibrate_call: CalibrateCall):
+        assert (
+            Path("./notebooks/test/test-cli.ipynb").read_bytes()
+            == self.user_nb.read_bytes()
+        )
+
+    def test_output_ipynb(self, calibrate_call: CalibrateCall):
+        notebook_path = calibrate_call.paths.notebooks
+        assert len(notebook_path) == 1
+
+        with notebook_path[0].open() as file:
+            notebook = nbformat.read(file, as_version=4)
+
+        parameters = {p.name: p.value for p in extract_parameters(notebook)}
+
+        assert parameters["number"] == 10
+
+
 class TestAgipdNotebook:
     @pytest.fixture(scope="function")
     def mock_proposal(self, tmpdir_factory):