Feat/user notebooks

Description

Adds in support for notebooks in user-defined spaces. Idea is that you can, in xfel_calibrate/notebooks.py add in a configuration like:

{
    ...
    "REMI": {
        "CORRECT": {
            "notebook": None,
            "user": {
                "notebook": "/gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/usr/calibration/notebooks/correct.ipynb",
                "venv": "/gpfs/exfel/sw/software/exfel_environments/sqs-remi-preview"
            },
            "concurrency": {
                "parameter": None,
                "use function": None,
                "default concurrency": None,
                "cluster cores": 1
            },
        },
    },
}

If ["notebook"] is set to None then user_notebook gets parsed and templated by CLI arguments, then the "notebook" key is set to this string and the rest continues as normal.

How Has This Been Tested?

• Add in tests
• Deploy and test with REMI CALLAB data on max-exfl17 with mymdc test instance

Manual test setup involved adding a notebook to /gpfs/exfel/exp/CALLAB/202130/p900203/usr/calibration/notebooks/correct.ipynb (this matches the REMI user notebook path pattern), then running xfel-calibrate REMI CORRECT --instrument CALLAB --cycle 202130 --proposal 900203.

This will then execute the test notebook.

The test notebook does... nothing really, it just creates an empty file called touch in the in_folder directory. If the SLURM job finishes without errors, the PDF renders correctly, and the touch file is present, then the user notebook with a user venv executed correctly.

Relevant Documents (optional)

Part of SQS Moonshot, see:

• https://git.xfel.eu/gitlab/moonshot/sqs/issues/13
• https://git.xfel.eu/gitlab/moonshot/sqs/issues/19

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

Notes:

• MR is done onto the https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/464 branch as this'll be merged only after that one is merged

tests/test_xfel_calibrate/test_cli.py

@@ -220,7 +220,7 @@ class TestIntelliList:
                 "--list-intellilist",
                 "1,2,5-8",
                 "--concurrency-parameter",
-                "1,2",
+                "0,1",
             ],
         )
 
@@ -240,7 +240,7 @@ class TestIntelliList:
                 assert parameters["number"] == 10
                 assert parameters["list_normal"] == [1, 2, 10]
                 assert parameters["list_intellilist"] == [1, 2, 5, 6, 7]
-                assert parameters["concurrency_parameter"][0] == i + 1
+                assert parameters["concurrency_parameter"][0] == i
 
 
 class TestAgipdNotebook: