Feat/user notebooks

Description

Adds in support for notebooks in user-defined spaces. Idea is that you can, in xfel_calibrate/notebooks.py add in a configuration like:

{
    ...
    "REMI": {
        "CORRECT": {
            "notebook": None,
            "user": {
                "notebook": "/gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/usr/calibration/notebooks/correct.ipynb",
                "venv": "/gpfs/exfel/sw/software/exfel_environments/sqs-remi-preview"
            },
            "concurrency": {
                "parameter": None,
                "use function": None,
                "default concurrency": None,
                "cluster cores": 1
            },
        },
    },
}

If ["notebook"] is set to None then user_notebook gets parsed and templated by CLI arguments, then the "notebook" key is set to this string and the rest continues as normal.

How Has This Been Tested?

• Add in tests
• Deploy and test with REMI CALLAB data on max-exfl17 with mymdc test instance

Manual test setup involved adding a notebook to /gpfs/exfel/exp/CALLAB/202130/p900203/usr/calibration/notebooks/correct.ipynb (this matches the REMI user notebook path pattern), then running xfel-calibrate REMI CORRECT --instrument CALLAB --cycle 202130 --proposal 900203.

This will then execute the test notebook.

The test notebook does... nothing really, it just creates an empty file called touch in the in_folder directory. If the SLURM job finishes without errors, the PDF renders correctly, and the touch file is present, then the user notebook with a user venv executed correctly.

Relevant Documents (optional)

Part of SQS Moonshot, see:

• https://git.xfel.eu/gitlab/moonshot/sqs/issues/13
• https://git.xfel.eu/gitlab/moonshot/sqs/issues/19

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

Notes:

• MR is done onto the https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/464 branch as this'll be merged only after that one is merged

notebooks/test/test-cli.ipynb

+%% Cell type:markdown id: tags:
 
+# Test Notebook - CLI
+
+Author: Robert Rosca
+
+Version: 0.1
+
+Notebook for use with the unit and continuous integration tests.
 
+%% Cell type:code id: tags:
 
+``` 
+root = "root/path/for/nb"  # Variables included in the user notebook path must
+# be included in the notebook for reasons
+
+in_folder = "/dev/null" # input folder
+out_folder = "/dev/null" # output folder
+list_normal = [10] # parameterized list, range allowed
+list_intellilist = [2345] # parameterized list with ranges, range allowed
+concurrency_parameter = [1] # concurrency parameter, range allowed
+number = 0 # parameterized number
 ```
 
+%% Cell type:markdown id: tags:
 
+Tests notebook execution by just creating an empty file in the output directory.
 
+%% Cell type:code id: tags:
 
+``` 
+from pathlib import Path
 ```
 
+%% Cell type:code id: tags:
 
+``` 
+in_folder = Path(in_folder)
+
+(in_folder / "touch").touch()
 ```