Feat/user notebooks

Description

Adds in support for notebooks in user-defined spaces. Idea is that you can, in xfel_calibrate/notebooks.py add in a configuration like:

{
    ...
    "REMI": {
        "CORRECT": {
            "notebook": None,
            "user": {
                "notebook": "/gpfs/exfel/exp/{instrument}/{cycle}/p{proposal}/usr/calibration/notebooks/correct.ipynb",
                "venv": "/gpfs/exfel/sw/software/exfel_environments/sqs-remi-preview"
            },
            "concurrency": {
                "parameter": None,
                "use function": None,
                "default concurrency": None,
                "cluster cores": 1
            },
        },
    },
}

If ["notebook"] is set to None then user_notebook gets parsed and templated by CLI arguments, then the "notebook" key is set to this string and the rest continues as normal.

How Has This Been Tested?

• Add in tests
• Deploy and test with REMI CALLAB data on max-exfl17 with mymdc test instance

Manual test setup involved adding a notebook to /gpfs/exfel/exp/CALLAB/202130/p900203/usr/calibration/notebooks/correct.ipynb (this matches the REMI user notebook path pattern), then running xfel-calibrate REMI CORRECT --instrument CALLAB --cycle 202130 --proposal 900203.

This will then execute the test notebook.

The test notebook does... nothing really, it just creates an empty file called touch in the in_folder directory. If the SLURM job finishes without errors, the PDF renders correctly, and the touch file is present, then the user notebook with a user venv executed correctly.

Relevant Documents (optional)

Part of SQS Moonshot, see:

• https://git.xfel.eu/gitlab/moonshot/sqs/issues/13
• https://git.xfel.eu/gitlab/moonshot/sqs/issues/19

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

Notes:

• MR is done onto the https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/464 branch as this'll be merged only after that one is merged

.gitlab-ci.yml

@@ -43,7 +43,7 @@ pytest:
   <<: *before_script
   script:
     - python3 -m pip install ".[test]"
-    - python3 -m pytest --verbose --cov=cal_tools --cov=xfel_calibrate
+    - python3 -m pytest --color yes --verbose --cov=cal_tools --cov=xfel_calibrate
 #  Nope... https://docs.gitlab.com/12.10/ee/user/project/merge_requests/test_coverage_visualization.html#enabling-the-feature
 #    - coverage xml
 #  artifacts:
@@ -56,4 +56,4 @@ cython-editable-install-test:
   <<: *before_script
   script:
     - python3 -m pip install -e ".[test]"
-    - python3 -m pytest ./tests/test_agipdalgs.py
+    - python3 -m pytest --color yes --verbose ./tests/test_agipdalgs.py