Add a pytest to run a dict of CALLAB test runs before releases

Description

Manually triggered automated tests for testing all CALLAB reference runs before releases and for individual MRs.

Triggering the test from gitlab

Using the CI the test can be triggered manually after pushing a commit and by setting 4 variables and their values.

CALIBRATION: dark, correct, or all (all is used by default).
DETECTORS: A detector name or all (all is used by default).
REFERENCE: The reference folder name (reference_folder is used by default).
OUTPUT: The output folder for the tested data (branch name is used by default).

If any or all of these variables are not given the default values are used.

The automated tests here use a yaml file similar to calibration_configurations to define the reference data and test calibration parameters.

What is the test about

• Testing the xfel-calibrate submission: use slurm or without based on the selected config. The CI is not using slurm.
• Testing the successful completion for the submitted xfel-calibrate command.
• Checking for the report availability.
• Validate the number of generated H5 files.
• Validate the numerical values for the calibrated H5 files against the reference files.
• Find the difference between one of the calibrated files and the reference files.

How does the test work

1. pytest is used.
2. Scheduled tests run sequentially.
3. The test can be using slurm or not based on the configuration. --use-slurm
4. The test can try to find the differences between the datasets and objects for the non-validated calibrated data.

Test file structure

pycalibration
|   ...
└───tests
|   |   pytest.ini  # Enable logging for the tests
|   |   ...
│   └───test_reference_runs
|       |   __init__.py
│       │   test_pre_deployment  # Main test module
│       │   callab_tests  # Main dict for all tests to run and its `xfel-calibrate` config
│       │   conftest.py
|       |   ...

Test configurations

1. Pick one specific test. --picked-test
2. select detectors to test. --calibration
3. select calibration to test. --detectors
4. Skip numerical validation for produced h5files. --no-numerical-validation
5. Apply numerical validation only between test and reference data --validation-only
6. Find the difference for non-validated files --find-difference

How Has This Been Tested?

• This has been tested by comparing Merge requests with the current master version.
• It has been tested for pre-release test: https://git.xfel.eu/detectors/pycalibration/-/tags/3.5.0b1

Relevant Documents (optional)

Types of changes

• Test (additional or refactored tests)

Checklist:

• Add some of 900113/900203 runs in callab_test.yml.

• Add docstrings and more documentation.

• Add documentation for the tests.

  • Added a page to the new documentation for pycalibration. But it needs to be updated after the latest changes. In the meantime, I have updated the description for this MR and added documentation in the code as well.

• Enable manually triggering the tests from Gitlab.

Test checklist:

• Review the reference test on the test node.

• Decide where to keep the reference and test output folder.

  • REFERENCE: /gpfs/exfel/data/scratch/xcaltst/test/reference_folder
  • OUTPUT: /gpfs/exfel/data/data/scratch/xcaltst/test/$BRANCH_NAME

• If a test is retried it should have default values otherwise it wont work without pushing to the branch. Decide how to fix this.

• Decide on the best workflow for using this type of test.

Reviewers

tests/callab_tests.yaml

@@ -66,7 +66,7 @@
     "out-folder": "{}/{}/{}"
     "in-folder": "/gpfs/exfel/exp/MID/202121/p002929/raw"
     "run": "21"
-    "karabo-id-control": "MID_IRU_AGIPD1M1"
+    "karabo-id-control": "MID_EXP_AGIPD1M1"
     "karabo-da-control": "AGIPD1MCTRL00"
     "karabo-id": "MID_DET_AGIPD1M-1"
     "slurm-mem": "750"
@@ -82,7 +82,7 @@
     "run-high": "4"
     "run-med": "5"
     "run-low": "6"
-    "karabo-id-control": "MID_IRU_AGIPD1M1"
+    "karabo-id-control": "MID_EXP_AGIPD1M1"
     "karabo-da-control": "AGIPD1MCTRL00"
     "karabo-id": "MID_DET_AGIPD1M-1"
     "slurm-mem": "750"
@@ -131,7 +131,7 @@
       - 160
       - 2
     "common-mode": true
-    "karabo-id-control": "MID_IRU_AGIPD1M1"
+    "karabo-id-control": "MID_EXP_AGIPD1M1"
     "karabo-da-control": "AGIPD1MCTRL00"
     "karabo-id": "MID_DET_AGIPD1M-1"
     "slurm-mem": "750"
@@ -150,7 +150,7 @@
       - 160
       - 2
     "no-common-mode": true
-    "karabo-id-control": "MID_IRU_AGIPD1M1"
+    "karabo-id-control": "MID_EXP_AGIPD1M1"
     "karabo-da-control": "AGIPD1MCTRL00"
     "karabo-id": "MID_DET_AGIPD1M-1"
     "slurm-mem": "750"
@@ -167,7 +167,6 @@
     "run-med": "48"
     "run-low": "49"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF500K"
     "karabo-da": "JNGFR03"
   "reference-folder": "{}/{}/{}"
@@ -180,7 +179,6 @@
     "in-folder": "/gpfs/exfel/exp/FXE/202101/p002478/raw/"
     "run": "52"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF500K"
     "karabo-da": "JNGFR03"
   "reference-folder": "{}/{}/{}"
@@ -190,12 +188,11 @@
   "cal_type": "DARK"
   "config":
     "out-folder": "{}/{}/{}"
-    "in-folder": "/gpfs/exfel/exp/FXE/202101/p002478/raw/"
+    "in-folder": "/gpfs/exfel/exp/FXE/202131/p900226/raw/"
     "run-high": "108"
     "run-med": "109"
     "run-low": "110"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF1M"
     "karabo-da":
       - "JNGFR01"
@@ -210,7 +207,6 @@
     "in-folder": "/gpfs/exfel/exp/FXE/202131/p900226/raw"
     "run": "106"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF1M"
     "karabo-da":
       - "JNGFR01"
@@ -227,7 +223,6 @@
     "run-med": "48"
     "run-low": "49"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF1M"
     "karabo-da":
       - "JNGFR01"
@@ -242,7 +237,6 @@
     "in-folder": "/gpfs/exfel/exp/FXE/202101/p002478/raw/"
     "run": "52"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "FXE_XAD_JF1M"
     "karabo-da":
       - "JNGFR01"
@@ -257,7 +251,6 @@
     "in-folder": "/gpfs/exfel/exp/SPB/202130/p900204/raw"
     "run": "91"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "SPB_IRDA_JF4M"
     "karabo-da":
       - "JNGFR01"
@@ -280,7 +273,6 @@
     "run-med": "89"
     "run-low": "90"
     "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "SPB_IRDA_JF4M"
     "karabo-da":
       - "JNGFR01"
@@ -300,8 +292,6 @@
     "out-folder": "{}/{}/{}"
     "in-folder": "/gpfs/exfel/exp/HED/202102/p002656/raw"
     "run": "230"
-    "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "HED_IA1_JF500K1"
     "karabo-da": "JNGFR01"
   "reference-folder": "{}/{}/{}"
@@ -315,8 +305,6 @@
     "run-high": "219"
     "run-med": "220"
     "run-low": "221"
-    "karabo-id-control": ""
-    "karabo-da-control": "JNGFRCTRL00"
     "karabo-id": "HED_IA1_JF500K1"
     "karabo-da": "JNGFR01"
   "reference-folder": "{}/{}/{}"
@@ -325,7 +313,7 @@
   "det_type": "EPIX100"
   "cal_type": "CORRECT"
   "config":
-    "out-folder": "{}/{}-1/{}"
+    "out-folder": "{}/{}/{}"
     "in-folder": "/gpfs/exfel/exp/HED/202102/p002656/raw"
     "run": "230"
     "karabo-id": "HED_IA1_EPX100-1"
@@ -341,7 +329,7 @@
     "run": "219"
     "karabo-id": "HED_IA1_EPX100-1"
     "karabo-da": "EPIX01"
-"reference-folder": "{}/{}/{}"
+  "reference-folder": "{}/{}/{}"
 
 "SQS_NQS_PNCCD1MP-CORRECT":
   "det_type": "PNCCD"