Add a pytest to run a dict of CALLAB test runs before releases

Description

Manually triggered automated tests for testing all CALLAB reference runs before releases and for individual MRs.

Triggering the test from gitlab

Using the CI the test can be triggered manually after pushing a commit and by setting 4 variables and their values.

CALIBRATION: dark, correct, or all (all is used by default).
DETECTORS: A detector name or all (all is used by default).
REFERENCE: The reference folder name (reference_folder is used by default).
OUTPUT: The output folder for the tested data (branch name is used by default).

If any or all of these variables are not given the default values are used.

The automated tests here use a yaml file similar to calibration_configurations to define the reference data and test calibration parameters.

What is the test about

• Testing the xfel-calibrate submission: use slurm or without based on the selected config. The CI is not using slurm.
• Testing the successful completion for the submitted xfel-calibrate command.
• Checking for the report availability.
• Validate the number of generated H5 files.
• Validate the numerical values for the calibrated H5 files against the reference files.
• Find the difference between one of the calibrated files and the reference files.

How does the test work

1. pytest is used.
2. Scheduled tests run sequentially.
3. The test can be using slurm or not based on the configuration. --use-slurm
4. The test can try to find the differences between the datasets and objects for the non-validated calibrated data.

Test file structure

pycalibration
|   ...
└───tests
|   |   pytest.ini  # Enable logging for the tests
|   |   ...
│   └───test_reference_runs
|       |   __init__.py
│       │   test_pre_deployment  # Main test module
│       │   callab_tests  # Main dict for all tests to run and its `xfel-calibrate` config
│       │   conftest.py
|       |   ...

Test configurations

1. Pick one specific test. --picked-test
2. select detectors to test. --calibration
3. select calibration to test. --detectors
4. Skip numerical validation for produced h5files. --no-numerical-validation
5. Apply numerical validation only between test and reference data --validation-only
6. Find the difference for non-validated files --find-difference

How Has This Been Tested?

• This has been tested by comparing Merge requests with the current master version.
• It has been tested for pre-release test: https://git.xfel.eu/detectors/pycalibration/-/tags/3.5.0b1

Relevant Documents (optional)

Types of changes

• Test (additional or refactored tests)

Checklist:

• Add some of 900113/900203 runs in callab_test.yml.

• Add docstrings and more documentation.

• Add documentation for the tests.

  • Added a page to the new documentation for pycalibration. But it needs to be updated after the latest changes. In the meantime, I have updated the description for this MR and added documentation in the code as well.

• Enable manually triggering the tests from Gitlab.

Test checklist:

• Review the reference test on the test node.

• Decide where to keep the reference and test output folder.

  • REFERENCE: /gpfs/exfel/data/scratch/xcaltst/test/reference_folder
  • OUTPUT: /gpfs/exfel/data/data/scratch/xcaltst/test/$BRANCH_NAME

• If a test is retried it should have default values otherwise it wont work without pushing to the branch. Decide how to fix this.

• Decide on the best workflow for using this type of test.

Reviewers

tests/test_pre_deployment.py

-"""
-1- RUN AGIPD xfel-calibrate (Run multiple threads/asyncio to run for different detectors/calibration-types)
-2- AWAIT the slurm jobs to finish
-3- DO healthy checks:
-  a- Check available report.
-  b- Check no errors in the rst files/slurm status.
-  c- Check number of corrected files? local constant files?
-  d- check expected time performance from time-summary files??
-  e- Copy latest out-data for comparison and validity check after ending the test???
-"""
-
-
-# 04.01.2022
-
 """[summary]
 
 1. Read a dict of all detectors and calibration types to test.
-2. Run xfel-calibrate for each detector and watch the SLURM jobs until it ends (TODO Asyncio/Background threads??).
+2. Run xfel-calibrate for each detector and watch the SLURM jobs until it ends.
 3. Validate that all SLURM jobs were COMPLETED.
-4. Validate the presence of the report within the last 10 seconds
-(TODO: Slurm folder and calibration metadata as well?)
-5. Validate number of generated h5files.
-6. Validate that the h5files are exactly the same.
+4. Validate the presence of the PDF report.
+5. Validate number of generated h5files against reference data.
+6. Validate that the h5files are exactly the same against reference data.
 7. TODO Validate expected time performace from the time-summary files?
-8. Write/Output a report for each test result? [HTML pytest output?]
+8. Write/Output a report for each test result, by running pytest using --html=<report_name>.html
 """
 
-import asyncio
 import hashlib
 import io
+import multiprocessing
 import pathlib
 import time
 from contextlib import redirect_stdout
 from subprocess import PIPE, run
+from typing import Any, Dict, List
 
 import pytest
 import yaml
-from typing import List, Any, Dict
+
 import xfel_calibrate.calibrate as calibrate
 
 with open("./callab_tests.yaml", "r") as f:
     callab_test_dict = yaml.load(f)
-import multiprocessing
+
 
 def validate_h5files(f, reference):
     def filemd5(filename, block_size=2**20):
@@ -56,6 +42,7 @@ def validate_h5files(f, reference):
 
     return filemd5(f) == filemd5(reference / f.name), f
 
+
 def parse_config(cmd: List[str], config: Dict[str, Any]) -> List[str]:
     """Convert a dictionary to a list of arguments.
 
@@ -84,7 +71,6 @@ def parse_config(cmd: List[str], config: Dict[str, Any]) -> List[str]:
 
 @pytest.mark.manual_run
 @pytest.mark.parametrize("calibration_test", list(callab_test_dict.items()))
-@pytest.mark.asyncio
 def test_xfel_calibrate(calibration_test):
     test_key, val_dict = calibration_test
 
@@ -148,4 +134,4 @@ def test_xfel_calibrate(calibration_test):
                 if i[0]:
                     non_valid_files.append(i[1])
             assert all(all_files_validated), f"{test_key} failure, while validating {non_valid_files}"  # noqa
-            print(f"{test_key}'s calibration h5files are validated successfully.")
\ No newline at end of file
+            print(f"{test_key}'s calibration h5files are validated successfully.")