Manually triggered automated tests for testing all CALLAB reference runs before releases and for individual MRs.
Triggering the test from gitlab
Using the CI the test can be triggered manually after pushing a commit and by setting 4 variables and their values.
CALIBRATION: dark, correct, or all (all is used by default). DETECTORS: A detector name or all (all is used by default). REFERENCE: The reference folder name (reference_folder is used by default). OUTPUT: The output folder for the tested data (branch name is used by default).
If any or all of these variables are not given the default values are used.
The automated tests here use a yaml file similar to calibration_configurations to define the reference data and test calibration parameters.
What is the test about
Testing the xfel-calibrate submission: use slurm or without based on the selected config. The CI is not using slurm.
Testing the successful completion for the submitted xfel-calibrate command.
Checking for the report availability.
Validate the number of generated H5 files.
Validate the numerical values for the calibrated H5 files against the reference files.
Find the difference between one of the calibrated files and the reference files.
How does the test work
pytest is used.
Scheduled tests run sequentially.
The test can be using slurm or not based on the configuration. --use-slurm
The test can try to find the differences between the datasets and objects for the non-validated calibrated data.
Test file structure
pycalibration| ...└───tests| | pytest.ini # Enable logging for the tests| | ...│ └───test_reference_runs| | __init__.py│ │ test_pre_deployment # Main test module│ │ callab_tests # Main dict for all tests to run and its `xfel-calibrate` config│ │ conftest.py| | ...
Test configurations
Pick one specific test. --picked-test
select detectors to test. --calibration
select calibration to test. --detectors
Skip numerical validation for produced h5files. --no-numerical-validation
Apply numerical validation only between test and reference data --validation-only
Find the difference for non-validated files --find-difference
How Has This Been Tested?
This has been tested by comparing Merge requests with the current master version.
Add some of 900113/900203 runs in callab_test.yml.
Add docstrings and more documentation.
Add documentation for the tests.
Added a page to the new documentation for pycalibration. But it needs to be updated after the latest changes. In the meantime, I have updated the description for this MR and added documentation in the code as well.
Enable manually triggering the tests from Gitlab.
Test checklist:
Review the reference test on the test node.
Decide where to keep the reference and test output folder.
