Script to validate YAML config against notebooks

Description

When we change the calibration notebooks, the YAML configuration used by the webservice sometimes needs to be modified as well, if we have removed, renamed or changed the type of parameters.

This script can check that config against the parameters in the correct & dark notebooks.

python -m xfel_calibrate.validate_nbs_config

Add --config-dir to specify the config folder path. The default should work for running as xcaltst.

How Has This Been Tested?

Run locally.

Relevant Documents (optional)

Sample output:

Detector Karabo ID not in data mapping
  AGIPD64K
  file: 202431/900463.yaml
  key: dark/SPB

Unknown detector type in data mapping
  AGIPD64K
  file: 202431/900465.yaml
  key: data-mapping/AGIPD64K/detector-type

Type differs from default in notebook
  notebook: 'off', config: False
  file: 202431/900465.yaml
  key: correct/MID/MID_DET_AGIPD1M-1/use-litframe-finder

Type differs from default in notebook
  notebook: 'cm', config: False
  file: 202431/900465.yaml
  key: correct/MID/MID_DET_AGIPD1M-1/use-super-selection

Parameter name not found in notebook
  path_inset
  file: 202431/900465.yaml
  key: dark/DETLAB/SCS_CDIDET_FCCD2M

It currently finds 1113 config issues, although a lot of these are essentially the same issue in various files. The 'agipd64k' detector type comes up quite a few times.

Types of changes

• New feature (non-breaking change which adds functionality)
• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.

Reviewers

@ahmedk @roscar @schmidtp

Nice and fast!

would it be possible to have this running as a part of the integration tests as well? Additional step for the tests detectors. Assuming it doesn't take very long to validate the configs as it can be doing the config check multiple times for the same detector in the integration test

We are already doing the tests on xcaltst. Can the containers access calibration_configuration in the xcaltst home directory?

With the 'trimmed' config, we're now down to 295 issues. detectors/calibration_configurations!34 should be useful in cleaning these up.

added 1 commit

• 584f478c - Make some quantity parameters for conditions floats instead of ints

Compare with previous version

Now at 84 issues, with the fixes here and in detectors/calibration_configurations!34.

added 44 commits

• 584f478c...1b39698e - 40 commits from branch master
• adb861ec - Script to validate YAML config against notebooks
• bafe6724 - --detector option to select detector type
• 6aa5248d - Show reprs for mismatched types instead of classes
• 2de0751a - Make some quantity parameters for conditions floats instead of ints

Compare with previous version

added 19 commits

• 2de0751a...8679274c - 15 commits from branch master
• 8847aef9 - Script to validate YAML config against notebooks
• a9e8c99b - --detector option to select detector type
• a329b610 - Show reprs for mismatched types instead of classes
• 8e00283f - Make some quantity parameters for conditions floats instead of ints

Compare with previous version

mentioned in merge request !1099

0 config issues found

added 1 commit

• 6f3a81a1 - Check notebooks against config repo in CI

Compare with previous version

added 1 commit

• ecfb937f - Clone calibration_configurations using deploy token

Compare with previous version

added 1 commit

• a4495a08 - Fix git URL for calibration_configurations repo

Compare with previous version

added 137 commits

• a4495a08...7c6127d1 - 130 commits from branch master
• dfaf6d84 - Script to validate YAML config against notebooks
• 463aa549 - --detector option to select detector type
• a95f106b - Show reprs for mismatched types instead of classes
• 62631a5d - Make some quantity parameters for conditions floats instead of ints
• 191a9124 - Check notebooks against config repo in CI
• 00055707 - Clone calibration_configurations using deploy token
• a01c5a2f - Fix git URL for calibration_configurations repo

Compare with previous version

OK, the check is now running & passing in CI; I think this is ready for review.

LGTM!

I'm wondering how we could maximize the convenience of this script for when people do manual edits.

resolved all threads

merged

changed milestone to %3.16.0

mentioned in commit 3976ee21