[AGIPD][TESTs]test_agipdlib AGIPDCtrl and get_bias_voltage for AGIPD1M and AGIPD500K

Based on https://git.xfel.eu/detectors/pycalibration/-/merge_requests/651

• Test AGIPDCtrl() methods to read conditions from Fast or slow data.
• differentiate between AGIPD1M and AGIPD500K sources and keys to read bias_voltage. (temporary unbreakable change.)

Description

Latest changes to agipdlib had a class AGIPDCtrl() created collecting all methods to read AGIPD parameter conditions from fast and slow data.

1. I have added tests for these methods
2. I have split some methods based on their functionality.
3. As bias_voltage is stored in different paths for AGIPD1M and AGIPD500K, a condition was added to select the right source.

• This is a temporary change and I think one should give the correct source and key through get_bias_voltage arguments.

4. All of the tests are based on EXtra-data mocked data. I have added mock_agipd1m_run, mock_agipd500K_run and mock_agipd1m_run_old at :

How Has This Been Tested?

Unit tests.

Relevant Documents (optional)

Types of changes

• Test (additional or refactored tests)

Checklist:

• My code follows the code style of this project.
• I have updated the documentation accordingly.
• I added tests where appropriate.

Reviewers

@calibration

changed title from get_bias_voltage for AGIPD1M and AGIPD500K and test_agipdlib AGIPDCtrl to [AGIPD][TESTs]test_agipdlib AGIPDCtrl and get_bias_voltage for AGIPD1M and AGIPD500K

added On hold label

changed the description

added 1 deleted label

mentioned in merge request !647 (merged)

changed target branch from master to refactor/split_AGIPDCtrl_functions

marked this merge request as draft

marked this merge request as ready

marked this merge request as draft

mentioned in merge request !651 (merged)

added 4 commits

• c401b3bb...5dceb0c9 - 2 commits from branch refactor/split_AGIPDCtrl_functions
• 352f6480 - get_bias_voltage for AGIPD1M and AGIPD500K and test_agipdlib AGIPDCtrl
• 42ab4439 - account for old run with no bias_voltage

Compare with previous version

added 1 commit

• b4cdc1f9 - add a temporary extra-data commit for running the tests

Compare with previous version

added 1 commit

• 5b5f89e6 - add a temporary extra-data commit for running the tests

Compare with previous version

added 7 commits

• 5b5f89e6...ddca1463 - 2 commits from branch refactor/split_AGIPDCtrl_functions
• b159cdc2 - use get_run_value instead of as_single_value and split functions for adding tests later
• 08ad9193 - get_bias_voltage for AGIPD1M and AGIPD500K and test_agipdlib AGIPDCtrl
• 94e71a2c - account for old run with no bias_voltage
• 3603b55f - add a temporary extra-data commit for running the tests
• 0b61553c - remove duplicated test and fix same function names

Compare with previous version

added 6 commits

• 69cc4336 - 1 commit from branch refactor/split_AGIPDCtrl_functions
• b3d975b0 - use get_run_value instead of as_single_value and split functions for adding tests later
• 6919c2d0 - get_bias_voltage for AGIPD1M and AGIPD500K and test_agipdlib AGIPDCtrl
• 49494d7d - account for old run with no bias_voltage
• 4ab0e931 - add a temporary extra-data commit for running the tests
• 9df238d8 - remove duplicated test and fix same function names

Compare with previous version

added 1 commit

• b0144934 - remove duplicated test and fix same function names

Compare with previous version

added 1 commit

• 11c196b9 - remove duplicated test and fix same function names

Compare with previous version

added 165 commits

• 11c196b9...1df6b681 - 59 commits from branch refactor/split_AGIPDCtrl_functions
• b5d0586c - Hopefully speed up baseline correction by stripe method
• 11c90a82 - Preserve behaviour of ignoring first and last shadowed row
• 58f65955 - Exclude first & last pixel of each shadowed stripe
• 8e48d290 - Allow resetting myMdC by proposal number & (range of) run numbers
• dbe7c0bd - Merge branch 'reset-mdc-by-prop-run-no' into 'master'
• 83adcfa9 - Less verbose log messages when orca is not running
• f0c63201 - Remove unused exception 'as' clauses
• 2690fd2b - Merge branch 'log-orca-connecterr-shh' into 'master'
• 6270b092 - Separate Slurm job monitoring in its own process
• 8ca53b93 - Use squeue --me option
• fa9d755b - Use module specific logger in more places
• 0c6e9c10 - argparse already does this if passed None
• 0d340834 - Merge branch 'feat/faster-blc-stripe' into 'master'
• 73db1e4b - Replace agipdalgs.gain_choose_int by fused type
• 6510d4d5 - Store calibration_metadata.yml primarily in work directory
• 93dac475 - Merge branch 'calmeta-in-work-dir' into 'master'
• efe2c2c0 - Rename mdc_config to restful_config and add CalCat credentials
• c23dfc1f - Add native implementation for LPD correction loop
• f40e41aa - Add low-level API for writing in EuXFEL data file format
• 00d09596 - Add new LPD correct notebook and use it for corrections
• 959066f7 - Copy calibration_metadata.yml file to output directory instead of symlinking
• b42f4478 - Use clearer symbol names in lpdalgs
• 4c60cc43 - Fix selection of gain in lpdalgs
• 14bd0d49 - Improve plotting details after gain fix
• f58affa3 - Add calibration_client dependency for direct CalCat access
• 91165e9c - Correct default values for list parameters in new LPD correct
• e6d0a1b9 - Merge branch 'copy-calmeta' into 'master'
• a68fb967 - Merge branch 'feat/calcat-credentials' into 'master'
• 5310b718 - Small clean-ups from MR discussion
• c74fca49 - Pass str, not Path, to shutil.move() for now
• 33142d42 - Merge branch 'shutil-move-str' into 'master'
• 28ad3538 - Check for transfer conditional on resolved symlink path
• 5fd234e5 - Make wait_transfers/wait_on_transfers easier to test
• b9084f70 - Update tests for wait_on_transfers
• a484761b - Specifically check if gpfs offline, add tests
• a63295f6 - Rename proposal_dir to in_folder for consistency
• e0acbd08 - Remove unnecessary mocking
• 446c2266 - Simplify tests
• 578110e0 - Apply 2 suggestion(s) to 2 file(s)
• e20c912f - Simplify checks
• 609350a3 - Symlinks for `raw` dir not run dir
• aa9f03f3 - Apply 1 suggestion(s) to 1 file(s)
• d3049246 - Return None if resolved path parents not for
• 6f4ebc04 - Add test for invalid run dir, test log output
• 7333897d - Fix `test_wait_on_transfer_exceptions` test
• eb298ba2 - Apply 1 suggestion(s) to 1 file(s)
• 373c80b2 - Apply 1 suggestion(s) to 1 file(s)
• 11ff26ec - Cast run to int, since it's str, correct type hint
• 9040acd1 - Improve xfel_calibrate process error logging
• 3175af8d - Update tests to mock stderr from run_proc_async
• 9eea3ef9 - Apply 1 suggestion(s) to 1 file(s)
• 9f1e2201 - Apply 1 suggestion(s) to 1 file(s)
• 2330824f - Scan for files to be processed directly without EXtra-data
• c17d7ff5 - Merge branch 'fix/migration-check-repo-only-offline' into 'master'
• 3ce8f69e - Make constant axis ordering configurable and adjust based on calibration type
• 267ae8a3 - Move cell axis to the front front in constants to accelerate native LPD correction code
• 5e21ac50 - Remove numpy headers and add gcc performance options to lpdalgs
• c354dd25 - [JUNGFRAU][CORRECT] Create/Expose cell_id_preview parameter
• c23f834a - Minor text and typo fixes in cal_tools.files
• 59d5be35 - Drop option of manual train offsets in InstrumentSource.create_index
• 330a6ff8 - More consistent naming and comments in LPD correct
• 79abd4e6 - Add chunking to LPD correct output data
• 660b4da5 - Improve default intensity binning in LPD correct
• c6078819 - collect cell_id_preview validation and cell_idx_preview assignment in one place
• 9af7f981 - Add legacy calDBRemote parameters access for webserver compatibility
• 2049c425 - Reorder writes in LPD correct for faster access
• 221d02e5 - Merge branch 'feat/select_cell_id_preview' into 'master'
• 43b6eb6b - Merge branch 'feat/LPD-revival' into 'master'
• 4dd02053 - Merge branch 'feat/gain-choose-fused-type' into 'master'
• e2d2f621 - Ensure restful_config.yaml gets installed
• 520574f9 - Add use_oauth2 option to allow using proxy without oauth
• 37d19b4a - Add creation of virtual CXI files to LPD correction
• 93f1edc8 - Set default chunk_data = 1 for more efficient reading
• 469dc05a - Temporarily revert OAuth2-related changes for notebook hotfix
• 485f6c8d - Merge branch 'lpd-make-virt-cxi-chunkwise' into 'master'
• d876e20f - Create virtual CXI files by sequence rather than by job chunk
• 1055cbb9 - Expose max-nodes parameter in LPD correct notebook
• 18f1004d - Apply 3 suggestion(s) to 1 file(s)
• 331d7eeb - Merge branch 'feat/LPD-max-nodes-parameter' into 'master'
• ba9e995b - Ensure restful_config.yaml gets installed as part of the package
• ede65ba8 - Merge branch 'cal-tools-pkg-data' into 'master'
• 20dab91b - Add run and proposal_folder placeholder in create_virtual_cxi_in argument
• fa628d20 - Merge branch 'feat/LPD-cxi-per-sequence' into 'master'
• 660dbe63 - Add job monitor logs to serve_overview
• 9bcd5077 - Fix trying to use report-to name for karabo-id when report is skipped
• 9a1eacd3 - Merge branch 'fix/xfel-calibrate-no-report-to' into 'master'
• 4d2db89c - Fix key in config name
• e17cb7d4 - Merge branch 'separate-job-monitor' into 'master'
• 512d5820 - Always convert bias_voltage to integer
• d23db4a4 - Merge branch 'fix/bias_voltage_integer' into 'master'
• 2f2edfd3 - Fix mangling of pulse axis for worksets with only a single train in LPD correct
• 922ec365 - Exclude trains with <4 modules from LPD VCXI files
• cf4de1f3 - Exclude trains with flag=0 from output file for LPD
• c83fc0a4 - Catch exception from creating LPD1M object
• 31882ca3 - Merge branch 'fix/LPD-correct-single-pulse' into 'master'
• 246dd3d0 - _uscancel e_voview to be able e to use .files
• 84f737cb - lose .files from AGIPD dark and FF
• 6d5a02b4 - Merge branch 'fix/Update_extra_data_version' into 'master'
• 9ddc981d - Merge branch 'lpd-vcxi-require-4-mods' into 'master'
• d9d432ee - use get_run_value instead of as_single_value and split functions for adding tests later
• 734cdfba - fix function names
• cc6e6a2a - use get_run_value instead of as_single_value and split functions for adding tests later
• bca16383 - get_bias_voltage for AGIPD1M and AGIPD500K and test_agipdlib AGIPDCtrl
• e31e4acc - account for old run with no bias_voltage
• 01b97ecf - remove duplicated test and fix same function names
• d5b016a7 - comment out bias_voltage old SPB run test until it is fixed

Compare with previous version

removed On hold label