3.11.0

I thought of using milestone descriptions to record some steps in a centralized place.

Deployment

The deployment was done on 12.07 as planned: https://git.xfel.eu/calibration/pycalibration/-/tags/3.11.0 at least on the test, it was officially available on production on 13.07 after requesting DESY to open the DMZ ports on the new nodes.

The service overview links max-exfl016:8008 and max-exfl017:8008 are updated to max-exfl-cal001:8008 and max-exfl-cal002:8008
CalibrationDB Remote has been moved to the new nodes.

Tests notes:

https://git.xfel.eu/calibration/pycalibration/-/jobs/446102 The link for the test pipeline on xcaltst Gotthard2 was not included in the test

=========================== short test summary info ============================

[1] AGIPD - CORRECT

All corrections for AGIPD succeeded except one because of wrong test configuration. The wrong run was given.

SPB_DET_AGIPD1M-1-CORRECT-ADAPTIVE FAILED LOGS

FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[SPB_DET_AGIPD1M-1-CORRECT-ADAPTIVE] - ValueError: Selected sequences [0] are not available in /gpfs/exfel/exp/CALLAB/202130/p900203/raw/r0262

[2] Epix100 - DARK failed:

The reason is that the BadPixels calculation is changed after this MR

EPIX100 DARK FAILED LOGS

    FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[HED_IA1_EPX100-1-DARK] - AssertionError: HED_IA1_EPX100-1-DARK failure, while validating metadata for [PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/HED_IA1_EPX100-1/HED_IA1_EPX100-1-DARK/const_BadPixelsDark_ePix100_M16.h5')]
    assert 1 == 0
    +  where 1 = len([PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/HED_IA1_EPX100-1/HED_IA1_EPX100-1-DARK/const_BadPixelsDark_ePix100_M16.h5')])
    FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[MID_EXP_EPIX-2-DARK] - AssertionError: MID_EXP_EPIX-2-DARK failure, while validating metadata for [PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/MID_EXP_EPIX-2/MID_EXP_EPIX-2-DARK/const_BadPixelsDark_ePix100_M18.h5')]
    assert 1 == 0
    +  where 1 = len([PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/MID_EXP_EPIX-2/MID_EXP_EPIX-2-DARK/const_BadPixelsDark_ePix100_M18.h5')])

[2] LPD

DARK:

The reason is all modules failed with error AssertionError: daemonic processes are not allowed to have children ???

FXE_DET_LPD1M-1-DARK FAILED LOGS

    FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[FXE_DET_LPD1M-1-DARK] - assert 1 == 0

CORRECT

The reference files for this test was corrected without any constants. For some reason it didn't receive constants previously. As a result all 13 LPD files were not numerically validated.

FXE_DET_LPD1M-1-CORRECT FAILED LOGS

    FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[FXE_DET_LPD1M-1-CORRECT] - AssertionError: FXE_DET_LPD1M-1-CORRECT failure, while validating metadata for [PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD09-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD04-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD05-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD08-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD01-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD13-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD15-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD11-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD02-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD06-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD07-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD03-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD14-S00000.h5')]
    assert 13 == 0
    +  where 13 = len([PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD09-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD04-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD05-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD08-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD01-S00000.h5'), PosixPath('/gpfs/exfel/data/scratch/xcaltst/test/test/automated_notebooks_tests_pre_deployments/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-CORRECT/CORR-R9008-LPD13-S00000.h5'), ...])

Final test message:

=========== 5 failed, 28 passed, 14 warnings in 23393.80s (6:29:53) ============ Cleaning up project directory and file based variables 00:00 ERROR: Job failed: exit status 1

Follow-up fixes for LPD dark:

[LPD] [Dark] Don't use multiprocessing inside multiprocessing when opening run [LPD1M][Dark] Return None data for modules with 0 trains to skip processing

After using the CI automated test again on both MRs:

The LPD dark failed again.

This is ok and should be expected after this change for using EXtra-data to create darks from >1 sequence file. This of course will lead to different data values for LPD constants.

FXE_DET_LPD1M-1-DARK FAILED LOGS


=========================== short test summary info ============================
FAILED tests/test_reference_runs/test_pre_deployment.py::test_xfel_calibrate[FXE_DET_LPD1M-1-DARK] - AssertionError: ['data'] datasets contain different values for /gpfs/exfel/data/scratch/xcaltst/test/test/find_difference_by_default/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-DARK/const_BadPixelsDark_LPD_SM-2021-003.h5 and /gpfs/exfel/data/scratch/xcaltst/test/reference_folder/FXE_DET_LPD1M-1/FXE_DET_LPD1M-1-DARK/const_BadPixelsDark_LPD_SM-2021-003.h5
assert not ['data']
=========== 1 failed, 32 skipped, 14 warnings in 2518.06s (0:41:58) ============

Follow-ups expected:

Update the reference files for LPD dark on xcaltst
Update the reference files for Epix100 dark on xcaltst