[Dark Request] Update MyMDC status
Description
The last of 3 MR to add MyMDC Darks support.
This is MR 3 of 3 to bring darks operation mode to xfel-calibrate (ticket).
In the first PR, webservice.py added support the new request parameters.
In the second PR, the notebooks were updated accordingly.
In this PR, an end to end refactoring will be done to clear temporary hacks and finalize MyMDC status updates.
This PR will also feature end to end testing.
This is temporarily pointing to the feat/209_2 branch, until merged into master
How Has This Been Tested?
Regression Tests
Dark
AGIPD1M@SPB - runs: [9992, 9991, 9990] cycle 202031, proposal 900113
[x] Success
[] Failed
Comments: report here: http://max-exfl017:8008//gpfs/exfel/d/cal_tst/caldb_store/xfel/reports/CALLAB/SPB_DET_AGIPD1M-1/dark/dark_900113_r9992_r9991_r9990_210130_194801.pdf
Correct
AGIPD1M@SPB - run 9992
[x] Success
[] Failed
Comments: Triggered from MyMDC, as it is normally done.
report here: http://max-exfl017:8008//gpfs/exfel/exp/CALLAB/202031/p900113/proc/r9992/SPB_DET_AGIPD1M-1_correct_900113_r9992_210201_173223.pdf
xfel-calibrate
Run calibration from the xfel-calibrate command line without the new --operation_mode argument.
AGIPD1M@SPB - runs: [9992, 9991, 9990] cycle 202031, proposal 900113
xfel-calibrate agipd DARK \
--concurrency-par karabo_da \
--slurm-name dark_CALLAB_agipd_202031_p900113_rr9992_r9991_r9990 \
--report-to /gpfs/exfel/data/scratch/xcaltst/out/no_new_flag \
--cal-db-interface tcp://max-exfl016:8015#8044 \
--receiver-id {}CH0 \
--karabo-id-control SPB_IRU_AGIPD1M1 \
--karabo-da-control AGIPD1MCTRL00 \
--h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
--in-folder /gpfs/exfel/exp/CALLAB/202031/p900113/raw \
--out-folder /gpfs/exfel/data/scratch/xcaltst/out/no_new_flag/SPB_DET_AGIPD1M_1 \
--karabo-id SPB_DET_AGIPD1M-1 \
--run-high 9992 \
--run-med 9991 \
--run-low 9990[x] Success
[] Failed
Comments: no report as script is expected to fail on older dataset
xfel-calibrate
Run calibration from the xfel-calibrate command line with the new --operation_mode argument.
AGIPD1M@SPB - runs: [9992, 9991, 9990] cycle 202031, proposal 900113
--concurrency-par karabo_da \
--slurm-name dark_CALLAB_agipd_202031_p900113_rr9992_r9991_r9990 \
--report-to /gpfs/exfel/data/scratch/xcaltst/out/new_flag \
--cal-db-interface tcp://max-exfl016:8015#8044 \
--receiver-id {}CH0 \
--karabo-id-control SPB_IRU_AGIPD1M1 \
--karabo-da-control AGIPD1MCTRL00 \
--h5path-ctrl /CONTROL/{}/MDL/FPGA_COMP \
--in-folder /gpfs/exfel/exp/CALLAB/202031/p900113/raw \
--out-folder /gpfs/exfel/data/scratch/xcaltst/out/new_flag/SPB_DET_AGIPD1M_1 \
--karabo-id SPB_DET_AGIPD1M-1 \
--operation-mode FIXED_GAIN \
--run-high 9992 \
--run-med 9991 \
--run-low 9990[x] Success
[] Failed
Comments: FIXED_GAIN is specified, to change the default ADAPTIVE_GAIN value,
although the FIXED_GAIN mode in the notebooks is not implemented.
no report as script is expected to fail on older dataset
Unit Tests
tests/test_webservice.py::test_file_check
Integration Tests
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| xcaltst & p2021 r1 Fixed Gain | Error | Pass | access rights not set |
| xcaltst & p2021 r1 Fixed Gain | Error flag MyMDC | Pass | Error flag and msg set |
| MDC migration time out (fake run) | Error | Pass | error logged |
| MDC migration time out (fake run) | Timeout flag MyMDC | Pass | Timeout flag and msg set |
| MDC Error state on broken input | Error | Pass | Error flag and msg set |
| Broken notebook | Error | Pass | Error flag and msg set |
| Slurm jobs exit anormaly or error | Error | Pass | Error flag and msg set |
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Input path set | GPFS path | Pass | |
| Output path set | GPFS path | Pass | |
| Report path set | GPFS path | Pass |
DarkRequest
AGIPD1M@SPB - runs: [9992, 9991, 9990] cycle 202031, proposal 900113 ADAPTIVE GAIN
[x] Success
[] Failed
Comments:
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
AGIPD1M@SPB - runs: [9992] cycle 202031, proposal 900113 FIXED GAIN
[x] Success
[] Failed
Comments:
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
JF1M@FXE - runs: [9999, 9998, 9997] ADAPTIVE GAIN
[x] Success
[] Failed
Comments: report here: /gpfs/exfel/d/cal_tst/caldb_store/xfel/reports/CALLAB/FXE_XAD_JF1M/dark/dark_900113_r9999_r9998_r9997_210202_154650.pdf
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
JF1M@FXE - runs: [9999] FIXED GAIN
[x] Success
[] Failed
Comments: report here: /gpfs/exfel/d/cal_tst/caldb_store/xfel/reports/CALLAB/FXE_XAD_JF1M/dark/dark_900113_r9999_210202_155117.pdf
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
Epix100@MID - runs: [9988]
[x] Success
[] Failed
Comments: report here: /gpfs/exfel/d/cal_tst/caldb_store/xfel/reports/CALLAB/MID_EXP_EPIX-1/dark/dark_900113_r9988_210202_153251.pdf
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
Epix10K@HED - runs: [9986]
[x] Success
[] Failed
Comments: report here: /gpfs/exfel/d/cal_tst/caldb_store/xfel/reports/CALLAB/HED_IA1_EPIX10K-1/dark/dark_900113_r9986_210202_150000/dark_900113_r9986_210202_150000.pdf
| Procedure | Expected Results | Pass/Fail | Actual Result/Comments |
|---|---|---|---|
| Submit job from MyMDC | Requested status | Pass | Requested status, req. msg. |
| Submit job to slurm | In Progress status | Pass | IP status, paths updated, slurm submission msg |
| Watchdog updates job state | Elapsed Time Updating | Pass | CP feedback msg. shows running time |
| All slurm job finish | Finished status | Pass | CP feedback msg. shows COMPLETED |
Types of changes
- New feature (non-breaking change which adds functionality)
Checklist
- My change requires a change to the documentation.