FIX: Add microseconds to datetime part in tempfolder name

Review changes
Download
Patches
Plain diff

Merged FIX: Add microseconds to datetime part in tempfolder name

fix/temp_folder_already_exists into master

Overview 3
Commits 1
Pipelines 0
Changes 1

Merged Karim Ahmed requested to merge fix/temp_folder_already_exists into master 3 years ago

Overview 3
Commits 1
Pipelines 0
Changes 1

Requesting the same two calibrations for the same detector type will result in an error.

Traceback (most recent call last):
  File "/home/xcal/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/xcal/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/xcal/deployments/development/git.xfel.eu/detectors/pycalibration/pycalibration-2021-11-09-3.4.7-e4710d2/src/xfel_calibrate/calibrate.py", line 1267, in <module>
    sys.exit(run())
  File "/home/xcal/deployments/development/git.xfel.eu/detectors/pycalibration/pycalibration-2021-11-09-3.4.7-e4710d2/src/xfel_calibrate/calibrate.py", line 1037, in run
    os.makedirs(run_tmp_path)
  File "/home/xcal/.pyenv/versions/3.8.11/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/xcal/deployments/development/git.xfel.eu/detectors/pycalibration/pycalibration-2021-11-09-3.4.7-e4710d2/temp/slurm_out_JUNGFRAU_DARK_t220203_103303'

Description

Automated pycalibration tests

While working on https://git.xfel.eu/detectors/pycalibration/-/merge_requests/610, after running multiple dark agipd xfel-calibrate command lines at the same time. FileExistsError was faced.

Dark processing at production

While requesting darks for multiple jungfrau modules in proposal 900247. an error was faced for random requests. After re-requesting the same dark, it worked fine.

Looking at the web.log there was no information except a FAILED command line. and using caldeploy logs one could see the mentioned FileExistsError.

I gather, if multiple dark processing were requested for the same detector type. But the selected runs have not been migrated, yet. As soon as they get migrated. These multiple requests will be pointing to the same temp folder and hence the FileExistsError.

2022-02-03 10:33:07,587 - root - INFO - [webservice.py:550] SUCCESS: Started dark characterization: proposal 900247, run 40
2022-02-03 10:33:05,063 - root - INFO - [webservice.py:510] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:32:04 --slurm-name dark_HED_jungfrau_202230_p900247_r38_r39_r40 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K3/dark/dark_900247_r38_r39_r40_220203_103305 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR03 --db-module Jungfrau_M242 --receiver-id JNGFR03 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_38_39_40/HED_IA1_JF500K3 --karabo-id HED_IA1_JF500K3 --operation-mode ADAPTIVE_GAIN --run-high 38 --run-med 39 --run-low 40
2022-02-03 10:33:05,062 - root - INFO - [webservice.py:1035] Set `in_folder` to /gpfs/exfel/exp/HED/202230/p900247/raw
2022-02-03 10:33:05,055 - root - INFO - [webservice.py:638] Transfer complete: proposal 900247, runs ['38', '39', '40']
2022-02-03 10:33:04,976 - root - INFO - [webservice.py:551] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:32:51 --slurm-name dark_HED_jungfrau_202230_p900247_r38_r39_r40 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K2/dark/dark_900247_r38_r39_r40_220203_103301 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR02 --db-module Jungfrau_M266 --receiver-id JNGFR02 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_38_39_40/HED_IA1_JF500K2 --karabo-id HED_IA1_JF500K2 --operation-mode ADAPTIVE_GAIN --run-high 38 --run-med 39 --run-low 40
2022-02-03 10:33:04,976 - root - INFO - [webservice.py:550] SUCCESS: Started dark characterization: proposal 900247, run 40
2022-02-03 10:33:04,028 - root - ERROR - [webservice.py:513] FAILED: Failed executing command: ['python', '-m', 'xfel_calibrate.calibrate', 'jungfrau', 'DARK', '--concurrency-par', 'karabo_da', '--slurm-scheduling', '10', '--slurm-partition', 'upex-high', '--request-time', '2022-02-03T10:32:28', '--slurm-name', 'dark_HED_jungfrau_202230_p900247_r38_r39_r40', '--report-to', '/gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K1/dark/dark_900247_r38_r39_r40_220203_103259', '--cal-db-interface', 'tcp://max-exfl016:8015#8044', '--db-output', '--karabo-da', 'JNGFR01', '--db-module', 'Jungfrau_M039', '--receiver-id', 'JNGFR01', '--karabo-da-control', 'JNGFRCTRL00', '--receiver-control-id', 'CONTROL', '--in-folder', '/gpfs/exfel/exp/HED/202230/p900247/raw', '--out-folder', '/gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_38_39_40/HED_IA1_JF500K1', '--karabo-id', 'HED_IA1_JF500K1', '--operation-mode', 'ADAPTIVE_GAIN', '--run-high', '38', '--run-med', '39', '--run-low', '40'], code: 1, please contact det-support@xfel.eu
2022-02-03 10:33:01,727 - root - INFO - [webservice.py:510] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:32:51 --slurm-name dark_HED_jungfrau_202230_p900247_r38_r39_r40 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K2/dark/dark_900247_r38_r39_r40_220203_103301 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR02 --db-module Jungfrau_M266 --receiver-id JNGFR02 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_38_39_40/HED_IA1_JF500K2 --karabo-id HED_IA1_JF500K2 --operation-mode ADAPTIVE_GAIN --run-high 38 --run-med 39 --run-low 40
2022-02-03 10:33:01,725 - root - INFO - [webservice.py:1035] Set `in_folder` to /gpfs/exfel/exp/HED/202230/p900247/raw
2022-02-03 10:33:01,719 - root - INFO - [webservice.py:638] Transfer complete: proposal 900247, runs ['38', '39', '40']
2022-02-03 10:32:59,365 - root - INFO - [webservice.py:510] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:32:28 --slurm-name dark_HED_jungfrau_202230_p900247_r38_r39_r40 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K1/dark/dark_900247_r38_r39_r40_220203_103259 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR01 --db-module Jungfrau_M039 --receiver-id JNGFR01 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_38_39_40/HED_IA1_JF500K1 --karabo-id HED_IA1_JF500K1 --operation-mode ADAPTIVE_GAIN --run-high 38 --run-med 39 --run-low 40
2022-02-03 10:32:59,364 - root - INFO - [webservice.py:1035] Set `in_folder` to /gpfs/exfel/exp/HED/202230/p900247/raw
2022-02-03 10:32:59,346 - root - INFO - [webservice.py:638] Transfer complete: proposal 900247, runs ['38', '39', '40']
2022-02-03 10:32:54,979 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (5/300)
2022-02-03 10:32:51,662 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (0/300)
2022-02-03 10:32:51,343 - root - INFO - [webservice.py:792] Handling request for action dark_request
2022-02-03 10:32:51,330 - root - INFO - [webservice.py:792] Handling request for action check_connection
2022-02-03 10:32:49,281 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (2/300)
2022-02-03 10:32:44,918 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (4/300)
2022-02-03 10:32:39,221 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (1/300)
2022-02-03 10:32:34,830 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (3/300)
2022-02-03 10:32:29,160 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (0/300)
2022-02-03 10:32:28,854 - root - INFO - [webservice.py:792] Handling request for action dark_request
2022-02-03 10:32:28,839 - root - INFO - [webservice.py:792] Handling request for action check_connection
2022-02-03 10:32:24,732 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (2/300)
2022-02-03 10:32:14,673 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (1/300)
2022-02-03 10:32:14,436 - root - INFO - [webservice.py:608] Proposal 900247 run 39 not migrated yet. Will try again (1/300)
2022-02-03 10:32:04,376 - root - INFO - [webservice.py:608] Proposal 900247 run 40 not migrated yet. Will try again (0/300)
2022-02-03 10:32:04,335 - root - INFO - [webservice.py:608] Proposal 900247 run 39 not migrated yet. Will try again (0/300)
2022-02-03 10:32:04,037 - root - INFO - [webservice.py:792] Handling request for action dark_request
2022-02-03 10:32:04,022 - root - INFO - [webservice.py:792] Handling request for action check_connection
2022-02-03 10:16:23,997 - root - INFO - [webservice.py:414] Jobs finished - action: DARK, run id: 1539, status: A
2022-02-03 10:14:23,159 - root - INFO - [webservice.py:414] Jobs finished - action: DARK, run id: 1537, status: A
2022-02-03 10:14:23,082 - root - INFO - [webservice.py:414] Jobs finished - action: DARK, run id: 1538, status: A
2022-02-03 10:14:23,008 - root - INFO - [webservice.py:414] Jobs finished - action: DARK, run id: 1536, status: A
2022-02-03 10:14:22,897 - root - INFO - [webservice.py:414] Jobs finished - action: DARK, run id: 1534, status: A
2022-02-03 10:12:58,382 - root - INFO - [webservice.py:551] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:12:55 --slurm-name dark_HED_jungfrau_202230_p900247_r34_r35_r36 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K2/dark/dark_900247_r34_r35_r36_220203_101255 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR02 --db-module Jungfrau_M266 --receiver-id JNGFR02 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_34_35_36/HED_IA1_JF500K2 --karabo-id HED_IA1_JF500K2 --operation-mode ADAPTIVE_GAIN --run-high 34 --run-med 35 --run-low 36
2022-02-03 10:12:58,382 - root - INFO - [webservice.py:550] SUCCESS: Started dark characterization: proposal 900247, run 36
2022-02-03 10:12:55,824 - root - INFO - [webservice.py:510] python -m xfel_calibrate.calibrate jungfrau DARK --concurrency-par karabo_da --slurm-scheduling 10 --slurm-partition upex-high --request-time 2022-02-03T10:12:55 --slurm-name dark_HED_jungfrau_202230_p900247_r34_r35_r36 --report-to /gpfs/exfel/d/cal/caldb_store/xfel/reports/HED/HED_IA1_JF500K2/dark/dark_900247_r34_r35_r36_220203_101255 --cal-db-interface tcp://max-exfl016:8015#8044 --db-output --karabo-da JNGFR02 --db-module Jungfrau_M266 --receiver-id JNGFR02 --karabo-da-control JNGFRCTRL00 --receiver-control-id CONTROL --in-folder /gpfs/exfel/exp/HED/202230/p900247/raw --out-folder /gpfs/exfel/u/usr/HED/202230/p900247/CalOutput/dark/runs_34_35_36/HED_IA1_JF500K2 --karabo-id HED_IA1_JF500K2 --operation-mode ADAPTIVE_GAIN --run-high 34 --run-med 35 --run-low 36