Skip to content

Reduce API Requests

Karim Ahmed requested to merge feat/reduce_constants_requests into master

This Merge request is related to issue: https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/120

This merge request helps in avoiding the retrieval of the same constants in each parallel node (corresponding to different sequences) and overloading the database interface with requests.

  1. A leading notebook retrieving the required constants and saving the metadata of these constants in a .yaml file.
  2. The main generated notebooks by xfel-calibrate corresponding to different sequences have slurm job dependency waiting for the constants to be retrieved.
  3. The constants than are loaded from the .yaml file which contains the creation-time and the constant file path.
  4. after successful execution the .yaml file is moved to the temp folder which is later moved with the report for reproducibility.

This code runs with having a leading notebook to retrieve the constants or without any yaml files as before.

This is a report for processing files using this merge request AGIPDOfflineCorrection.pdf

@karnem @kluyvert @tmichela @kamile @santosh

Edited by Karim Ahmed

Merge request reports