Reduce API Requests
This Merge request is related to issue: https://git.xfel.eu/gitlab/detectors/calibration_workshop/issues/120
This merge request helps in avoiding the retrieval of the same constants in each parallel node (corresponding to different sequences) and overloading the database interface with requests.
- A leading notebook retrieving the required constants and saving the metadata of these constants in a .yaml file.
- The main generated notebooks by xfel-calibrate corresponding to different sequences have slurm job dependency waiting for the constants to be retrieved.
- The constants than are loaded from the .yaml file which contains the creation-time and the constant file path.
- after successful execution the .yaml file is moved to the temp folder which is later moved with the report for reproducibility.
This code runs with having a leading notebook to retrieve the constants or without any yaml files as before.
This is a report for processing files using this merge request AGIPDOfflineCorrection.pdf
Edited by Karim Ahmed