[webservice] No longer recorrect files when output files seem to still be around
Description
ITDM would like to remove AGIPD proc data from older proposals which we consider as reproducible, but such runs are unfortunately accompanied by other detectors without this guarantee.
Since the recorrect webservice action via myMdC is anyway meant to be used for deleted data and not so much restoring prior, arbitrary versions, this MR changes the webservice to check for the existance of the first sequence file of the first aggregator, and skip recorrection if it exists.
How Has This Been Tested?
TBD
Types of changes
- Bug fix (non-breaking change which fixes an issue)
Reviewers
Merge request reports
Activity
1244 1244 prior_request_time = calmeta["runtime-summary"]\ 1245 1245 ["pipeline-steps"]["request-time"] 1246 karabo_id = calmeta["calibration-configurations"]["karabo-id"] 1246 1247 calargs = calmeta["calibration-configurations"] 1248 karabo_id = calargs["karabo-id"] 1249 karabo_da = calargs["karabo-da"] 1250 out_path = Path(calargs["out-folder"]) 1247 1251 except KeyError: 1248 1252 logging.warning("Did not find expected metadata in %s", 1249 1253 calmeta_pth, exc_info=True) 1250 else: 1254 continue 1255 1256 try: 1257 next(out_path.glob("*-R*-{karabo_da[0]}-S*.h5")) changed this line in version 2 of the diff
added 1 commit
- 444ba990 - No longer recorrect files when output files seem to still be around
1244 1244 prior_request_time = calmeta["runtime-summary"]\ 1245 1245 ["pipeline-steps"]["request-time"] 1246 karabo_id = calmeta["calibration-configurations"]["karabo-id"] 1246 1247 calargs = calmeta["calibration-configurations"] 1248 karabo_id = calargs["karabo-id"] 1249 karabo_da = calargs["karabo-da"] 1250 out_path = Path(calargs["out-folder"]) 1247 1251 except KeyError: 1248 1252 logging.warning("Did not find expected metadata in %s", 1249 1253 calmeta_pth, exc_info=True) 1254 continue 1255 1256 for da in karabo_da: 1257 try: 1258 next(out_path.glob(f"*-R*-{da}-S*.h5")) -
I just remembered that some of the configured karabo-da values aren't directly part of the filename. This is how the webservice currently handles that:
changed this line in version 3 of the diff
added 6 commits
-
444ba990...5fa9daa3 - 5 commits from branch
master - d2be78b2 - No longer recorrect files when output files seem to still be around
-
444ba990...5fa9daa3 - 5 commits from branch
1263 1279 ) 1264 1280 return msg.encode() 1265 1281 1282 if len(mddirs_by_krb_id) == 0: So I moved this one to give a nicer immediate response when there's nothing to do.
But it made me realize that we generally avoid file system access before the coroutine, exactly to have a quick response. It's possible we hit the timeout on myMdC because of that.
Should we move the entire aggregator check into
_coroutine?
