fix/create_outfile_before_copying

This is a fix for the rsync error that was see in the webservice.nohup at xcal.

Description

The issue was that the outfolder creation is done only at the xfel-calibrate side. and as this is after starting copying files. We miss copying the files that were being copied before xfel-calibrate create a new directory.

How Has This Been Tested?

This was tested on xcaltst@max-exfl017

Relevant Documents (optional)

The error messages.

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

Reviewers

@danilevc @schmidtp

webservice/webservice.py

@@ -454,7 +454,7 @@ copy_blocklist_pattern = re.compile(
     r'\S*RAW-R\d{4}-(AGIPD|LPD|DSSC)\d{2}-S\d{5}.h5$')
 
 
-async def copy_untouched_files(file_list):
+async def copy_untouched_files(files_set):
     """ Copy those files which are not touched by the calibration
     to the output directory.
 
@@ -462,7 +462,13 @@ async def copy_untouched_files(file_list):
 
     Copying is done via an asyncio subprocess call
     """
-    for f in file_list:
+    if files_set:
+        Path(next(iter(files_set)).replace("raw", "proc")).parent.mkdir(
+            parents=True, exist_ok=True)
+    else:
+        return
+
+    for f in files_set:
         if copy_blocklist_pattern.match(f):
             continue
 
@@ -905,8 +911,8 @@ class ActionsServer:
                             thisconf["priority"] = str(priority)
 
                         detectors[karabo_id] = thisconf
-                copy_file_list = copy_file_list.difference(corr_file_list)
-                asyncio.ensure_future(copy_untouched_files(copy_file_list))
+                copy_file_set = copy_file_list.difference(corr_file_list)  # noqa
+                asyncio.ensure_future(copy_untouched_files(copy_file_set))
             except MigrationError as e:
                 logging.error("Migration issue", exc_info=e)
                 await update_mdc_status(self.mdc, 'correct', rid, str(e))