[Webservice] Restructure database to give more meaningful success/failure information

Description

At present, the success/failure information we send to myMdC is unreliable in various important ways - see calibration/planning#104 . In addition, the correction_complete events sent via Kafka are sent per detector, but they don't actually tell you which detector they relate to (they give a detector type, such as JUNGFRAU, but not the Karabo ID of a specific detector). Replacing the webservice still looks like a big enough job that it seemed worth spending a bit of time improving the old one. I hope this might also ease the transition to the new one when it's ready.

The jobs database is completely redesigned, with three tables instead of one: each request from myMdC can have several executions, each of which can have several slurm_jobs. I think these terms line up with what Robert is using in Orca. I haven't tried to do any migration of existing data - we're only using it to track active jobs, so I'm assuming we'll just start with an empty database when deploying this change.

The jobs monitor code is also largely refactored, with the loop broken up into several methods in a JobsMonitor class.

How Has This Been Tested?

Deployed on max-exfl017, and submitted correction & dark requests from the test myMdC instance for the CALLAB proposal.

This revealed a number of previously hidden issues which are (hopefully) specific to CALLAB. The presence of JNGFR01 in the filenames triggers an execution for both FXE_XAD_JF1M and SPB_IRDA_JF4M, at least one of which will fail on any given run (as it can't find its data). Previously, the webservice would hide this, because the failing jobs would fail quickly, and the status would only come from the last jobs running, but with these changes the failures are visible.

Relevant Documents (optional)

https://in.xfel.eu/test_metadata/proposals/259/runs/63612

https://in.xfel.eu/test_metadata/proposals/259/runs/86193

https://in.xfel.eu/test_metadata/proposals/259#proposal-calibration

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Refactor (refactoring code with no functionality changes)

Checklist:

• My code follows the code style of this project.

Reviewers

@ahmedk @roscar @schmidtp

webservice/job_monitor.py

@@ -231,10 +231,11 @@ class JobsMonitor:
 
     def process_request_finished(self, req_id):
         """Send notifications that a request has finished"""
-        exec_successes = {r[0] for r in self.job_db.execute(
-            "SELECT success FROM executions WHERE req_id = ?", (req_id,)
+        krb_id_successes = {r[0]: r[1] for r in self.job_db.execute(
+            "SELECT karabo_id, success FROM executions WHERE req_id = ?",
+            (req_id,)
         ).fetchall()}
-        success = (exec_successes == {1})
+        success = (set(krb_id_successes.values()) == {1})
 
         r = self.job_db.execute(
             "SELECT * FROM requests WHERE req_id = ?", (req_id,)
@@ -258,15 +259,11 @@ class JobsMonitor:
         if success:
             msg = "Calibration jobs succeeded"
         else:
-            # Count failed jobs
-            job_statuses = [r[0] for r in self.job_db.execute(
-                "SELECT slurm_jobs.status FROM slurm_jobs "
-                "INNER JOIN executions USING (exec_id) "
-                "INNER JOIN requests USING (req_id) "
-                "WHERE req_id = ?", (req_id,)
-            ).fetchall()]
-            n_failed = sum(s != 'COMPLETED' for s in job_statuses)
-            msg = f"{n_failed}/{len(job_statuses)} calibration jobs failed"
+            # List success & failure by karabo_id
+            krb_ids_ok = {k for (k, v) in krb_id_successes if v == 1}
+            ok = ', '.join(sorted(krb_ids_ok)) if krb_ids_ok else 'none'
+            krb_ids_failed = {k for (k, v) in krb_id_successes if v == 0}
+            msg = f"Succeeded: {ok}; Failed: {', '.join(krb_ids_failed)}"
 
         log.debug("Update MDC for %s, %s: %s", r['action'], r['mymdc_id'], msg)