Use status AW in myMdC if correction failed for some detectors in a run

Description

We want to provide a clearer status when there are multiple detectors in a run and correction has failed for some but not all of them. The status 'AW' means 'available with warning'.

While trying to test this, I also noticed a problem with my recent changes to how we detect jobs are finished: we were checking the abbreviated Slurm state (e.g. CD) against the names of full states (COMPLETED), so I've tried to make this consistent. I can split this out if needed

How Has This Been Tested?

Deployed on max-exfl017, tried with myMdC test instance with p900113, r9981.

Relevant Documents (optional)

Types of changes

• Bug fix (arguably)

Checklist:

• My code follows the code style of this project.

Reviewers

@schmidtp @ahmedk @roscar

webservice/job_monitor.py

@@ -27,6 +27,10 @@ STATES_FINISHED = {  # https://slurm.schedmd.com/squeue.html#lbAG
     'BOOT_FAIL',  'CANCELLED', 'COMPLETED',  'DEADLINE', 'FAILED',
     'OUT_OF_MEMORY', 'SPECIAL_EXIT', 'TIMEOUT',
 }
+STATE_ABBREVS = {
+    'PENDING': 'PD',
+    'RUNNING': 'R',
+}
 
 
 class NoOpProducer:
@@ -55,7 +59,7 @@ def slurm_status(filter_user=True):
     :return: a dictionary indexed by slurm jobid and containing a tuple
              of (status, run time) as values.
     """
-    cmd = ["squeue", "--states=all"]
+    cmd = ["squeue", "--states=all", "--format=%i %T %M"]
     if filter_user:
         cmd += ["--me"]
     res = run(cmd, stdout=PIPE, stderr=PIPE)
@@ -64,7 +68,7 @@ def slurm_status(filter_user=True):
         statii = {}
         for r in rlines[1:]:
             try:
-                jobid, _, _, _, status, runtime, _, _ = r.split()
+                jobid, status, runtime = r.split()
                 jobid = jobid.strip()
                 statii[jobid] = status, runtime
             except ValueError:  # not enough values to unpack in split
@@ -179,13 +183,14 @@ class JobsMonitor:
             if str(r['job_id']) in statii:
                 # statii contains jobs which are still going (from squeue)
                 slstatus, runtime = statii[str(r['job_id'])]
-                execn_ongoing_jobs.append(f"{slstatus}-{runtime}")
-
             else:
                 # These jobs have finished (successfully or otherwise)
                 _, runtime, slstatus = slurm_job_status(r['job_id'])
 
             finished = slstatus in STATES_FINISHED
+            if not finished:
+                short_state = STATE_ABBREVS.get(slstatus, slstatus)
+                execn_ongoing_jobs.append(f"{short_state}-{runtime}")
 
             updates.append((finished, runtime, slstatus, r['job_id']))
 
@@ -295,7 +300,12 @@ class JobsMonitor:
         log.debug("Update MDC for %s, %s: %s", r['action'], r['mymdc_id'], msg)
 
         if r['action'] == 'CORRECT':
-            status = 'A' if success else 'E'  # Available/Error
+            if success:
+                status = 'A'  # Available
+            elif set(krb_id_successes.values()) == {0, 1}:
+                status = 'AW' # Available with Warning (failed for some detectors)
+            else:
+                status = 'E'  # Error
             self.mymdc_update_run(r['mymdc_id'], msg, status)
         else:  # r['action'] == 'DARK'
             status = 'F' if success else 'E'  # Finished/Error

[Thomas Kluyver]

%T gives us the full state name, e.g. 'COMPLETED' instead of 'CD'.

[Thomas Kluyver]

This translates the most common states we'll see back to their short forms: PD & R. We have a .startswith('PD-') below to skip updates when all jobs are pending, so it's not entirely cosmetic.