[Webservice] Don't mark jobs as finished just because they disappear from squeue output

Description

We discovered some calibrations appeared to have failed in myMdC even though they succeeded. When we looked in the calibration webservice DB, we found jobs marked finished but with status 'RUNNING' or 'PENDING'. Failure is reported when finished jobs have a status other than COMPLETED.

My best guess is that jobs which are pre-empted and requeued briefly disappear from the output of squeue. The docs for squeue say that "If no state is specified then pending, running, and completing jobs are reported," which appears to exclude the REQUEUED state. Then when we look up these jobs with sacct, they are PENDING or RUNNING again.

This determines when a job is finished based on its state instead. I've collected all the states which I think should be final; this list is already used in sfollow. I think the list of non-final states would be slightly longer.

We discussed whether there should be a timeout after which we stop checking jobs. I haven't implemented this yet because it's extra complexity and I'm not sure of the details - e.g. do we start timing only once jobs leave PENDING state?

How Has This Been Tested?

TBD

Types of changes

• Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code follows the code style of this project.

Reviewers

@ahmedk @schmidtp

webservice/job_monitor.py

@@ -25,7 +25,7 @@ log = logging.getLogger(__name__)
 
 STATES_FINISHED = {  # https://slurm.schedmd.com/squeue.html#lbAG
     'BOOT_FAIL',  'CANCELLED', 'COMPLETED',  'DEADLINE', 'FAILED',
-    'OUT_OF_MEMORY', 'PREEMPTED', 'SPECIAL_EXIT', 'TIMEOUT',
+    'OUT_OF_MEMORY', 'SPECIAL_EXIT', 'TIMEOUT',
 }
 
 
@@ -55,7 +55,7 @@ def slurm_status(filter_user=True):
     :return: a dictionary indexed by slurm jobid and containing a tuple
              of (status, run time) as values.
     """
-    cmd = ["squeue"]
+    cmd = ["squeue", "--states=all"]
     if filter_user:
         cmd += ["--me"]
     res = run(cmd, stdout=PIPE, stderr=PIPE)