[Webservice] Monitor Slurm jobs in separate process

Description

Now that we have a process supervisor for the webservice & serve_overview process, it makes sense for this to be a separate process as well, rather than a thread in the webservice process (which was convenient when we launched it manually). The diff here is mostly just moving code that already exists into a separate file for clarity.

This means its logs will be visible separately, and the supervisor can restart the job monitor if it fails. The overview server (http://max-exfl016.desy.de:8008/ ) will no longer show log messages from job monitoring, which it currently does. We could add them as a separate block if needed, or use caldeploy logs to look at them.

A related change will be needed in the deployment tools.

How Has This Been Tested?

Run on max-exfl017, see comment below.

Types of changes

• Refactor (refactoring code with no functionality changes)

Checklist:

• My code follows the code style of this project.

Reviewers

@schmidtp @roscar

webservice/job_monitor.py

@@ -106,11 +106,11 @@ def update_job_db(config):
             c = conn.cursor()
             c.execute("SELECT * FROM jobs WHERE status IN ('R', 'PD', 'CG') ")
             combined = {}
-            log.debug("SLURM info {}".format(statii))
+            log.debug(f"SLURM info {statii}")
 
             for r in c.fetchall():
                 rid, jobid, proposal, run, status, _time, det, action = r
-                log.debug("DB info {}".format(r))
+                log.debug(f"DB info {r}")
 
                 cflg, cstatus, *_ = combined.setdefault((rid, action), (
                     [], [], proposal, run, det
@@ -121,7 +121,7 @@ def update_job_db(config):
                     c.execute(query, (slstatus, runtime, jobid))
 
                     cflg.append('R')
-                    cstatus.append("{}-{}".format(slstatus, runtime))
+                    cstatus.append(f"{slstatus}-{runtime}")
                 else:
                     _, sltime, slstatus = slurm_job_status(jobid)
                     query = "UPDATE jobs SET status=? WHERE jobid LIKE ?"