Draft: Rough sketch of injecting new constants after report creation

This is a sketch of an idea we discussed last week, to inject new constants after the notebooks have run, and the PDF report has been generated. With this in place, we would adapt the dark notebooks (or other characterisations run through the webservice) to just write constants with all the necessary metadata in their output directory, and remove the db_output parameter. Then the finalize job would find these constants and inject them.

Caveats:

1. We can avoid some CalCat API calls when injecting a group of constants all together, but I haven't done this yet. I don't think it fundamentally changes the logic.
2. This doesn't aim to support any manual review before injection - the newly generated constants are still injected straight away, just a bit later in the pipeline.
3. It means a couple more minutes delay before new constants are available - the main notebooks finish, the summary notebook runs, and the report is built. We could move injection before the report building to mitigate that a bit, but it would still be later than it currently is.
4. This is only meant for constants generated via the webservice (jobs running as xcal); we'd still need an injector service for manually generated constants before we could fully replace calDBRemote.

src/xfel_calibrate/calibrate.py

@@ -207,7 +207,9 @@ def create_finalize_script(fmt_args, cal_work_dir, job_list) -> str:
                              report_to='{{report_to}}',
                              data_path='{{in_folder}}',
                              request_time='{{request_time}}',
-                             submission_time='{{submission_time}}')
+                             submission_time='{{submission_time}}',
+                             inject={{inject}},
+                            )
 
                     """)
 
@@ -850,6 +852,13 @@ def run(argv=None):
         ))
         errors = False
 
+    # If the notebook has a db_output parameter, assume it will do the injection
+    # itself.
+    inject = args['db_output'] and 'db_output' not in default_params_by_name
+    if inject and nb_details.caltype == 'CORRECT':
+        print("Ignoring --db-output for CORRECT task (this is for characterisation")
+        inject = False
+
     fmt_args = {'cal_work_dir': cal_work_dir,
                 'out_path': out_path,
                 'version': version,
@@ -859,6 +868,7 @@ def run(argv=None):
                 'in_folder': folder,
                 'request_time': request_time.isoformat(),
                 'submission_time': submission_time.isoformat(),
+                'inject': inject,
                 }
 
     joblist.append(run_finalize(