Skip to content

Only start ipcluster if the notebook uses cluster_profile

Thomas Kluyver requested to merge refine-ipcluster into master

AGIPD correction now runs three different notebooks: one "pre", the main one performing the correction (run in several copies), and one "dep" notebook to summarise the results. In master, we either start ipcluster for all of these or none of them, depending on whether the main notebook has a cluster_profile parameter.

We have switched the main notebook to use multiprocessing, but the "pre" notebook still uses ipcluster for now. This change starts ipcluster only if the specific notebook for that job wants it.

It's not a particular problem to start ipcluster and then not use it, but:

  • It's extra clutter in logs and running processes, which can distract someone trying to debug a problem.
  • We sleep 15 after starting ipcluster, presumably to give it time to start properly. So any job which doesn't need ipcluster can save 15 seconds.

Merge request reports