Only start ipcluster if the notebook uses cluster_profile

AGIPD correction now runs three different notebooks: one "pre", the main one performing the correction (run in several copies), and one "dep" notebook to summarise the results. In master, we either start ipcluster for all of these or none of them, depending on whether the main notebook has a cluster_profile parameter.

We have switched the main notebook to use multiprocessing, but the "pre" notebook still uses ipcluster for now. This change starts ipcluster only if the specific notebook for that job wants it.

It's not a particular problem to start ipcluster and then not use it, but:

• It's extra clutter in logs and running processes, which can distract someone trying to debug a problem.
• We sleep 15 after starting ipcluster, presumably to give it time to start properly. So any job which doesn't need ipcluster can save 15 seconds.

cc @ahmedk @danilevc @santosh

You've moved the dependency of the ip-cluster profile away to calibrate.py, thus making the notebook agnostic to parallelisation.

This incidentally also simplifies the setting up of the machinery.

added 6 commits

• 811cc07f...8230caaa - 5 commits from branch master
• 1f60aef3 - Merge branch 'master' into 'refine-ipcluster'

Compare with previous version

I think this was how it was supposed to be at some point, when each correction type was defined by a single notebook. I'm guessing that the addition of dep_notebooks and then pre_notebooks complicated things a bit.

The changes are sound, and it looks good to me. A second opinion is needed though :)

Thanks!

@ahmedk , I guess you might be the best person to give a second opinion. No particular urgency, though.

resolved all discussions

LGTM!

Thank you @kluyvert for the help and sorry for the delay.

Thanks @ahmedk

merged

mentioned in commit 46166970