Correction runners as friends

Status

Done with what I had in mind for now. Sorry that the diff is so big. At least, looking at git diff --numstat master, the individual too long files are a bit more balanced; some of the complexity of BaseCorrection is now kept in BaseKernelRunner and PreviewFriend. Also, detector-specific correction classes are much simpler.

Will do some more testing and figure out how to review.

Goal

Simplify interaction between correction device and kernel runner. To this end, must make the kernel runner smarter and more responsible for handling its relevant subschemata. Some plans:

• Device passes most correction schema (changes) along to runner which has reconfigure which looks at hash (before: device figures out everything, uses loads of special methods to configure runner)
  • This, in particular, means that the "correction flags" (in the end passed to correction kernel) live
• Device gets simpler interaction for running corrections and generating previews
  • Related to flags, now the kernel runner must itself figure out what to do
  • Before: host device tells it to load, correct, do previews, maybe correct again etc.
  • Medium-long term plan (after this MR): should allow concurrent processing, meaning stateful "load this, now process whatever you've loaded" is out
• Other things like specialized _load_constant_to_runner goes away because each runner has uniform load_constant interface

Input handler flow

One of the big changes is the flow of the input handler. This section explains a solution to the issues I was pondering earlier (left as quote below). Before, BaseCorrection would have the main input handler which would extract (most, though not always all) relevant data from the hash and handle (most) quirks we tend to encounter before handing over to subclass-specific process_data. However, these different process_data implementations had a lot of overlap. Two obvious options: keep them separate, encapsulating all the things they do in small reusable functions or unify them and provide hooks / utility functions to fill in per subclass. Currently trying out the latter:

• The input handler (now ON_DATA again*) is only in BaseCorrection
• It uses _get_data_from_hash (may be subclass-specific) to get all the relevant data out of the hash
• It asks the kernel runner about expected_input_shape (needed to fix up wrong axis order from DAQ)
• It asks the kernel runner about expected_output_shape (needed to prepare the next shmem output slot)
• It then passes the extracted data to the kernel runner's correct method

And similarly, the BaseKernelRunner hosts the overarching correct method, letting subclass define certain parts:

• correct in BaseKernelRunner gets all the data from _get_data_from_hash
• It uses _make_output_buffers (may be subclass-specific) to set up some buffers the kernel will write to
• It uses the subclass-specific _correct to actually do the correction
  • This _correct gets all the data and the output buffers, so most of _get_data_from_hash and _make_output_buffers goes here
• Then, previews are computed, with _preview_data_views (may be subclass-specific) providing the views of the relevant ndarrays to preview
  • This gets the image data, any additional data from _get_data_from_hash, and the output buffers from _make_output_buffers and can return arbitrary list of buffer views
  • Each buffer view returned should match a preview output of the device; this allow preview computation to just be applying the same preview reduction function on each view and write it out

Although an improvement to the previous version, I'm not pleased that the different parts are still so tightly coupled. Not sure how to avoid this.

*A long time ago, we switched from ON_DATA to ON_INPUT, partly because - in case we'd get irrelevant sources coming in - the latter would allow us to not read content of certain input hashes unless relevant to us. ON_DATA is simpler, however, and as I understand from Gero's description of the surrounding C++ code changes, we don't decide whether or not to read the input any longer (the C++ code reads hashes and just gives them to us), so the potential performance edge of ON_INPUT should be gone. And we don't need trainmatched data, so it's not important to line up hashes on same input event.

Follow-up work

I've removed the frame filter functionality for now for two reasons

• It complicates BaseCorrection, configuration, and shape handling
• It slices data out before it gets previewable; could be an issue for detector monitoring
  • OTOH, the operator may want the nanmean of only the relevant / lit frames; but frame filter cannot accommodate both use cases

I know that MID tends to use the frame filter, so I'd just add a trivial frame selection kernel with the same logic.

Deployment notes

Having restructured the preview output infrastructure anyway, I also tied it to more node-local subschemata. So now each preview output can have its own settings about how to reduce (it was kind of silly to, say, look at mean of corrected and then also have to get mean of gain map). With a node for each output, preview.[preview_name], I figured we could move the outputs there, so preview.[preview_name].output. This means that managers all over will need an update to the preview layer mapping* and stored scenes referencing previews need updating.

*Technically, the manager can now figure out the preview outputs easily by looking (in the schema) at the keys under preview. Indeed, the correction_device_overview scene does exactly this to generate per-device preview scene links.

src/calng/CalibrationManager.py

@@ -26,7 +26,7 @@ from karabo.middlelayer import (
     UInt16, UInt32, Bool, Float, Double, String, VectorString, VectorHash,
     background, call, callNoWait, setNoWait, sleep, instantiate, slot, coslot,
     get_property, getTopology, getConfiguration, getConfigurationFromPast,
-    getConfigurationFromName)
+    getConfigurationFromName, getInstanceInfo)
 
 from . import scenes
 from ._version import version as deviceVersion