Skip to content

Base correction device, CalCat interaction, DSSC and AGIPD devices

David Hammer requested to merge add-agipd-device into master

This MR will add the basic components for correction devices, including communication with CalCat, execution of GPU kernels and more. The branch for this MR already includes the CalibrationManager - if MR for manager is merged first, I will rebase this MR (only small tweaks to manager introduced here).

Main class structure:

  • BaseCorrection is base class of correction devices (Karabo bound devices), sets up common schema elements and does most shared Karabo stuff.
    • Subclasses: AgipdCorrection and DsscCorrection
    • There are a some class attributes the subclasses must set.
    • There are some things (like schema extensions) the subclasses have to do on their own.
  • BaseGpuRunner is base class of the objects which handle all GPU interaction.
    • Subclasses: AgipdGpuRunner and DsscGpuRunner
    • A {}GpuRunner lives inside a {}Correction, but does not know about this (not tightly integrated into Karabo device).
      • It holds all its own buffers, can cleanly be deleted and reinstantiated.
      • It can be tested independently.
    • In agipd_gpu.py (home to AgipdGpuRunner), there are also some enums. Of particular importance is CorrectionFlags which is used both to give bit mask to kernel for enabling / disabling correction steps, but also to generate part of the device schema.
    • Each GPU runner class has a .cpp file with a correction kernel.
  • BaseCalcatFriend (in need of more serious name) handles interaction with CalCat.
    • Subclasses: AgipdCalcatFriend and DsscCalcatFriend (all living in calcat_utils.py for now).
    • Similar to Philipp's prototype CalCat API wrapper, but these wrap themselves around the device schema for constant parameters (tight integration into Karabo device).
    • Specifically, a {}CalcatFriend manages the nodes constantParameters and foundConstants on its correction device (both to get parameters for querying and to set IDs and statuses when it's working).
    • Subclasses mostly serve to set up condition and constant mapping for a given detector.
    • In calcat_utils.py, there are also enums like AgipdConstants which list all the constants to fetch - also relevant for generation of device schema (see ex. _add_status_schema_from_enum).
    • They expect a secrets file for CalCat. By default, will look in ${KARABO}/var/data/calibration-client-secrets.json.

I have started adding some rudimentary tests. So far, tests for GPU runners compare the outputs of the GPU runners against naive NumPy implementations and there's a test of the CalCat code to check that something can be found.

I have also started adding some scenes. Using the extensive device schemas, I intend for these to be generated programmatically (e.g. iterating over constantParameters node to generate config table or over foundConstants to generate overview of what's loaded). This is coming together nicely, but not important for reviewing at this moment.

Edited by David Hammer

Merge request reports