Skip to content

Add support to splice Influx-derived data into raw data

Philipp Schmidt requested to merge feat/splice-influx-data into master

Data imported from Influx via https://git.xfel.eu/amore/influx2h5 can be hard to integrate into existing raw data, as different files may not contain data for the same source and train, no matter whether they contain duplicate or distinct keys.

This MR adds a reduction plugin that can properly integrate such Influx-derived data, either by simply replacing the data for any key retrieved from Influx or removing any affected source entirely from the original files and moving them to their own aggregator.

Two new reduction instruction are used to implement this:

  • set-aggregator allows to override the aggregator a source is written to.
  • replace-keys allows to replace a key's data from the input collection with data from another location

Merge request reports