- Open the device scene for individual correction devices and look at the device status log.
If the DAQ is monitoring, but without detector data coming in, log should show warnings about this.
2. Is pipeline down or misconfigured?
- Are the preview assemblers receiving data?
- Is the geometry device up?
3. Is preview matching breaking down?
- Note: for fast detectors, data transfer from DAQ to correction device can hit network limits.
This can cause **matching** on preview assemblers to break down.
If rates on correction devices fluctuate or are lower than expected, try increasing the **DAQ train stride** to slow down incoming data.
## Analysis software not receiving data
Assumption: online analysis connected to the [Karabo Bridge](https://rtd.xfel.eu/docs/data-analysis-user-documentation/en/latest/karabo_bridge/protocol.html) output of a **group matcher** or a **full matcher**.
These matchers (along with the preview assemblers) are built around [TrainMatchers](https://git.xfel.eu/karaboDevices/TrainMatcher), so familiarity with TrainMatchers may help in troubleshooting common issues.
1. Is the matcher not receiving data?
- Check that detector is sending data and calibration pipeline generally works.
- Check that the sources are configured correctly, that the matcher is started, and that it has established connections with the source channels.
2. Is **matching ratio** very low?
- A group or full matcher can easily run into network limits for big detectors.
This can cause individual trains from individual modules to randomly not reach the matcher, leading to very low **matching ratio**.
If received rates for individual sources fluctuate or are lower than expected, try increasing the **DAQ train stride** or lowering the number of frames per train (optionally via **frame filter** on correction devices) to slow down incoming data.
- In case the matcher itself has a hard time keeping up (e.g. stacking full matcher for big detector), check performance tweaks like "use thread pool" and disabling Karabo channel outputs.
3. Is a missing module breaking matching?
- If a module is known to be missing, consider deselecting it in the sources configuration.
- To gracefully handle situations where individual modules disappear / stop working for a while, consider using the **Max Idle** TrainMatcher feature on the matcher.
In case of a full detector matcher, do check what this is set to for the individual group matchers, too.
## Network limits and data rates
calng correction devices can typically process data faster than we can reliably send it over the network.
It is not immediately obvious when a fast data stream runs into network limits.
Typically, some trains seem to silently go missing.
For different output needs, different throttling options are available:
1. From detector module to DAQ (outside scope of calng)
- Number of frames per train is important, keep in mind for later steps
2. From DAQ to correction device
- Can tweak **DAQ train stride** via calibration manager: positive integer `n` where DAQ only sends every `n`th train on monitoring output (higher value means slower data)
3. From correction device to group matcher
- Can tweak number of frames of frames processed and forwarded using a **frame filter**
For full detector matching in particular, it is important to choose DAQ train stride appropriately given the desired number of frames per train (either from detector or via frame filter on correction side).
A comprehensive set of recommended values is not yet ready; please do make note of stable configurations.
- LPD1M at FXE (full stacking matcher tests in May 2022)
