Fix stacking buffer shape issue

This add stacking preparation which checks data across all stacking sources and prepares accordingly stacking buffers.

This fixes #31 (closed)

The original stacking code tries to assign data to the stacking buffer. If this assignment raises the IndexError then the code reallocates buffer with proper shape. The issue related to the numpy array broadcasting. If the current train has only one frame due to the filtering, then all arrays in image.* broadcast to buffer from previous train with any shape. If the previous train has zero frames than data just throw away.

@hammerd

changed title from Add stacking preparation which checks data across all stacking sources and... to Fix stacking buffer shape issue

changed the description

We could actually use this to keep track parts of the resulting stacked array not getting filled (there was a TODO about zeroing unfilled entries).

Indeed, I forgot that we have such an option. So we need instead to record the missed indices and fill buffer with zeros later.

Zeros or something configurable (could even add a column to the stacking configuration table for what missing entries should be in each stacking group). In my own test environment, I've at some point hotpatched to zero-initialize for testing LPD + CrystFEL because geometry expects shape for 16 modules worth of data - with only 15 modules, there would be a lot of arbitrarily initialized data coming through. But generally, yes, we should handle missing data per train.

I've added filling with zeros

Can avoid checking the shape variable for exiting;

if key in data_hash:
    merge_data = data_hash[key]
    merge_data_shape = merge_data.shape
else:
    ignore_stacking[(new_source, key)] = "Some data is missed"
    continue

Added this, thanks

Nice catch that implicit broadcasting breaks the try/except solution! The stacking implementation is generally a bit verbose and not very self-contained (have been meaning to refactor), but checking up front and then stacking is a reasonable approach to fix the current issue. Have added some comments.

mentioned in commit 0ef3efad

added 3 commits

• af4f1f23 - Add filling of places for missed sources in stacked data with zeros
• 0ef3efad - Avoid shape check as David suggested in !72 (merged)
• 828bbce4 - Add logging of the stacking failures

Compare with previous version

Currently, orig_size is not used (I see commented out check - should it be assertion?)

Ahhh.. And I think, why did I comment on this line: https://git.xfel.eu/calibration/calng/-/blob/fix/stacking-buffer-shape/src/calng/ShmemTrainMatcher.py#L364

I don't know what to do, because it depends on the filtering part. And now there is no check, and filtering should raise exception if the selection mask doesn't match the train size.

I've change filtering to ignore datasets if they don't match selection mask. Now I use origin_size to deduce the stacking buffer size. Let's try without exceptions

The fix LGTM.

added 1 commit

• cb7013db - Ignore filtering if mask doesn't match data size

Compare with previous version

added 1 commit

• 65f724b0 - Ignore filtering if mask doesn't match data size

Compare with previous version

merged

mentioned in commit c3a136fc