fix/AGIPD_CM_processing_issue_redmine_#75266

SUMMARY:

This MR is related to a det-support Redmine ticket https://in.xfel.eu/redmine/issues/75266?jump=issues. There were some reporter errors in the Data Processing Sections in the report for sequences 13, 20, and 52.

I understood that these errors occurred for images with fewer n_imgs than its preceding processed core (while applying cm_correction()) and the same value of trainIds was stored at the end of the shared_mem array.

Error's Screenshot:

Reviewers

@danilevc @kamile @kluyvert

added 9 commits

• 4c77457a...b6496248 - 8 commits from branch master
• 1a6e3057 - resolve conflicts fix/det_ticket_75266_CM_bug

Compare with previous version

Here's why it LGTMs:

Behind the scenes, sharedmem makes use of a binary file to store the data, based of numpy.memmap

Each process in our pool accessing the data does so by seeking to a specific position in the file, and then manipulating the file within its range.
However, there's little protection for accessing data beyond the allocated space (due to it being, well, a file).

As such, it's up to us to make sure we don't read too far. That's why there's a bunch of self.sharedmem['whatever'][:n_img]: ensuring we stay within the allocated n_img boundaries.

The fix LGTM too, but I don't think memmap is actually relevant to understanding it. The issue is just that we're reusing an array which is bigger than the data we're interested in.

If we actually read beyond the end of the allocated space, I believe we'd see segfaults rather than a Numpy exception. NumPy should do the necessary bounds checks to avoid that.

Thank you both for the review.

mentioned in commit 9244777c

merged