Shared memory for dssc return module data
The serialization engine used by joblibs backend 'loky' has an internal size limitation around 2.6GB. Once the return values of the binning routine exceed this limit the processing fails. In principle the meaning of the data reduction is to end up with reasonable file sizes, but nevertheless it makes sense to avoid this limitation.
The solution is to create a shared memory into which the individual processes can directly write the data. In that case there is no need to stream back the return values to the main process.
To be implemented in !87 (merged)
Edited by Rafael Gort