Skip to content

Shared memory for dssc return module data

The serialization engine used by joblibs backend 'loky' has an internal size limitation around 2.6GB. Once the return values of the binning routine exceed this limit the processing fails. In principle the meaning of the data reduction is to end up with reasonable file sizes, but nevertheless it makes sense to avoid this limitation.

The solution is to create a shared memory into which the individual processes can directly write the data. In that case there is no need to stream back the return values to the main process.

Example: https://joblib.readthedocs.io/en/latest/auto_examples/parallel_memmap.html#writable-memmap-for-shared-memory-joblib-parallel

To be implemented in !87 (merged)

Edited by Rafael Gort
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information