Skip to content

Support additional and maintain existing data compression

Philipp Schmidt requested to merge feat/compress-keys into master

While running in proc data, we noticed an increase in size after rewriting as the compressed image.gain and image.mask sets were no longer compressed. This MR adds the awareness of compressed datasets as well as introduces a new reduction operation compress-keys to apply additional compression.

For now chunking is somewhat restricted when used with compression to directly use the parallelized writing code from https://git.xfel.eu/calibration/pycalibration, but this can be fixed if necessary in the future.

Also renamed partial-copy to subslice-keys to be inline with other operation always denoting the target.

@kluyvert @turkot

Merge request reports