Writing corrected files is currently a bottleneck for AGIPD correction. With the parameters I'm testing with (mostly the defaults from the notebook), typical timings are something like 40 seconds reading raw data, 45 seconds processing (no common mode), 150 seconds writing. Watching it while writing, you mostly see 4 cores at 100% (writing 4 files) and the rest at almost 0.
HDF5 can't write in parallel without using MPI. However, I guessed that a large chunk of the 'writing' time was actually compressing the gain & mask data - they both compress very well, but DEFLATE is slow. However, we can compress the data ourselves and use a lower-level API to write it to HDF5, allowing us to parallelise the compression.
It seems to work - in my tests, writing each batch of files went from about 150 seconds to about 60, reducing the load-correct-write time by about 40% (or ~30% of the total time for the Slurm jobs).
However, I have also disabled the fletcher32 checksum algorithm for the compressed (gain & mask) data. If this is important, we would have to find or implement a reasonably fast version of this which could be called from Python.
