Skip to content

Initial performance optimizations for frame selection pilots

Philipp Schmidt requested to merge feat/speedups into master

First attempts at improving the abysmal performance observed while applying frame selection at the proposal stage.

Profiling based on py-spy and !7 (merged) pointed to two major culprits:

  • Reading the full list of keys of AGIPD sources

    • Fixed in 0509da5e with an optimized version of SourceData._get_index_group_sample (@kluyvert I will upstream a general verison of this in EXtra-data later.)
    • Fixed in 6d0fd2bd by skipping an over-eager check for compliance
  • Reading very large datasets (e.g. image.data)

    • Fixed in 20e6d23b through parallel reading of large datasets (> 1 GiB for now) with pasha

Both measures sped up processing of large runs by ~3x.

Merge request reports