Initial performance optimizations for frame selection pilots
First attempts at improving the abysmal performance observed while applying frame selection at the proposal stage.
Profiling based on py-spy
and !7 (merged) pointed to two major culprits:
-
Reading the full list of keys of AGIPD sources
-
Reading very large datasets (e.g.
image.data
)- Fixed in 20e6d23b through parallel reading of large datasets (> 1 GiB for now) with
pasha
- Fixed in 20e6d23b through parallel reading of large datasets (> 1 GiB for now) with
Both measures sped up processing of large runs by ~3x.