Revised CalCat API

I'm going to leave this as a draft, but it's slowly taking shape. In a separate branch, I've adapted the LPD notebook to use this API. I've checked that it runs, but not what the results look like.

https://git.xfel.eu/calibration/pycalibration/-/blob/feat/calcat-api-2/lpd-use/notebooks/LPD/LPD_Correct_Fast.ipynb

mentioned in merge request !899 (merged)

added 1 commit

a5299e1d - Methods to get multi-module constants as Numpy or Xarray arrays

Compare with previous version

added 1 commit

5b6ce2d4 - Add DSSCConditions to new CalCat API

Compare with previous version

added 1 commit

cae322cb - Useful repr for CalibrationData

Compare with previous version

Where would the .from_data(run: DataCollection) method fit in here?

Giving it a bit of thought we can actually put it into CalibrationData and gain even more automation than the first draft had (which required you to pick the detector type). Based on the detector name, one can obtain the type via PDU and from there take it to detector-specific implementations. The signature would then simply be:

CalibrationData.from_data(run: DataCollection, detector_name: str)

Strictly speaking the detector name could itself be optional, but frankly the benefit seems small at comparably large cost (encoding name patterns).

Adding some more thoughts here, and experiences from users working with it. Naturally these things can follow in future PRs:

Going from aggregator to detector source in files is not obvious, the aggregator concept continues to be quite cryptic and unintuitive. In https://git.xfel.eu/calibration/planning/-/issues/112 we looked at the columns in CalCat's detector table. Some appear to be unused, so we could start using them now to encode the number-from-aggregator-to-source-pattern and present users with proper source names as mapping.
EXtra-data presents data with different axes than constants are in, the API should abstract this. We currently encode this detail in notebooks/utility modules.
Not sure about this one, but missing modules could be (optionally) filled up by neutral-element constants.
CalibrationData.from_data() can call into xxxCondition.from_data() methods to fill in the condition part while contributing itself the creation time

I'm trying to get a handle on the module naming thing - I've started a thread in Zulip.

Axis order: I wonder if we could have an API when you request the constant like .ndarray(dims=['module', 'slow_scan', 'fast_scan', 'gain']) to specify the order you want, with sensible defaults. Then if we change how we're storing the constants, so long as we know the axis order in the file, we can transparently rearrange it to match.

Putting 'gaps' for missing modules implies that there's a matching gap in the data, so you're wasting time and energy processing non-data. I know DSSC currently has 12 of 16 modules. I think we should try to encourage people to handle the data without the missing modules. But I accept that may not always be straightforward.