[REMI] Use DataFile API and support asymmetric pump-probe patterns
Description
This MR is a minor update to the REMI reconstruction notebook to use the DataFile API introduced with the new LPD notebook and add limited support of asymmetric pump-probe patterns.
Unlike the 2D X-ray detectors that don't actually have any notion of pulses but only frames/cells, the REMI notebook must figure out where and how many spectra corresponding to (FEL) pulses are on digitizer traces to group them together and reconstruct detector hits from them. Previously this only looked at FEL pulses based on the pulse pattern table (PPT). When using the pump-probe laser (PPL) in addition to the FEL, SQS recently began to offset these with respect to each other, so every train contains a couple of pumped pulses (FEL+PPL) as well as FEL-only and PPL-only pulses. The complication here is that due to technical reasons, the PPL always starts at 0 in the PPT no matter where the pulses actually are, thus making it impossible to correlate them just based on this data. Talking to LAS this may change in the future, but not anytime soon.
As a limited workaround for this, there is now a manual parameter ppl_offset than allows to specify an integer number of FEL pulses the PPL is offset to the FEL creating an asymmetric pattern. This is then tracked via additional fel and ppl columns in the triggers dataset in order to assign the pulses to each excitation. This method assumes that FEL and PPL use the same repetition rate, and that the first_pulse_offset is always set to the first pulse on the trace no matter the excitation method.
EDIT: As it became increasingly useful to already test the experiment with only PPL when FEL is not available, I decided to generalize the FEL/PPL support after all. In hindsight, my mistake was attempting to adapt the existing code rather than just write a new kernel to cleanly split FEL and PPL pulses. Now there may be any number of respective excitation pulses and they may be in any order - however the spectra (i.e. pulse lengths) are still assumed to be constant over a run, and thus the minimum is always used as distance. Strictly speaking this limitation may not be necessary, at most for the plotting code, but it's a sensible assumption for now. The other change is that the ppl_offset must now be expressed in terms of the PPT in general and not the FEL pulse distance in order to generalize it to any patterns.
Furthermore, I'm working on removing the need of this mysterious EXtra-remi package. All its content can be moved into a small cal_tools library with the exception of the Cython bindings for the proprietary resort library,( which can by itself be a very small package (I don't want to have to ship the binary shared library with pycalibration). I've also isolated which calibration constants would be needed to get rid of the extra YAML file this notebook needs, with all other parameters simply moving into the notebook. Both of these changes however are fairly large and will not be ready in time for next week.
How Has This Been Tested?
Running with
run = 409 # Run ID.
in_folder = '/gpfs/exfel/exp/SQS/202131/p900236/raw' # Partial input path appended with run ID.
out_folder = '/gpfs/exfel/exp/SQS/202131/p900236/scratch/cal_test' # Full path to output folder.
calib_config_path = '/gpfs/exfel/exp/SQS/202102/p002902/usr/remi_proc_config.yaml' # Path to correction and transform configuration
ppl_offset = -4 # In units of the FEL pulses.This run shows a clear separation of PPL-only pulses in the beginning, followed by 46 PPL+FEL pulses and finally 4 FEL-only pulses.
Relevant Documents (optional)
Example for an asymmetric FEL/PPL pattern:
Types of changes
- New feature (non-breaking change which adds functionality)