"There is also an unreleased function from `extra_data`, called `plot_missing_trains()`, to be released in version 1.13.0 (see [here](https://extra-data.readthedocs.io/en/latest/reading_files.html#extra_data.DataCollection.plot_missing_data)) that should shows similar quantities."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
...
...
%% Cell type:markdown id: tags:
# Loading data in memory with the SCS ToolBox
%% Cell type:markdown id: tags:
## ToolBox mnemonics
%% Cell type:markdown id: tags:
Within the framework of the [extra_data](https://extra-data.readthedocs.io/en/latest/) package, which the SCS ToolBox is built upon, the European XFEL data is organized in a hierachical structure, in which a *source* (for instance, a motor, or the output of a digitizer) contains a few datasets, accessed with a *key* (the actual position of the motor, the various channels of the digitizer). The ToolBox *mnemonics* are simple words that represent frequently used variables at the SCS instrument. Each menmonic is associated with a dictionnary containing the source, the key and the dimension names of the variable.
The mnemonics are stored in a dictionnary, accessible as `toolbox_scs.mnemonics`. Let us read the content of the mnemonic `SCS_SA3`, which corresponds to the pulse energy of the SASE 3 pulses measured by the XGM in the SCS experiment hutch:
%% Cell type:code id: tags:
``` python
importtoolbox_scsastb
tb.mnemonics['SCS_XGM']
```
%% Output
Cupy is not installed in this environment, no access to the GPU
({'source': 'SCS_BLU_XGM/XGM/DOOCS:output',
'key': 'data.intensityTD',
'dim': ['XGMbunchId']},)
%% Cell type:markdown id: tags:
The list of available mnemonics can vary from run to run, depending on which sources were recorded. The function `mnemonics_for_run` returns the mnemonics that correspond to actual data sources in a run. The input parameters can be the proposal and run numbers of the run or the run itself (`extra_data``DataCollection`):
%% Cell type:code id: tags:
``` python
# providing the proposal and run numbers
run_mnemonics=tb.mnemonics_for_run(3485,52)
# alternative, providing the DataCollection as input argument
The mnemonics are by no means an exhaustive list of the contents of a run, but rather convenience shortcuts to the mostly used data sources at SCS. Please refer to the [extra_data](https://extra-data.readthedocs.io/en/latest/) package to access the full list of data sources present in a run.
</div>
%% Cell type:markdown id: tags:
It is possible to extract the "run value" (see EXtra-Data [get_run_value()](https://extra-data.readthedocs.io/en/latest/reading_files.html#extra_data.DataCollection.get_run_value) for details) of a source/key combination by using the function `load_run_values()`.
This is a convenient way of quickly checking the values of the most relevant parameters of a run, like the opening of the exit slit of the monochromator ('ESLIT' im mm) or the transmission of the gas attenuator ('transmission' in %) without loading the full data, which would take much more time and require large memory.
The run value is indeed only one value. This means that the variables that have more than one values like digitizer or 2D detectors do not have a run value. The corresponding mnemonics get a run value of `None`, as in the following example:
The run value of a source/key combination is stored at the beginning of the run. **The run value DOES NOT show nor it checks the variations of a variable in a run** and can only be representative if the value has not changed. The full check can be done with EXtra-Data [as_single_value()](https://extra-data.readthedocs.io/en/latest/reading_files.html#extra_data.KeyData.as_single_value) function or using the `load` function described below.
</div>
%% Cell type:markdown id: tags:
## The `load` function
%% Cell type:markdown id: tags:
The `load` function of the ToolBox loads the variables recorded in a run into memory. Given a proposal number and a run number, the function in its simplest form takes a list of mnemonics as the `fields` argument. The data associated to the mnemonics is loaded and all variables are aligned by train Id and pulse Id.
The function returns an `extra_data``DataCollection` (run) and an `xarray``Dataset` (data, which is displayed here in a summarized form). The DataCollection is the key element of the `extra_data` package and it is used in many functions of the ToolBox. It contains information on the run and enables data handling and loading (see the `extra_data`[documentation](https://extra-data.readthedocs.io/en/latest/) for details). The Dataset data is the main result of our loading operation. In it, we can find:
* Dimensions `pulse_slot`, `trainId`, `sa3_pId`
* Coordinates: `trainId` and `sa3_pId`: the train Id values and the SASE 3 pulse Id values.
* Data variables: The loaded data arrays. In this example, nrj is the monochromator energy, in eV, for each train. MCP3peaks is one of the MCPs of the TIM detector, SCS_SA3 is the pulse energy of the SASE 3 pulses measured by the XGM in the SCS hutch. The bunchPatternTable is loaded if the number of pulses has changed during the run. It is an array of 2700 values per train (the maximum number of pulses at 4.5 MHz provided by the machine) and contains information on how the pulses are distributed among SASE 1, 2, 3, and the various lasers at European XFEL. The `sa3_pId` coordinates are extracted from this table.
* Attribute `runFolder`, the name of the folder that contains the raw files of the run. It can be accessed via: `data.attrs['runFolder']`.
%% Cell type:markdown id: tags:
The (maximum) number of pulses per train is given by `data.sa3_pId.size`
%% Cell type:markdown id: tags:
## Accessing the raw arrays
%% Cell type:markdown id: tags:
The function `load`, by default, loads the raw arrays using the `get_array` function of `extra_data`, and extracts only the relevant data from them, according to the bunch pattern table. It may be required, in some cases, to access the raw array of a specific mnemonic. For this, we can use the `DataCollection` returned earlier by the call to `load`:
The `raw_traces``DataArray` contains the digitizer raw traces generated by the MCP 2 of the TIM detector. The array has dimensions `trainId` and `samplesId` (the latter given by `tb.mnemonics['MCP2raw']['dim']`). Quick visual inspection of the trace of the first train can be performed using the built-in plotting function of `xarray`:
%% Cell type:code id: tags:
``` python
raw_traces.isel(trainId=0).plot()
```
%% Output
[<matplotlib.lines.Line2D at 0x2b2ef42ca320>]
%% Cell type:markdown id: tags:
## Missing trains
%% Cell type:markdown id: tags:
The data rate, or percentage of trains containing data, is checked in the `load` function, and a warning is displayed if less than 95% of data is present. This can be useful to identify DAQ problems during a beamtime.
%% Cell type:code id: tags:
``` python
fields=['SCS_HAMP_HV','SCS_SA3']
run,ds=tb.load(5836,162,fields)
```
%% Output
SCS_SA3: only 85.6% of trains (2122 out of 2479) contain data.
%% Cell type:markdown id: tags:
A function `check_data_rate` allows to extract the fraction of trains containing data for given mnemonics:
It will return the data rate for all mnemonics in the run if `fields` is omitted
%% Cell type:code id: tags:
``` python
tb.check_data_rate(run)
```
%% Output
{'sase3': 1.0,
'sase2': 1.0,
'sase1': 1.0,
'laser': 1.0,
'maindump': 1.0,
'bunchpattern': 1.0,
'bunchPatternTable': 1.0,
'npulses_sase3': 1.0,
'npulses_sase1': 1.0,
'npulses_laser': 1.0,
'BAM414': 0.9693424768051634,
'BAM1932M': 0.982654296087132,
'BAM1932S': 0.9669221460266236,
'DPS2CAM2': 0.0,
'XTD10_photonFlux': 1.0,
'XTD10_photonFlux_sigma': 1.0,
'XTD10_XGM': 0.9983864461476402,
'XTD10_XGM_sigma': 0.9983864461476402,
'XTD10_SA3': 0.9983864461476402,
'XTD10_SA3_sigma': 0.9983864461476402,
'XTD10_SA1': 0.9983864461476402,
'XTD10_SA1_sigma': 0.9983864461476402,
'XTD10_slowTrain': 1.0,
'XTD10_slowTrain_SA1': 1.0,
'XTD10_slowTrain_SA3': 1.0,
'SCS_photonFlux': 1.0,
'SCS_photonFlux_sigma': 1.0,
'SCS_HAMP_HV': 1.0,
'SCS_XGM': 0.8559903186768858,
'SCS_XGM_sigma': 0.8559903186768858,
'SCS_SA1': 0.8559903186768858,
'SCS_SA1_sigma': 0.8559903186768858,
'SCS_SA3': 0.8559903186768858,
'SCS_SA3_sigma': 0.8559903186768858,
'SCS_slowTrain': 1.0,
'SCS_slowTrain_SA1': 1.0,
'SCS_slowTrain_SA3': 1.0,
'AFS_DelayLine': 1.0,
'AFS_FocusLens': 1.0,
'PP800_PhaseShifter': 1.0,
'PP800_SynchDelayLine': 1.0,
'PP800_DelayLine': 1.0,
'PP800_HalfWP': 1.0,
'PP800_FocusLens': 1.0,
'FFT_FocusLens': 1.0,
'ZABER110_ODL': 1.0,
'FastADC0peaks': 0.0,
'FastADC0raw': 0.0,
'FastADC1peaks': 0.0,
'FastADC1raw': 0.0,
'FastADC2peaks': 0.0,
'FastADC2raw': 0.0,
'FastADC3peaks': 1.0,
'FastADC3raw': 1.0,
'FastADC4peaks': 0.0,
'FastADC4raw': 0.0,
'FastADC5peaks': 1.0,
'FastADC5raw': 1.0,
'FastADC6peaks': 0.0,
'FastADC6raw': 0.0,
'FastADC7peaks': 0.0,
'FastADC7raw': 0.0,
'FastADC8peaks': 0.0,
'FastADC8raw': 0.0,
'FastADC9peaks': 1.0,
'FastADC9raw': 1.0,
'FastADC2_0peaks': 0.0,
'FastADC2_0raw': 0.0,
'FastADC2_1peaks': 0.0,
'FastADC2_1raw': 0.0,
'FastADC2_2peaks': 0.0,
'FastADC2_2raw': 0.0,
'FastADC2_3peaks': 0.0,
'FastADC2_3raw': 0.0,
'FastADC2_4peaks': 0.0,
'FastADC2_4raw': 0.0,
'FastADC2_5peaks': 0.0,
'FastADC2_5raw': 0.0,
'FastADC2_6peaks': 1.0,
'FastADC2_6raw': 1.0,
'FastADC2_7peaks': 1.0,
'FastADC2_7raw': 1.0,
'FastADC2_8peaks': 0.0,
'FastADC2_8raw': 0.0,
'FastADC2_9peaks': 0.0,
'FastADC2_9raw': 0.0,
'Gotthard1': 1.0,
'Gotthard2': 0.0}
%% Cell type:markdown id: tags:
There is also an unreleased function from `extra_data`, called `plot_missing_trains()`, to be released in version 1.13.0 (see [here](https://extra-data.readthedocs.io/en/latest/reading_files.html#extra_data.DataCollection.plot_missing_data)) that should shows similar quantities.
