doc/DSSC.rst

+DSSC
+----
+
+DSSC data binning
+#################
+
+In scattering experiment one typically wants to bin DSSC image data versus
+time delay between pump and probe or versus photon energy. After this first
+data reduction steps, one can do azimuthal integration on a much smaller
+amount of data.
+
+The DSSC data binning procedure is based on the notebook
+:doc:`Dask DSSC module binning <Dask DSSC module binning>`. It performs
+DSSC data binning against a coordinate specified by *xaxis* which can
+be *nrj* for the photon energy, *delay* in which case the delay stage
+position will be converted in picoseconds and corrected but the BAM, or
+another slow data channel. Specific pulse pattern can be defined, such as:
+
+.. code:: python
+ 
+    ['pumped', 'unpumped']
+
+which will be repeated. XGM data will also be binned similarly to the DSSC
+data.
+
+Since this data reduction step can be quite time consuming for large datasets,
+it is recommended to launch the notebook via a SLURM script. The script can be
+downloaded from :download:`scripts/bin_dssc_module_job.sh` and reads as:
+
+.. literalinclude:: scripts/bin_dssc_module_job.sh
+   :language: bash
+   :linenos:
+
+It is launched with the following:
+
+.. code:: bash
+    
+    sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 0 -x delay -b 0.1
+    sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 1 -x delay -b 0.1
+    sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 2 -x delay -b 0.1
+    sbatch ./bin_dssc_module_job.sh -p 2719 -d 180 -r 179 -m 3 -x delay -b 0.1
+
+where 2719 is the proposal number, 180 is the dark run number, 179 is the run
+nummber and 0, 1, 2 and 3 are the 4 module group, each job processing a set of
+4 DSSC module, delay is the bin axis and 0.1 is the bin width.
+
+The result will be 16 \*.h5 files, one per module, saved in the folder specified
+in the script, a copy of which can be found in the *scripts* folder in the
+toolbox source. This files can then be loaded and combined with:
+
+.. code:: python
+   
+    import xarray as xr
+    data = xr.open_mfdataset(path + '/*.h5', parallel=True, join='inner')
+
+
+DSSC azimuthal integration
+##########################
+
+Azimuthal integration can be performed with pyFAI_ which can utilize the
+hexagonal pixel shape information from the DSSC geometry to split
+the intensity in a pixel in the bins covered by it. An example notebook
+:doc:`Azimuthal integration of DSSC with pyFAI.ipynb <Azimuthal integration of DSSC with pyFAI>` is available.
+
+A second example notebook
+:doc:`DSSC scattering time-delay.ipynb <DSSC scattering time-delay>`
+demonstrates how to:
+
+- refine the geometry such that the scattering pattern is centered before
+  azimuthal integration
+
+- perform azimuthal integration on a time delay
+  dataset with ``xr.apply_ufunc`` for multiprocessing.
+
+- plot a two-dimensional map of the scattering change as function of
+  scattering vector and time delay
+
+- integrate certain scattering vector range and plot a time trace
+
+DSSC fine timing
+################
+
+When DSSC is reused after a period of inactivity or when the DSSC gain setting
+use a different operation frequency the DSSC fine trigger delay needs to be
+checked. To analysis runs recorded with different fine delay, one can use
+the notebook :doc:`DSSC fine delay with SCS toolbox.ipynb <DSSC fine delay with SCS toolbox>`.
+
+DSSC quadrant geometry
+######################
+
+To check or refined the DSSC geometry or quadrants position, the following
+notebook can be used :doc:`DSSC create geometry.ipynb <DSSC create geometry>`.
+
+Legacy DSSC binning procedure
+#############################
+
+Most of the functions within toolbox_scs.detectors can be accessed directly. This is useful during development, or when working in a non-standardized way, which is often neccessary during data evaluation. For frequent routines there is the possibility to use dssc objects that guarantee consistent data structure, and reduce the amount of recurring code within the notebook.
+
+* bin data using toolbox_scs.tbdet -> *to be documented*.
+* :doc:`bin data using the DSSCBinner <dssc/DSSCBinner>`.
+* post processing, data analysis -> *to be documented*
+
+.. _pyFAI: https://pyfai.readthedocs.io

doc/Dask DSSC module binning.ipynb

+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+%matplotlib notebook
+import matplotlib.pyplot as plt
+plt.rcParams['figure.constrained_layout.use'] = True
+
+import dask
+print(f'dask: {dask.__version__}')
+import dask.array as da
+dask.config.set({'array.chunk-size': '512MiB'})
+
+import xarray as xr
+```
+
+%% Output
+
+    dask: 2.11.0
+
+%% Cell type:code id: tags:
+
+``` python
+import sys
+print(sys.executable)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+from psutil import virtual_memory
+import gc
+# gc.collect() # run garbage collection to free possible memory
+
+mem = virtual_memory()
+print(f'Physical memory: {mem.total/1024/1024/1024:.0f} Gb')  # total physical memory available
+```
+
+%% Output
+
+    Physical memory: 504 Gb
+
+%% Cell type:code id: tags:
+
+``` python
+import logging
+logging.basicConfig(filename='example.log', level=logging.DEBUG)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+%load_ext autoreload
+
+%autoreload 2
+
+import toolbox_scs as tb
+print(tb.__file__)
+from toolbox_scs.routines.boz import load_dssc_module
+
+from extra_data import open_run
+```
+
+%% Output
+
+    /home/lleguy/notebooks/ToolBox/src/toolbox_scs/__init__.py
+
+%% Cell type:markdown id: tags:
+
+# Parameters
+
+%% Cell type:code id: tags:parameters
+
+``` python
+proposalNB = 2719
+dark_runNB = 180
+runNB = 179
+module_group = 0
+pulse_pattern = ['pumped', 'unpumped']
+xaxis = 'delay' # 'nrj'
+bin_width = 0.1 # [ps]
+path = f'/gpfs/exfel/exp/SCS/202002/p002719/scratch/tests/r{runNB}/'
+```
+
+%% Cell type:code id: tags:
+
+``` python
+moduleNB = list(range(module_group*4, (module_group+1)*4))
+```
+
+%% Cell type:markdown id: tags:
+
+# Processing function
+
+%% Cell type:code id: tags:
+
+``` python
+def process(module):
+    # Load dark
+    arr_dark, tid_dark = load_dssc_module(proposalNB, dark_runNB, module, drop_intra_darks=False)
+    arr_dark = arr_dark.rechunk(('auto', -1, -1, -1))
+    dark_img = arr_dark.mean(axis=0).compute()
+
+    # Load module data
+    arr, tid = load_dssc_module(proposalNB, runNB, module, drop_intra_darks=False)
+    arr = arr.rechunk(('auto', -1, -1, -1))
+
+    # dark and intra dark correction
+    arr = arr - dark_img
+    arr = arr[:, ::2, :, :] - arr[:, 1::2, :, :]
+
+    # Load slow data against which to bin
+    if xaxis == 'delay':
+        run, v = tb.load(proposalNB, runNB, ['PP800_DelayLine', 'BAM1932M', 'SCS_XGM'])
+    else:
+        run, v = tb.load(proposalNB, runNB, [xaxis, 'SCS_XGM'])
+
+    # select part of the run
+    # v = v.isel({'trainId':slice(0,3000)})
+
+    # combine slow and DSSC module data
+    xr_data = xr.DataArray(arr,
+                       coords={'trainId': tid,
+                               'sa3_pId': v['sa3_pId'].values},
+                       dims = ['trainId', 'sa3_pId', 'y', 'x'])
+    xr_data = xr_data.expand_dims(module=[module], axis=2)
+    r = xr.merge([xr_data.to_dataset(name='DSSC'), v], join='inner')
+
+    # calculate bins
+    if xaxis == 'delay':
+        r['delay'] = tb.misc.positionToDelay(r['PP800_DelayLine'])
+        bam = r['BAM1932M'] - r['BAM1932M'].mean()
+        r['bin_delay'] = ((r['delay'] - bam)/bin_width).round()*bin_width
+    else:
+        r['bin_' + xaxis] = (r[xaxis]/bin_width).round()*bin_width
+
+    # add the pulse pattern coordinates
+    Nrepeats = int(len(v['sa3_pId'].values)/len(pulse_pattern))
+    pp = pulse_pattern*Nrepeats
+    pp = np.array(pp)
+    r = r.assign_coords(pp=("sa3_pId", pp))
+
+    # select pattern and bin data
+    bin_data = None
+    for p in np.unique(pp):
+        # slice using non-index coordinates
+        # https://github.com/pydata/xarray/issues/2028
+        sub_r = r.sel(sa3_pId=(r.pp == p))
+
+        # calculate mean on bin, then mean to remove the dimension
+        res = sub_r.groupby('bin_'+xaxis).mean().mean(['sa3_pId'])
+
+        if bin_data is None:
+            bin_data = res
+            bin_data['DSSC'] = res['DSSC'].expand_dims(pp=[p])
+            bin_data['SCS_SA3'] = res['SCS_SA3'].expand_dims(pp=[p])
+        else:
+            bin_data = xr.merge([bin_data,
+                                 res['DSSC'].expand_dims(pp=[p]),
+                                 res['SCS_SA3'].expand_dims(pp=[p])])
+
+    # save the result
+    fname = path + f'run{runNB}-darkrun{dark_runNB}-module{module}.h5'
+    print(fname)
+    bin_data.to_netcdf(fname, format='NETCDF4', engine='h5netcdf')
+```
+
+%% Cell type:markdown id: tags:
+
+# Processing
+
+%% Cell type:code id: tags:
+
+``` python
+for m in moduleNB:
+    process(m)
+```

doc/SLURM.rst

+SLURM, sbatch, partition, reservation
+-------------------------------------
+
+Scripts launched by ``sbatch`` command can employ magic cookie with
+``#SBATCH`` to pass options SLURM, such as which partition to run on.
+To work, the magic cookie has to be at the beginning of the line.
+This means that:
+
+* to comment out a magic cookie, adding another "#" before it is sufficient
+* to comment a line to detail what the option does, it is best practice
+  to put the comment on the line before
+
+Reserved partition are of the form "upex_003333" where 3333 is the proposal
+number. To check what reserved partition are existing, their start and end
+date, one can ``ssh`` to ``max-display`` and use the command ``sview``.
+
+.. image:: sview.png
+
+To use a reserved partition with ``sbatch``, one can use the magic cookie
+
+``#SBATCH --reservation=upex_003333``
+
+instead of the usual
+
+``#SBATCH --partition=upex``

doc/bunch_pattern_decoding.rst

+Reading the bunch pattern
+=========================
+
+The bunch pattern table is an array of 2700 values per train (the maximum number of pulses at 4.5 MHz provided by the machine) and contains information on how the pulses are distributed among SASE 1, 2, 3, and the various lasers at European XFEL.
+The data stored in the bunch pattern table (mnemonic *bunchPatternTable*) can be extracted using the wrappers to the `euxfel_bunch_pattern <https://pypi.org/project/euxfel-bunch-pattern/>`_ package as follows:
+
+.. code:: ipython3
+
+    import toolbox_scs as tb
+
+    proposalNB = 2953
+    runNB = 507
+
+    run, ds = tb.load(proposalNB, runNB, "bunchPatternTable")
+    ppl_mask = tb.is_ppl(ds["bunchPatternTable"])
+
+ppl_mask is a boolean DataArray of dimensions trainId x 2700, where True values indicate where a laser pulse from the PP laser was triggered.
+
+.. note::
+   The position of the PP laser pulses with respect to that of the SASE 3 pulses is arbitrary. The PP laser pattern always starts at pulse Id 0, while that of SASE 3 can vary, depending on the machine parameters. One can use the ToolBox function `align_ol_to_fel_pId()` to shift the PP laser pulse Ids to match those of SASE 3.
+
+From this mask, one can obtain the number of pulses per train by summing along the 'pulse_slot' dimension:
+
+.. code:: ipython3
+
+   ppl_npulses = ppl_mask.sum(dim='pulse_slot')
+
+There is also the function `get_sase_pId()` that can be used to directly extract the pulse Ids of a particular location:
+
+.. code:: ipython3
+
+   ppl_pId = tb.get_sase_pId(run, loc='scs_ppl')
+
+This provides a list of the pulse Ids used during a run but does not track changes of number of pulses. For this, the mnemonic `npulses_laser` can be loaded to get the number of pulses in each trainId and the mnemonic `laser` can be loaded to get the pulse Ids of the scs_ppl in each trainId:
+
+.. code:: ipython3
+
+   run, ds = tb.load(proposalNB, runNB, ['laser', 'npulses_laser'])

doc/changelog.rst

+Release Notes
+=============
+
+unreleased
+----------
+
+- **Bug fixes**
+
+    - fix :issue:`75` regarding pulse Id assignment when reading BAM data :mr:`272`
+    - fix :issue:`80` regarding new bunch pattern table size when reading BAM data :mr:`279`
+    - fix :issue:`84` regarding loading only a given amount of BOZ DSSC data in memory :mr:`288`
+    - fix :issue:`86` regarding loading XGM :mr:`291`
+    - fix :issue:`88` regarding loading data from multiple folders :mr:`303`
+
+- **Improvements**
+
+    - update documentation on knife-edge scan and fluence calcuation :mr:`276`, :mr:`278`
+    - update scannerY mnemonics :mr:`281`
+    - move loading logic to mnemonics :mr:`283`, :mr:`286`
+    - add MaranaX mnemonics :mr:`285`
+    - add Chem diagnostics and Gotthard II mnemonics :mr:`292`
+    - make hRIXS centroid threshold configurable :mr:`298`
+    - drop MaranaX non-existing mnemonics and add attributes on dataset :mr:`300`
+    - streamline digitizer functions :mr:`304`
+    - add BAM average and BAM feedbacks mnemonics, Viking mnemonics :mr:`305`, :mr:`306`
+    - improved function to load PES spectra :mr:`309`
+    - Gotthard-II mnemonics and pulse alignment :mr:`310`, :mr:`311`, `mr:330`
+    - Adds ``AppleX`` polarization mnemonics :mr:`313`
+    - Add BAM 2955_S3 and update BAM mnemonics :mr:`315`
+    - improve peak-finding algorithm for digitizer trace peak extraction :mr:`312`
+    - update documentation on digitizer peak extraction :mr: `312`
+    - split documentation howtos and change theme :mr:`328`
+
+- **New Features**
+
+    - fix :issue:`75` add frame counts when aggregating RIXS data :mr:`274`
+    - BOZ normalization using 2D flat field polylines for S K-edge :mr:`293`
+    - BOZ GPU notebook :mr:`294`
+    - update hRIXS class for MaranaX :mr:`287`
+    - introduce MaranaX class for parallelized centroiding :mr:`297`
+    - New PES functions to save / load average traces :mr:`309`
+    - save peaks from raw digitizer traces as processed data :mr:`312`
+    - allow different modes for peak integration :mr:`329`
+
+1.7.0
+-----
+
+- **Bug fixes**
+
+    - fix :issue:`61` regarding sign of XAS in some cases :mr:`207`
+    - Use xarray.values instead of .to_numpy() for backward-compatibility :mr:`214`
+    - fix :issue:`23` regarding API documentation generation :mr:`221`
+    - fix :issue:`64` regarding loading a subset of trains :mr:`226`, :mr:`230`
+    - fix :issue:`67` regarding bunch pattern :mr:`245`
+    - fix :issue:`44` regarding notebook for DSSC geometry quadrants alignment :mr:`247`
+    - fix :issue:`52` use extra-geom detector helper function for azimuthal integration :mr:`248`
+    - fix :issue:`18` regarding using reserved partition with sbatch :mr:`250`
+    - fix :issue:`68` now using findxfel command :mr:`256`
+    - fix :issue:`69` to speed up BOZ Small data and deal with missing intra-darks :mr:`256`
+    - fix :issue:`71` regarding rounding error in BOZ analysis related to timing :mr:`256`
+    - fix :issue:`40` update documentation regarding toolbox deployment folder in scratch :mr:`268`
+
+
+- **Improvements**
+
+    - remove calls to matplotlib tight_layout :mr:`206`
+    - Improved hRIXS class and utilities :mr:`182`
+    - Documentation on extracting digitizer peaks, clean up of digitizer functions :mr:`215`
+    - Improved peak-finding algorithm for digitizer traces :mr:`216`, :mr:`227`
+    - Only load bunch pattern table when necessary :mr:`234`, :mr:`245`
+    - Document the HRIXS class :mr:`238`
+    - notebook example of DSSC azimuthal integration for time delay scans :mr:`249`
+    - provide drop-intra-darks option in BOZ analysis :mr:`256`
+    - improve automatic ROIs finding for BOZ analysis :mr:`256`
+    - prevent flat field correction from turning negative in BOZ analysis :mr:`259`
+    - update documentation to the new exfel-python environment :mr:`266`
+
+
+- **New Features**
+
+    -  Read signal description from Fast ADC and ADQ412 digitizers :mr:`209`, :mr:`212`
+    -  Mnemonics for XRD devices :mr:`208`
+    -  Add function to align OL to FEL pulse Id :mr:`218`
+    -  Add reflectivity routine :mr:`218`
+    -  Possibility to extract run values of mnemonics :mr:`220`, :mr:`232`
+
+    -  Add get_undulator_config function :mr:`225`
+    -  Document the HRIXS class :mr:`238`
+    -  Include double counts for hRIXS SPC algorithms :mr:`239`
+    -  Add Viking spectrometer analysis class :mr:`240`
+    -  Add GPU accelaration for BOZ correction determination :mr:`254`
+    -  Issues warning when loading data with > 5% missing trains :mr:`263`
+
+
+1.6.0
+-----
+
+- **Bug fixes**
+
+    - fix :issue:`45` SLURM scripts embedded in and download link available
+      from documentation :mr:`171`
+    - fix :issue:`8` regarding azimuthal integration with pyFAI and hexagonal
+      DSSC pixel splitting by providing an example notebook :mr:`174`
+    - fix :issue:`46` with a change in dask groupby mean behavior :mr:`174`
+    - fix :issue:`47` SLURM script not using the correct kernel :mr:`176`
+    - fix :issue:`51` make sure that BAM units are in ps :mr:`183`
+    - fix :issue:`50` and :issue:`54` relating to package dependencies
+    - fix :issue:`57` adds target mono energy mnemonic
+    - fix :issue:`55` implementingd dask auto rechunking in notebooks
+    - fix :issue:`53` wrong flat field correction sometimes being calculated 
+    - fix :issue:`56` future warning on xarray.ufuncs :mr:`189`
+
+- **Improvements**
+
+    - update version of BAM mnemonics :mr:`175`
+    - update version GATT-related mnemonics, add `transmission_col2` :mr:`172`
+    - reorganize the Howto section :mr:`169`
+    - improve SLURM scripts with named arguments :mr:`176`
+    - adds notebook for DSSC fine timing analysis :mr:`184` and :mr:`185`
+    - numerous improvements for the flat field correction calculation
+      in the BOZ analysis, including fitting domain functions, hexagonal
+      pixel lattice, possibility to switch off flat field symmetry constraints
+      and a refine fit function with regularization term :mr:`186`
+    - simplifies flat field calculation by using directly the refined fit
+      procedure which works with far fewer input parameters :mr:`202`
+
+- **New Features**
+
+    - add routine for fluence calibration :mr:`180`
+    - add Fast ADC 2 mnemonics :mr:`200`
+
+1.5.0
+-----
+
+- **Bug fixes**
+
+    - fix :issue:`39` providing a changelog in the documentation :mr:`164`
+    - fix :issue:`37` BOZ analysis :mr:`158`
+    - fix :issue:`36` mnemonics :mr:`159`
+    - fix :issue:`35` BOZ notebook dependencies :mr:`157`
+    - fix :issue:`34` BOZ time delay calculations and plotting :mr:`154`
+    - fix :issue:`32` significantly speeding up in XAS binning calculation :mr:`151`
+    - fix :issue:`27` improving BOZ analysis :mr:`146`
+    - fix :issue:`28` pp pattern in DSSC dask binning :mr:`144`
+    - fix :issue:`26` several BOZ analysis improvements :mr:`135`
+
+- **Improvements**
+
+    - checks if single-version mnemonics is in all_sources :mr:`163`
+    - add :code:`get_bam_params()` :mr:`160`
+    - only check keys if menmonic has more than one version :commit:`ae724d3c`
+    - add FFT focus lens mnemonics :mr:`156`
+    - add dask as dependency :mr:`155`
+    - renamed FFT sample Z mnemonics :mr:`153`
+    - add virtual sample camera LLC_webcam1 into mnemonics :mr:`152`
+    - fix digitizer check params :mr:`149`
+    - improve installation instruction :mr:`145`
+    - add Newton camera :mr:`142`
+    - simplified :code:`mnemonics_for_run()` :commit:`3cc98c16`
+    - adds Horizontal FDM to mnemonics :mr:`141`
+    - add setup documentation :mr:`140`
+    - numerous PES fixes :mr:`143`, :mr:`130`, :mr:`129`, :mr:`138`, :mr:`137`
+    - change in FFT sample Z mnemonics :mr:`125` and :mr:`124`
+    - add MTE3 camera :mr:`123`
+    - add KB benders averager :mr:`120` and :mr:`119`
+
+- **New Features**
+
+    - implement the Beam-splitting Off-axis Zone-plate analysis:
+      :mr:`150`, :mr:`139`, :mr:`136`, :mr:`134`, :mr:`133`, :mr:`132`,
+      :mr:`131`, :mr:`128`, :mr:`127`, :mr:`126`, :mr:`115`
+    - introduce dask assisted DSSC binning, fixing :issue:`24` and :issue:`17`
+
+1.4.0
+-----
+
+- **Bug fixes**
+
+    - fix :issue:`22` using extra-data read machinery :mr:`105`
+    - fix :issue:`21` and :issue:`12` introducing mnemonics version :mr:`104`
+
+- **Improvements**
+
+    - fix :code:`get_array()`, add wrappers to some of `extra_data` basic
+      functions :mr:`116`
+    - new FastADC mnemonics :mr:`112`
+    - refactor packaging :mr:`106`
+    - add :code:`load_bpt()` function :commit:`9e2c1107`
+    - add XTD10 MCP mnemonics :commit:`8b550c9b`
+    - add :code:`digitizer_type()` function :commit:`75eb0bca`
+    - separate FastADC and ADQ412 code :commit:`939d32b9`
+    - documentation centralized on rtd.xfel.eu :mr:`103`
+    - simplify digitizer functions and pulseId coordinates assignment for XGM
+      and digitizers :mr:`100`
+
+- **New Features**
+
+    - base knife-edge scan analysis implementation :mr:`107`
+    - add PES :mr:`108`
+    - integrate documentation to rtd.xfel.eu :mr:`103` and :mr:`99`
+
+
+1.1.2rc1
+--------
+
+- **Bug Fixes**
+
+- **Improvements**
+
+- **New Feature**
+
+    - introduce change in package structure, sphinx documentation and
+      DSSC binning class :mr:`87`

doc/conf.py

+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('..'))
+
+# -- Project information -----------------------------------------------------
+
+project = 'SCS Toolbox'
+copyright = '2025, SCS'
+author = 'SCS'
+
+# The short X.Y version
+version = ''
+# The full version, including alpha/beta/rc tags
+release = ''
+
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.mathjax',
+    'sphinx.ext.viewcode',
+    'sphinx.ext.coverage',
+    'sphinx.ext.napoleon',
+    'sphinx.ext.extlinks',
+    'autoapi.extension',
+    'nbsphinx'
+]
+
+nbsphinx_execute = 'never'
+
+autoapi_dirs = ['../src/toolbox_scs']
+autoapi_ignore = ['*/deprecated/*']
+
+extlinks = {'issue': ('https://git.xfel.eu/SCS/ToolBox/-/issues/%s',
+                      'issue:%s'),
+            'mr': ('https://git.xfel.eu/SCS/ToolBox/-/merge_requests/%s',
+                   'MR:%s'),
+            'commit': ('https://git.xfel.eu/SCS/ToolBox/-/commit/%s',
+                       'commit:%s')
+            }
+
+# Don't add .txt suffix to source files:
+html_sourcelink_suffix = ''
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = None
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'pydata_sphinx_theme'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'SCSToolboxdoc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'SCSToolbox.tex', 'SCS Toolbox Documentation',
+     'SCS', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'scstoolbox', 'SCS Toolbox Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'SCSToolbox', 'SCS Toolbox Documentation',
+     author, 'SCSToolbox', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+# -- Options for Epub output -------------------------------------------------
+
+# Bibliographic Dublin Core info.
+epub_title = project
+
+# The unique identifier of the text. This can be a ISBN number
+# or the project homepage.
+#
+# epub_identifier = ''
+
+# A unique identification for the text.
+#
+# epub_uid = ''
+
+# A list of files that should not be packed into the epub file.
+epub_exclude_files = ['search.html']
+
+
+# -- Extension configuration -------------------------------------------------

doc/digitizers.rst

+Digitizers
+==========
+
+.. toctree::
+
+   Extracting digitizer peaks <How to extract peaks from digitizer traces>
+
+
+Point detectors
+---------------
+Detectors that produce one point per pulse, or 0D detectors, are all handled in
+a similar way. Such detectors are, for instance, the X-ray Gas Monitor (XGM),
+the Transmitted Intensity Monitor (TIM), the electron Bunch Arrival
+Monitor (BAM) or the photo diodes monitoring the PP laser.
+
+

doc/dssc/DSSCBinner.rst

+Basic principle
+===============
+
+In a DSSC binner object we basically define maps, here called 'binners'. They define into which bucket a value along a certain dimension will be put into.
+
+map: coordinate array along specified dimension -> array of buckets
+
+
+Example 1
+=========
+
+Bin static run
+~~~~~~~~~~~~~~
+
+.. code:: ipython3
+
+    import os
+    import logging
+    import importlib
+    
+    import numpy as np
+    import xarray as xr
+    import pandas as pd
+    
+    import extra_data as ed
+    import toolbox_scs as tb
+    import toolbox_scs.detectors as tbdet
+    import toolbox_scs.detectors.dssc_plot as dssc_plot
+
+    #logging.basicConfig(level=logging.INFO)
+
+.. code:: ipython3
+
+    # run settings
+    proposal_nb = 2711
+    run_nb = 97
+
+.. code:: ipython3
+
+    # create a extra_data run object and collect detector information
+    run = tb.load_run(proposal_nb, run_nb)
+    dssc_info = tbdet.load_dssc_info(proposal_nb, run_nb)
+
+.. code:: ipython3
+
+    # create toolbox bin object
+    bin_obj = tbdet.DSSCBinner(proposal_nb, run_nb)
+
+.. code:: ipython3
+
+    # create array that will map the pulse dimension into 
+    # buckets 'image' and 'dark'  -> resulting dimension length = 2
+    buckets_pulse = ['image', 'dark'] * 30 # put all 60 pulses into buckets image and dark
+    
+    
+    # create array that will map the trainId dimension into 
+    # buckets [0]  -> resulting dimension length = 1
+    buckets_train = np.zeros(len(run.train_ids)).astype(int) # put all train Ids to the same bucket called 0.
+
+.. code:: ipython3
+
+    #create binners (xarray data arrays that basically act as a map)
+    fpt = dssc_info['frames_per_train']
+    binnertrain = tbdet.create_dssc_bins("trainId",run.train_ids,buckets_train)
+    binnerpulse = tbdet.create_dssc_bins("pulse",np.linspace(0,fpt-1,fpt, dtype=int),buckets_pulse)
+
+
+.. code:: ipython3
+
+    # add binners to bin object
+    bin_obj.add_binner('trainId', binnertrain)
+    bin_obj.add_binner('pulse', binnerpulse)
+
+.. code:: ipython3
+
+    # get a prediction of how the data will look like (minus module dimension)
+    bin_obj.get_info()
+
+
+.. parsed-literal::
+
+    Frozen(SortedKeysDict({'trainId': 1, 'pulse': 2, 'x': 128, 'y': 512}))
+
+
+.. code:: ipython3
+
+    # bin 2 modules in parallel
+    
+    mod_list = [0,15]
+    bin_obj.bin_data(chunksize=248, modules=mod_list, filepath='./')
+
+.. code:: ipython3
+
+    # Save metadata into file
+    fname = 'testfile.h5'
+    tbdet.save_xarray(fname, binnertrain, group='binner1', mode='a')
+    tbdet.save_xarray(fname, binnerpulse, group='binner2', mode='a')
+
+Example2
+========
+
+bin pump-probe data
+~~~~~~~~~~~~~~~~~~~
+
+.. code:: ipython3
+
+    # run settings
+    proposal_nb = 2212
+    run_nb = 235
+
+.. code:: ipython3
+
+    # Collect information about run
+    run_obj = tb.load_run(proposal_nb, run_nb)
+    detector_info = tbdet.load_dssc_info(proposal_nb, run_nb)
+
+.. code:: ipython3
+
+    bin_obj = tbdet.DSSCBinner(proposal_nb, run_nb)
+
+.. code:: ipython3
+
+    # define buckets
+    buckets_trainId = (tb.get_array(run_obj, 'PP800_PhaseShifter', 0.03)).values
+    buckets_pulse = ['pumped', 'unpumped'] * 10
+    
+    # create binner
+    binnerTrain = tbdet.create_dssc_bins("trainId",
+                                    detector_info['trainIds'],
+                                    buckets_trainId)
+    binnerPulse = tbdet.create_dssc_bins("pulse",
+                                    np.linspace(0,19,20, dtype=int),
+                                    buckets_pulse)
+
+.. code:: ipython3
+
+    bin_obj.add_binner('trainId', binnerTrain)
+    bin_obj.add_binner('pulse', binnerPulse)
+
+
+.. code:: ipython3
+
+    # get a prediction of how the data will look like (minus module dimension)
+    bin_obj.get_info()
+
+
+.. parsed-literal::
+
+    Frozen(SortedKeysDict({'trainId': 271, 'pulse': 2, 'x': 128, 'y': 512}))
+
+
+.. code:: ipython3
+
+    # bin 2 modules using the joblib module to precess the two modules 
+    # in parallel.
+
+    bin_obj.bin_data(chunksize=248, modules=mod_list, filepath='./')

doc/howtos.rst

+.. _how to's:
+
+How to's
+~~~~~~~~
+
+.. toctree::
+   :maxdepth: 1
+
+   Loading data <load>
+   Reading bunch pattern <bunch_pattern_decoding>
+   Knife-edge and fluence calculation <Knife edge scan and fluence calculation>
+   Extracting digitizers traces <digitizers>
+   Fine timing <transient reflectivity>
+   Processing DSSC data <DSSC>
+   BOZ analysis <BOZ>
+   PES analysis <PES_spectra_extraction>
+   HRIXS analysis <HRIXS>
+   Viking analysis <Analysis_of_Viking_spectrometer_data>
+   SLURM
+
+

doc/index.rst

+SCS Toolbox
+===========
+
+.. toctree::
+   :caption: Contents:
+   :maxdepth: 2
+
+   howtos.rst
+   maintainers.rst
+   changelog.rst
+
+Installation
+------------
+
+Recommended: Proposal Environment setup
++++++++++++++++++++++++++++++++++++++++
+
+The proposal specific environment is installed by launching in a shell
+terminal on Maxwell:
+
+.. code:: bash
+
+    module load exfel exfel-python
+    scs-activate-toolbox-kernel --proposal 2780
+
+where in this example 2780 is the proposal number. After this and refreshing the
+browser, a new kernel named ``SCS Toolbox (p002780)`` is available and should
+be used to run jupyter notebooks on the Maxwell Jupyter hub.
+
+Figures setup: enabling matplotlib constrained layout
++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+To get the best looking figures generated by the SCS Toolbox, you
+need to enable the experimental constrained_layout_ solver in matplotlib. This
+is done in jupyter notebook with adding at the start the following lines:
+
+.. code:: python
+
+   import matplotlib.pyplot as plt
+   plt.rcParams['figure.constrained_layout.use'] = True
+
+.. _constrained_layout: https://matplotlib.org/stable/tutorials/intermediate/constrainedlayout_guide.html
+
+
+Alternative: Manual ToolBox Installation
+++++++++++++++++++++++++++++++++++++++++
+
+The ToolBox may be installed in any environment. However, it depends on the
+extra_data and the euxfel_bunch_pattern package, which are no official third
+party python modules. Within environments where the latter are not present, they
+need to be installed by hand.
+
+In most cases, you will want to install it in the *exfel_python* environment
+which is the one corresponding to the *xfel* kernel on max-jhub.
+
+To do so, first activate that environment:
+
+.. code:: bash
+
+    module load exfel exfel-python
+    
+Then, check that the scs_toolbox is not already installed:
+
+.. code:: bash
+
+    pip show toolbox_scs
+
+If the toolbox has been installed in your home directory previously, everything
+is set up. If you need to upgrade the toolbox to a more recent version, you
+have to either uninstall the current version:
+
+.. code:: bash
+
+    pip uninstall toolbox_scs
+
+or if it was installed in development mode, go in the toolbox directory and
+pull the last commits from git:
+
+.. code:: bash
+
+    cd #toolbox top folder
+    git pull
+
+Otherwise it needs to be installed (only once). In that you first need to
+clone the toolbox somewhere (in your home directory for example) and install
+the package. Here the -e flag install the package in development mode, which
+means no files are copied but only linked such that any changes made to the
+toolbox files will have effect:
+
+.. code:: bash
+
+    cd ~
+    git clone https://git.xfel.eu/SCS/ToolBox.git
+    cd ~/ToolBox
+    pip install --user -e ".[maxwell]"
+
+
+Transferring Data
+-----------------
+
+The DAQ system save data on the online cluster. To analyze data on the
+Maxwell offline cluster, they need to be transferred there. This is achieved by
+login at:
+
+https://in.xfel.eu/metadata
+
+and then navigating to the proposal, then to the ``Runs`` tab from where runs can
+be transferred to the offline cluster by marking them as ``Good`` in the
+``Data Assessment``. Depending on the amount of data in the run, this can take a
+while.
+
+.. image:: metadata.png
+
+Processing Data
+---------------
+
+On the Maxwell Jupyter hub:
+
+https://max-jhub.desy.de
+
+notebooks can be executed using the corresponding Proposal Environment
+kernel or the ``xfel`` kernel if the toolbox was manually installed. For quick
+startup, example notebooks (.ipynb files) can be directly downloaded from the
+:ref:`How to's` section by clicking on the ``View page source``.
+
+
+Contribute
+----------
+
+For reasons of readability, contributions preferrably comply with the PEP8_ code structure guidelines.
+
+.. _PEP8: https://www.python.org/dev/peps/pep-0008/#a-foolish-consistency-is-the-hobgoblin-of-little-minds 
+
+The associated code checker, called 'flake8', can be installed via PyPi.
+