diff --git a/docs/source/available_notebooks.rst b/docs/source/available_notebooks.rst new file mode 100644 index 0000000000000000000000000000000000000000..8123e3ed4a525822368a471b93befd8e300f1810 --- /dev/null +++ b/docs/source/available_notebooks.rst @@ -0,0 +1,204 @@ +.. available_notebooks: + +Available Notebooks +=================== + +The following notebooks are currently integrated into the European XFEL +Offline Calibration tool chain. + + +LPD +--- + +Dark Images Evaluation +~~~~~~~~~~~~~~~~~~~~~~ + +This notebook produces offset, noise and derived bad pixel data from LPD +dark images for one capacitor setting. + +.. code:: + + % python calibrate_nbc.py LPD dark --help + usage: calibrate_nbc.py [-h] [--cluster-profile str] --in-folder str + --out-folder str [--sequences int [int ...]] + --capacitor-setting int --run-high int --run-med int + --run-low int [--mem-cells int] [--local-output] + [--db-output] [--bias-voltage int] + [--cal-db-interface str] + [--thresholds-offset-sigma float] + [--thresholds-offset-hard int [int ...]] + [--thresholds-noise-sigma float] + [--thresholds-noise-hard int [int ...]] + [--skip-first-ntrains int] [--use-dir-creation-date] + [--instrument str] [--ntrains int] + [--high-res-badpix-3d] + DETECTOR TYPE + + Main entry point for offline calibration + + positional arguments: + DETECTOR The detector to calibrate + TYPE Type of calibration: AGIPD,LPD + + optional arguments: + -h, --help show this help message and exit + --cluster-profile str + The ipcluster profile to use. Default: noDB + --in-folder str path to input data, required. Default: None + --out-folder str path to output to, required. Default: None + --sequences int [int ...] + sequence files to evaluate.. Default: [0] + --capacitor-setting int + capacitor_setting for which data was taken, required. + Default: None + --run-high int run number in which high gain data was recorded, + required. Default: None + --run-med int run number in which medium gain data was recorded, + required. Default: None + --run-low int run number in which low gain data was recorded, + required. Default: None + --mem-cells int number of memory cells used. Default: 128 + --local-output output constants locally. Default: True + --db-output output constants to database. Default: True + --bias-voltage int detector bias voltage. Default: 500 + --cal-db-interface str + the database interface to use. Default: tcp://max- + exfl015:5005 + --thresholds-offset-sigma float + Default: 3.0 + --thresholds-offset-hard int [int ...] + Default: [400, 1500] + --thresholds-noise-sigma float + Default: 7.0 + --thresholds-noise-hard int [int ...] + Default: [1, 35] + --skip-first-ntrains int + Default: 10 + --use-dir-creation-date + use the creation date of the directory for database + time derivation. Default: True + --instrument str Default: FXE + --ntrains int Default: 300 + --high-res-badpix-3d Default: False + + +Pulse Capacitor Evaluation +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This notebook produces per-pixel relative gain, normalizing all memory cells +for a given pixel w.r.t. the three gain stages for a given capacitor setting. +It requires offset maps as input. + +.. code:: + + % python calibrate_nbc.py LPD PC --help + usage: calibrate_nbc.py [-h] [--cluster-profile str] --in-folder str + --out-folder str [--offset-store str] + [--modules int [int ...]] --capacitor-setting int + [--mem-cells int] [--local-output] [--db-output] + [--db-input] [--bias-voltage int] + [--cal-db-interface str] [--use-dir-creation-date] + [--instrument str] [--high-res-badpix-3d] + DETECTOR TYPE + + Main entry point for offline calibration + + positional arguments: + DETECTOR The detector to calibrate + TYPE Type of calibration: AGIPD,LPD + + optional arguments: + -h, --help show this help message and exit + --cluster-profile str + The ipcluster profile to use. Default: noDB555 + --in-folder str path to input data, required. Default: None + --out-folder str path to output to, required. Default: None + --offset-store str path to offset store, overwrite use_db_input. Default: + /gpfs/exfel/exp/FXE/201830/p900020/proc/calibration/da + rk/lpd_offset_store_r0206_r0207_r0208_5pf.h5 + --modules int [int ...] + Default: None + --capacitor-setting int + capacitor_setting for which data was taken, required. + Default: None + --mem-cells int number of memory cells used. Default: 128 + --local-output output constants locally. Default: True + --db-output output constants to database. Default: False + --db-input Default: True + --bias-voltage int detector bias voltage. Default: 500 + --cal-db-interface str + the database interface to use. Default: tcp://max- + exfl015:5005 + --use-dir-creation-date + use the creation date of the directory for database + time derivation. Default: True + --instrument str Default: FXE + --high-res-badpix-3d Default: False + + +Correction +~~~~~~~~~~ + +Corrects an LPD run with calibration constants from the database and/or files. + +.. code:: + + % python calibrate_nbc.py LPD correct --help + usage: calibrate_nbc.py [-h] --in-folder str --run int --out-folder str + [--relgain-store str] --offset-store str + [--ff-store str] [--sequences str [str ...]] + [--mem-cells int] [--overwrite] [--no-relative-gain] + [--no-flat-fields] [--cluster-profile str] + [--max-pulses int] [--use-dir-creation-date] + [--max-cells-db int] [--rawversion int] + [--instrument str] [--local-input] [--db-input] + [--bias-voltage int] [--cal-db-interface str] + [--geometry-file str] + [--beam-center-offset float [float ...]] + DETECTOR TYPE + + Main entry point for offline calibration + + positional arguments: + DETECTOR The detector to calibrate + TYPE Type of calibration: LPD,AGIPD + + optional arguments: + -h, --help show this help message and exit + --in-folder str the folder to read data from, required. Default: None + --run int runs to process, required. Default: None + --out-folder str the folder to output to, required. Default: None + --relgain-store str path to base constant store. Default: /gpfs/exfel/exp/ + FXE/201830/p900020/proc/calibration/CI5pf_pp/lpd_ci_st + ore_CHANID_16_5pf.h5 + --offset-store str path to offset constant store, required. Default: None + --ff-store str path to flat fields. Default: /gpfs/exfel/exp/FXE/2018 + 30/p900020/proc/calibration/flat_field_ci_pp/lpd_flatf + ield_store_r0230.h5 + --sequences str [str ...] + sequences to correct, set to -1 for all, range + allowed. Default: [-1] + --mem-cells int memory cells in data. Default: 512 + --overwrite set to True if existing data should be overwritten. + Default: False + --no-relative-gain do not do relative gain correction. Default: False + --no-flat-fields do not do flat field correction. Default: False + --cluster-profile str + Default: noDB + --max-pulses int Default: 128 + --use-dir-creation-date + Default: False + --max-cells-db int Default: 128 + --rawversion int Default: 2 + --instrument str Default: FXE + --local-input output constants locally. Default: True + --db-input output constants to database. Default: True + --bias-voltage int detector bias voltage. Default: 500 + --cal-db-interface str + the database interface to use. Default: tcp://max- + exfl014:5005 + --geometry-file str the geometry file to use, MAR 2018. Default: + /gpfs/exfel/d/cal/exchange/lpdMF_00.h5 + --beam-center-offset float [float ...] + offset from the beam center, MAR 2018. Default: [1.5, + 1] diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst index fba37b8a469a3fe0e4280777b1174db332e8758b..5e4e6bcd4ba5549a395db2c7a45b5946c58b77b6 100644 --- a/docs/source/configuration.rst +++ b/docs/source/configuration.rst @@ -1,4 +1,4 @@ -.. _configuration +.. _configuration: Configuration ============= diff --git a/docs/source/how_it_works.rst b/docs/source/how_it_works.rst new file mode 100644 index 0000000000000000000000000000000000000000..474990a873076a1efa21453f5d7aac726251c6a9 --- /dev/null +++ b/docs/source/how_it_works.rst @@ -0,0 +1,15 @@ +How it Works +============ + +The European XFEL Offline Calibration utilizes the tools nbconvert_ and nbparameterise_ +to expose Jupyter_ notebooks to a command line interface. + +Additionally, it leverages the DESY/XFEL Maxwell cluster to run these jobs in parallel +via SLURM_. + +Here is a list of `available notebooks`_. + +.. _nbparameterise: https://github.com/takluyver/nbparameterise +.. _nbconver: https://github.com/jupyter/nbconvert +.. _jupyter: http://jupyter.org/ +.. _SLURM: https://slurm.schedmd.com \ No newline at end of file diff --git a/docs/source/installation.rst b/docs/source/installation.rst index e8234b41333232fdeb27676e5ac13bb7057067d1..91a2fece1b4a77f7e405a2e49203f1a431fc7ee1 100644 --- a/docs/source/installation.rst +++ b/docs/source/installation.rst @@ -39,5 +39,5 @@ all prerequisites and this tool chain by running:: pip install -r requirements.txt . -After installation, you should adjust the `configuration`_ and settings to +After installation, you should adjust the configuration_ and settings to match your environment. \ No newline at end of file diff --git a/docs/source/workflow.rst b/docs/source/workflow.rst index c501405dbcffb4ea0b68074debb4addf723a861b..5a7191c21e3c740abef1073ae3d431a81496f691 100644 --- a/docs/source/workflow.rst +++ b/docs/source/workflow.rst @@ -294,7 +294,7 @@ launch a concurrent job and generate a report out of notebook. If it fails to ru normal Jupyter notebook environment, it will certainly fail in the tool chain environment. Once you are satisfied with your current state of initial development, you can add it -to the list of notebooks as mentioned in the `configuration`_ section. +to the list of notebooks as mentioned in the configuration_ section. Any changes you now make in the notebook will be automatically propagated to the command line. Specifically, you should verify that all arguments are parsed correctly, e.g. by calling::