diff --git a/docs/advanced.md b/docs/advanced.md
new file mode 100644
index 0000000000000000000000000000000000000000..36dec436479a26fabef7736031e0b22cf30251ed
--- /dev/null
+++ b/docs/advanced.md
@@ -0,0 +1,169 @@
+# Advanced Topics
+
+The following tasks should only be carried out by trained staff.
+
+Request dark characterization
+-----------------------------
+
+The script runs dark characterization notebook with default parameters
+via web service. The data needs to be transferred via the MDC, however,
+the web service will wait for any transfer to be completed. The detector
+is chosen automatically with respect to given instrument
+([\--instrument]). For AGIPD, LPD, Jungfrau runs for the
+three gain stages need to be given ([\--run-high],
+[\--run-med], [\--run-low]). For FastCCD and
+ePIX only a single run needs to be given ([\--run]).
+
+The complete list of parameters is:
+
+ -h, --help show this help message and exit
+ --proposal PROPOSAL The proposal number, without leading p, but with
+ leading zeros
+ --instrument {SPB,MID,FXE,SCS,SQS,HED}
+ The instrument
+ --cycle CYCLE The facility cycle
+ --run-high RUN_HIGH Run number of high gain data as an integer
+ --run-med RUN_MED Run number of medium gain data as an integer
+ --run-low RUN_LOW Run number of low gain data as an integer
+ --run RUN Run number as an integer
+
+The path to data files is defined from script parameters. A typical data
+path, which can be found in the MDC is:
+
+ /gpfs/exfel/exp/MID/201930/p900071/raw
+
+Where [MID] is an instrument name, [201930] is a
+cycle, [900071] is a proposal number.
+
+Extending Correction Notebooks on User Request
+----------------------------------------------
+
+Internally, each automated correction run will trigger
+[calibrate\_nbc.py] to be called anew on the respective
+notebook. This means that any changes to save to this notebook will be
+picked up for subsequent runs.
+
+This can be useful to add user requests while running. For this:
+
+1. create a working copy of the notebook in question, and create a
+ commit of the production notebook to fall back to in case of
+ problems:
+
+ > `` ` git add production_notebook_NBC.py git commit -m "Known working version before edits" cp production_notebook_NBC.py production_notebook_TEST.py ``\`
+
+2. add any feature there and *thoroughly* test them
+3. when you are happy with the results, copy them over into the
+ production notebook and save.
+
+
+???+ warning
+
+ Live editing of correction notebooks is fully at your responsibility. Do
+ not do it if you are not 100% sure you know what you are doing.
+
+1. If it fails, revert to the original state, ideally via git:
+
+ git checkout HEAD -- production_notebook_NBC.py
+
+2. Any runs which did not correct do to failures of the live edit can
+ then be relaunched manually, assuming the correction notebook allows
+ run and overwrite parameters:
+
+ xfel-calibrate ...... --run XYZ,ZXY-YYS --overwrite
+
+Using a Parameter Generator Function
+------------------------------------
+
+By default, the parameters to be exposed to the command line are deduced
+from the first code cell of the notebook, after resolving the notebook
+itself from the detector and characterization type. For some
+applications it might be beneficial to define a context-specific
+parameter range within the same notebook, based on additional user
+input. This can be done via a parameter generation function which is
+defined in one of the code cell:
+
+ def extend_parms(detector_instance):
+ from iCalibrationDB import Conditions
+ import inspect
+ existing = set()
+ def extract_parms(cls):
+ args, varargs, varkw, defaults = inspect.getargspec(cls.__init__)
+ pList = []
+ for i, arg in enumerate(args[1:][::-1]):
+ if arg in existing:
+ continue
+
+ existing.add(arg)
+
+ if i < len(defaults):
+ default = defaults[::-1][i]
+ if str(default).isdigit():
+ pList.append("{} = {}".format(arg, default))
+ elif default is None or default == "None":
+ pList.append("{} = \"None\"".format(arg))
+ else:
+ pList.append("{} = \"{}\"".format(arg, default))
+ else:
+ pList.append("{} = 0. # required".format(arg))
+ return set(pList[::-1]) # mandatories first
+ dtype = "LPD" if "LPD" in detector_instance.upper() else "AGIPD"
+ all_conditions = set()
+ for c in dir(Conditions):
+ if c[:2] != "__":
+ condition = getattr(Conditions, c)
+ parms = extract_parms(getattr(condition, dtype))
+ [all_conditions.add(p) for p in parms]
+ return "\n".join(all_conditions)
+
+
+???+ note
+
+ Note how all imports are inlined, as the function is executed outside
+ the notebook context.
+
+
+In the example, the function generates a list of additional parameters
+depending on the [detector\_instance] given. Here,
+[detector\_instance] is defined in the first code cell the
+usual way. Any other parameters defined such, that have names matching
+those of the generator function signature are passed to this function.
+The function should then return a string containing additional code to
+be appended to the first code cell.
+
+To make use of this functionality, the parameter generator function
+needs to be configured in [notebooks.py], e.g. :
+
+ ...
+ "GENERIC": {
+ "DBTOH5": {
+ "notebook": "notebooks/generic/DB_Constants_to_HDF5_NBC.ipynb",
+ "concurrency": {"parameter": None,
+ "default concurrency": None,
+ "cluster cores": 32},
+ "extend parms": "extend_parms",
+ },
+ }
+ ...
+
+To generically query which parameters are defined in the first code
+cell, the code execution history feature of iPython can be used:
+
+ ip = get_ipython()
+ session = ip.history_manager.get_last_session_id()
+ first_cell = next(ip.history_manager.get_range(session, 1, 2, raw=True))
+ _, _, code = first_cell
+ code = code.split("\n")
+ parms = {}
+ for c in code:
+ n, v = c.split("=")
+ n = n.strip()
+ v = v.strip()
+ try:
+ parms[n] = float(v)
+ except:
+ parms[n] = str(v) if not isinstance(v, str) else v
+ if parms[n] == "None" or parms[n] == "'None'":
+ parms[n] = None
+
+This will create a dictionary [parms] which contains all
+parameters either as [float] or [str] values.
diff --git a/docs/available_notebooks.md b/docs/available_notebooks.md
new file mode 100644
index 0000000000000000000000000000000000000000..0d1c6956d16624fc442897f2006f3487fd9e380b
--- /dev/null
+++ b/docs/available_notebooks.md
@@ -0,0 +1,5437 @@
+
+.. _available_notebooks:
+
+Available Notebooks
+===================
+
+The following notebooks are currently integrated into the European XFEL
+Offline Calibration tool chain.
+
+
+AGIPD
+-----
+
+Combine Constants
++++++++++++++++++
+
+Author: S. Hauf, Version: 0.1
+
+This notebook combines constants from various evaluations
+
+- dark image analysis, yielding offset and noise
+- flat field analysis, yielding X-ray gain
+- pulse capacitor analysis, yielding medium gain stage slopes and
+ thresholds
+- charge injection analysis, yielding low gain stage slopes and
+ thresholds
+
+into a single set of calibration constants. These constants do not
+include offset and noise as they need to be reevaluated more frequently.
+
+Additionally, a bad pixel mask for all gain stages is deduced from the
+input. The mask contains dedicated entries for all pixels and memory
+cells as well as all three gains stages.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD COMBINE --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--modules str [str ...]]
+ --out-folder str [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--bias-voltage int]
+ [--cal-db-interface str] [--mem-cells int]
+ [--instrument str] [--photon-energy float]
+ [--offset-store str] [--gain-store str]
+ [--pc-store-base str] [--ci-store str]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ [--db-input | --no-db-input]
+ [--deviation-threshold float]
+ [--max-dev-high-gain float] [--max-dev-med-gain float]
+ [--threshold-bounds-high-med int [int ...]]
+ [--thresholds-offset-sigma float]
+ [--thresholds-offset-hard int [int ...]]
+ [--thresholds-noise-sigma float]
+ [--thresholds-noise-hard int [int ...]]
+ [--thresholds-xraygain-hard-high float [float ...]]
+ [--thresholds-xraygain-hard-med float [float ...]]
+ [--thresholds-xraygain-hard-low float [float ...]]
+ [--no-flat-fields | --no-no-flat-fields]
+ AGIPD COMBINE
+
+ # Combine Constants #
+ ,
+ Author: S. Hauf, Version: 0.1,
+ ,
+ This notebook combines constants from various evaluations,
+ ,
+ * dark image analysis, yielding offset and noise,
+ * flat field analysis, yielding X-ray gain,
+ * pulse capacitor analysis, yielding medium gain stage slopes and thresholds,
+ * charge injection analysis, yielding low gain stage slopes and thresholds,
+ ,
+ into a single set of calibration constants. These constants do not include
+ offset and noise as they need to be reevaluated more frequently.,
+ ,
+ Additionally, a bad pixel mask for all gain stages is deduced from the input.
+ The mask contains dedicated entries for all pixels and memory cells as well
+ as all three gains stages.
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ COMBINE Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --modules str [str ...]
+ modules to work on, range allowed. Default: [-1]
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --bias-voltage int detector bias voltage. Default: 300
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:5005
+ --mem-cells int number of memory cells used. Default: 64
+ --instrument str Default: SPB
+ --photon-energy float
+ the photon energy in keV. Default: 9.2
+ --offset-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20
+ 1830/p900019/usr/calibration0618/dark0.5MHz/agipd_offs
+ et_store_r0852_r0853_r0854.h5
+ --gain-store str for file-based access. Default: /gpfs/exfel/exp/SPB/20
+ 1830/p900019/usr/calibration0618/FF/agipd_gain_store_r
+ 0820_modules_CHANID.h5
+ --pc-store-base str for file-based access, use CHANID to indicate module
+ ranges. Default: /gpfs/exfel/exp/SPB/201831/p900039/us
+ r/AGIPD/PC/Veto_1MHZ/agipd_pc_store_38_39_40_41_42_43_
+ 44_60_CHANID_CHANID.h5
+ --ci-store str for file based access. Default: /gpfs/exfel/data/scrat
+ ch/haufs/gain_Data/AGIPD_ci_data_74.h5
+ --high-res-badpix-3d set this to True if you need high-resolution 3d bad
+ pixel plots. Runtime: ~ 1h. Default: False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+ --db-input retreive data from calibration database, setting
+ offset-store will overwrite this. Default: False
+ --no-db-input Opposite of --db-input (default: False)
+ --deviation-threshold float
+ peaks with an absolute location deviation larger than
+ the medium are are considere bad. Default: 0.75
+ --max-dev-high-gain float
+ Default: 0.2
+ --max-dev-med-gain float
+ Default: 0.5
+ --threshold-bounds-high-med int [int ...]
+ Default: [100, 8100]
+ --thresholds-offset-sigma float
+ Default: 5.0
+ --thresholds-offset-hard int [int ...]
+ Default: [4000, 9000]
+ --thresholds-noise-sigma float
+ Default: 10.0
+ --thresholds-noise-hard int [int ...]
+ Default: [1, 20]
+ --thresholds-xraygain-hard-high float [float ...]
+ Default: [0.75, 1.25]
+ --thresholds-xraygain-hard-med float [float ...]
+ Default: [0.001, 0.25]
+ --thresholds-xraygain-hard-low float [float ...]
+ Default: [0.0001, 0.025]
+ --no-flat-fields Default: False
+ --no-no-flat-fields Opposite of --no-flat-fields (default: False)
+
+ required arguments:
+ --out-folder str path to output to, required. Default: None
+
+
+AGIPD Offline Correction
+++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+Offline Calibration for the AGIPD Detector
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequences str [str ...]] [--modules str [str ...]]
+ [--train-ids str [str ...]] --run int [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-template str]
+ [--path-template str] [--instrument-source-template str]
+ [--index-source-template str]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--slopes-ff-from-files str] [--creation-time str]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--creation-date-offset str] [--mem-cells int]
+ [--bias-voltage int] [--acq-rate float]
+ [--gain-setting int] [--gain-mode int]
+ [--max-pulses int [int ...]] [--mem-cells-db int]
+ [--integration-time int] [--blc-noise-threshold int]
+ [--cm-dark-fraction float]
+ [--cm-dark-range float [float ...]] [--cm-n-itr int]
+ [--hg-hard-threshold int] [--mg-hard-threshold int]
+ [--noisy-adc-threshold float] [--ff-gain float]
+ [--photon-energy float]
+ [--only-offset | --no-only-offset]
+ [--rel-gain | --no-rel-gain]
+ [--xray-gain | --no-xray-gain]
+ [--blc-noise | --no-blc-noise]
+ [--blc-stripes | --no-blc-stripes]
+ [--blc-hmatch | --no-blc-hmatch]
+ [--match-asics | --no-match-asics]
+ [--adjust-mg-baseline | --no-adjust-mg-baseline]
+ [--zero-nans | --no-zero-nans]
+ [--zero-orange | --no-zero-orange]
+ [--blc-set-min | --no-blc-set-min]
+ [--corr-asic-diag | --no-corr-asic-diag]
+ [--force-hg-if-below | --no-force-hg-if-below]
+ [--force-mg-if-below | --no-force-mg-if-below]
+ [--mask-noisy-adc | --no-mask-noisy-adc]
+ [--common-mode | --no-common-mode]
+ [--melt-snow | --no-melt-snow]
+ [--mask-zero-std | --no-mask-zero-std]
+ [--low-medium-gap | --no-low-medium-gap]
+ [--round-photons | --no-round-photons]
+ [--use-ppu-device str] [--ppu-train-offset int]
+ [--use-litframe-finder str] [--litframe-device-id str]
+ [--energy-threshold int] [--use-super-selection str]
+ [--use-xgm-device str] [--recast-image-data str]
+ [--compress-fields str [str ...]]
+ [--skip-plots | --no-skip-plots] [--cell-id-preview int]
+ [--chunk-size int] [--n-cores-correct int]
+ [--n-cores-files int] [--sequences-per-node int]
+ [--max-nodes int] [--max-tasks-per-worker int]
+ AGIPD CORRECT
+
+ # AGIPD Offline Correction #
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ Offline Calibration for the AGIPD Detector
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --modules str [str ...]
+ modules to correct, set to -1 for all, range allowed.
+ Default: [-1]
+ --train-ids str [str ...]
+ train IDs to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-template str
+ inset for receiver devices. Default: {}CH0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{:05d}.h5
+ --instrument-source-template str
+ path in the HDF5 file to images. Default:
+ {}/DET/{}:xtdf
+ --index-source-template str
+ path in the HDF5 file to images. Default:
+ INDEX/{}/DET/{}:xtdf/
+ --ctrl-source-template str
+ path to control information. Default: {}/MDL/FPGA_COMP
+ --karabo-id-control str
+ karabo-id for control device. Default:
+ MID_EXP_AGIPD1M1
+ --slopes-ff-from-files str
+ Path to locally stored SlopesFF and BadPixelsFF
+ constants, loaded in precorrection notebook. Default:
+ --creation-time str To overwrite the measured creation_time. Required
+ Format: YYYY-MM-DD HR:MN:SC e.g. "2022-06-28
+ 13:00:00". Default:
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8015#8045
+ --cal-db-timeout int in milliseconds. Default: 30000
+ --creation-date-offset str
+ add an offset to creation date, e.g. to get different
+ constants. Default: 00:00:00
+ --mem-cells int Number of memory cells used, set to 0 to automatically
+ infer. Default: -1
+ --bias-voltage int bias voltage, set to 0 to use stored value in slow
+ data.. Default: -1
+ --acq-rate float the detector acquisition rate, use 0 to try to auto-
+ determine. Default: -1.0
+ --gain-setting int the gain setting, use -1 to use value stored in slow
+ data.. Default: -1
+ --gain-mode int gain mode (0: adaptive, 1-3 fixed high/med/low, -1:
+ read from CONTROL data). Default: -1
+ --max-pulses int [int ...]
+ range list [st, end, step] of memory cell indices to
+ be processed within a train. 3 allowed maximum list
+ input elements.. Default: [0, 352, 1]
+ --mem-cells-db int set to a value different than 0 to use this value for
+ DB queries. Default: -1
+ --integration-time int
+ integration time, negative values for auto-detection..
+ Default: -1
+ --blc-noise-threshold int
+ above this mean signal intensity now baseline
+ correction via noise is attempted. Default: 5000
+ --cm-dark-fraction float
+ threshold for fraction of empty pixels to consider
+ module enough dark to perform CM correction. Default:
+ 0.66
+ --cm-dark-range float [float ...]
+ range for signal value ADU for pixel to be consider as
+ a dark pixel. Default: [-50.0, 30]
+ --cm-n-itr int number of iterations for common mode correction.
+ Default: 4
+ --hg-hard-threshold int
+ threshold to force medium gain offset subtracted pixel
+ to high gain. Default: 1000
+ --mg-hard-threshold int
+ threshold to force medium gain offset subtracted pixel
+ from low to medium gain. Default: 1000
+ --noisy-adc-threshold float
+ threshold to mask complete adc. Default: 0.25
+ --ff-gain float conversion gain for absolute FlatField constants,
+ while applying xray_gain. Default: 7.2
+ --photon-energy float
+ photon energy in keV, non-positive value for XGM
+ autodetection. Default: -1.0
+ --only-offset Apply only Offset correction. if False, Offset is
+ applied by Default. if True, Offset is only applied..
+ Default: False
+ --no-only-offset Opposite of --only-offset (default: False)
+ --rel-gain do relative gain correction based on PC data. Default:
+ False
+ --no-rel-gain Opposite of --rel-gain (default: False)
+ --xray-gain do relative gain correction based on xray data.
+ Default: False
+ --no-xray-gain Opposite of --xray-gain (default: False)
+ --blc-noise if set, baseline correction via noise peak location is
+ attempted. Default: False
+ --no-blc-noise Opposite of --blc-noise (default: False)
+ --blc-stripes if set, baseline corrected via stripes. Default: False
+ --no-blc-stripes Opposite of --blc-stripes (default: False)
+ --blc-hmatch if set, base line correction via histogram matching is
+ attempted. Default: False
+ --no-blc-hmatch Opposite of --blc-hmatch (default: False)
+ --match-asics if set, inner ASIC borders are matched to the same
+ signal level. Default: False
+ --no-match-asics Opposite of --match-asics (default: False)
+ --adjust-mg-baseline adjust medium gain baseline to match highest high gain
+ value. Default: False
+ --no-adjust-mg-baseline
+ Opposite of --adjust-mg-baseline (default: False)
+ --zero-nans set NaN values in corrected data to 0. Default: False
+ --no-zero-nans Opposite of --zero-nans (default: False)
+ --zero-orange set to 0 very negative and very large values in
+ corrected data. Default: False
+ --no-zero-orange Opposite of --zero-orange (default: False)
+ --blc-set-min Shift to 0 negative medium gain pixels after offset
+ corr. Default: False
+ --no-blc-set-min Opposite of --blc-set-min (default: False)
+ --corr-asic-diag if set, diagonal drop offs on ASICs are corrected.
+ Default: False
+ --no-corr-asic-diag Opposite of --corr-asic-diag (default: False)
+ --force-hg-if-below set high gain if mg offset subtracted value is below
+ hg_hard_threshold. Default: False
+ --no-force-hg-if-below
+ Opposite of --force-hg-if-below (default: False)
+ --force-mg-if-below set medium gain if mg offset subtracted value is below
+ mg_hard_threshold. Default: False
+ --no-force-mg-if-below
+ Opposite of --force-mg-if-below (default: False)
+ --mask-noisy-adc Mask entire ADC if they are noise above a relative
+ threshold. Default: False
+ --no-mask-noisy-adc Opposite of --mask-noisy-adc (default: False)
+ --common-mode Common mode correction. Default: False
+ --no-common-mode Opposite of --common-mode (default: False)
+ --melt-snow Identify (and optionally interpolate) 'snowy' pixels.
+ Default: False
+ --no-melt-snow Opposite of --melt-snow (default: False)
+ --mask-zero-std Mask pixels with zero standard deviation across train.
+ Default: False
+ --no-mask-zero-std Opposite of --mask-zero-std (default: False)
+ --low-medium-gap 5 sigma separation in thresholding between low and
+ medium gain. Default: False
+ --no-low-medium-gap Opposite of --low-medium-gap (default: False)
+ --round-photons Round to absolute number of photons, only use with
+ gain corrections. Default: False
+ --no-round-photons Opposite of --round-photons (default: False)
+ --use-ppu-device str Device ID for a pulse picker device to only process
+ picked trains, empty string to disable. Default:
+ --ppu-train-offset int
+ When using the pulse picker, offset between the PPU's
+ sequence start and actually picked train. Default: 0
+ --use-litframe-finder str
+ Process only illuminated frames: 'off' - disable,
+ 'device' - use online device data, 'offline' - use
+ offline algorithm, 'auto' - choose online/offline
+ source automatically (default). Default: off
+ --litframe-device-id str
+ Device ID for a lit frame finder device, empty string
+ to auto detection. Default:
+ --energy-threshold int
+ The low limit for the energy (uJ) exposed by frames
+ subject to processing. If -1000, selection by pulse
+ energy is disabled. Default: -1000
+ --use-super-selection str
+ Make a common selection for entire run: 'off' -
+ disable, 'final' - enable for final selection, 'cm' -
+ enable only for common mode correction. Default: cm
+ --use-xgm-device str DoocsXGM device ID to obtain actual photon energy,
+ operating condition else.. Default:
+ --recast-image-data str
+ Cast data to a different dtype before saving. Default:
+ --compress-fields str [str ...]
+ Datasets in image group to compress.. Default:
+ ['gain', 'mask']
+ --skip-plots exit after writing corrected files and metadata.
+ Default: False
+ --no-skip-plots Opposite of --skip-plots (default: False)
+ --cell-id-preview int
+ cell Id used for preview in single-shot plots.
+ Default: 1
+ --chunk-size int Size of chunk for image-wise correction. Default: 1000
+ --n-cores-correct int
+ Number of chunks to be processed in parallel. Default:
+ 16
+ --n-cores-files int Number of files to be processed in parallel. Default:
+ 4
+ --sequences-per-node int
+ number of sequence files per cluster node if run as
+ SLURM job, set to 0 to not run SLURM parallel.
+ Default: 2
+ --max-nodes int Maximum number of SLURM jobs to split correction work
+ into. Default: 8
+ --max-tasks-per-worker int
+ the number of tasks a correction pool worker process
+ can complete before it will exit and be replaced with
+ a fresh worker process. Leave as -1 to keep worker
+ alive as long as pool.. Default: 1
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --run int runs to process, required. Default: None
+
+
+AGIPD Characterize Dark Images
+++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+The following code analyzes a set of dark images taken with the AGIPD
+detector to deduce detector offsets , noise, bad-pixel maps and
+thresholding. All four types of constants are evaluated per-pixel and
+per-memory cell. Data for the detector’s three gain stages needs to be
+present, separated into separate runs.
+
+The evaluated calibration constants are stored locally and injected in
+the calibration data base.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--modules str [str ...]] --run-high int --run-med int
+ --run-low int [--operation-mode str] [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-template str]
+ [--instrument-source-template str]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--mem-cells int]
+ [--bias-voltage int] [--gain-setting int]
+ [--gain-mode int] [--integration-time int]
+ [--acq-rate float] [--interlaced | --no-interlaced]
+ [--thresholds-offset-sigma float]
+ [--thresholds-offset-hard int [int ...]]
+ [--thresholds-offset-hard-hg int [int ...]]
+ [--thresholds-offset-hard-mg int [int ...]]
+ [--thresholds-offset-hard-lg int [int ...]]
+ [--thresholds-offset-hard-hg-fixed int [int ...]]
+ [--thresholds-offset-hard-mg-fixed int [int ...]]
+ [--thresholds-offset-hard-lg-fixed int [int ...]]
+ [--thresholds-noise-sigma float]
+ [--thresholds-noise-hard int [int ...]]
+ [--thresholds-noise-hard-hg int [int ...]]
+ [--thresholds-noise-hard-mg int [int ...]]
+ [--thresholds-noise-hard-lg int [int ...]]
+ [--thresholds-gain-sigma float] [--max-trains int]
+ [--min-trains int]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ AGIPD DARK
+
+ # AGIPD Characterize Dark Images #
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ The following code analyzes a set of dark images taken with the AGIPD
+ detector to deduce detector offsets noise, bad-pixel maps and thresholding.
+ All four types of constants are evaluated per-pixel and per-memory cell. Data
+ "for the detectors three gain stages needs to be present, separated into "
+ separate runs.,
+ ,
+ The evaluated calibration constants are stored locally and injected in the
+ calibration data base.,
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --modules str [str ...]
+ list of modules to evaluate, RANGE ALLOWED. Default:
+ [-1]
+ --operation-mode str Detector operation mode, optional (defaults to
+ "ADAPTIVE_GAIN"). Default: ADAPTIVE_GAIN
+ --karabo-id str karabo karabo_id. Default: HED_DET_AGIPD500K2G
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-template str
+ inset for receiver devices. Default: {}CH0
+ --instrument-source-template str
+ path in the HDF5 file to images. Default:
+ {}/DET/{}:xtdf
+ --ctrl-source-template str
+ path to control information. Default: {}/MDL/FPGA_COMP
+ --karabo-id-control str
+ karabo-id for control device '. Default:
+ HED_EXP_AGIPD500K2G
+ --use-dir-creation-date
+ use dir creation date as data production reference
+ date. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8020
+ --cal-db-timeout int timeout on caldb requests". Default: 3000000
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --mem-cells int number of memory cells used, set to 0 to automatically
+ infer. Default: 0
+ --bias-voltage int bias voltage, set to 0 to use stored value in slow
+ data.. Default: 0
+ --gain-setting int the gain setting, use -1 to use value stored in slow
+ data.. Default: -1
+ --gain-mode int gain mode, use -1 to use value stored in slow data..
+ Default: -1
+ --integration-time int
+ integration time, negative values for auto-detection..
+ Default: -1
+ --acq-rate float the detector acquisition rate, use 0 to try to auto-
+ determine. Default: 0.0
+ --interlaced assume interlaced data format, for data prior to Dec.
+ 2017. Default: False
+ --no-interlaced Opposite of --interlaced (default: False)
+ --thresholds-offset-sigma float
+ offset sigma thresholds for offset deduced bad pixels.
+ Default: 3.0
+ --thresholds-offset-hard int [int ...]
+ For setting the same threshold offset for the 3 gains.
+ Left for backcompatability. Default [0, 0] to take the
+ following parameters.. Default: [0, 0]
+ --thresholds-offset-hard-hg int [int ...]
+ High-gain thresholds in absolute ADU terms for offset
+ deduced bad pixels. Default: [3000, 7000]
+ --thresholds-offset-hard-mg int [int ...]
+ Medium-gain thresholds in absolute ADU terms for
+ offset deduced bad pixels. Default: [6000, 10000]
+ --thresholds-offset-hard-lg int [int ...]
+ Low-gain thresholds in absolute ADU terms for offset
+ deduced bad pixels. Default: [6000, 10000]
+ --thresholds-offset-hard-hg-fixed int [int ...]
+ Same as thresholds_offset_hard_hg, but for fixed gain
+ operation. Default: [3500, 6500]
+ --thresholds-offset-hard-mg-fixed int [int ...]
+ Same as thresholds_offset_hard_mg, but for fixed gain
+ operation. Default: [3500, 6500]
+ --thresholds-offset-hard-lg-fixed int [int ...]
+ Same as thresholds_offset_hard_lg, but for fixed gain
+ operation. Default: [3500, 6500]
+ --thresholds-noise-sigma float
+ noise sigma thresholds for offset deduced bad pixels.
+ Default: 5.0
+ --thresholds-noise-hard int [int ...]
+ For setting the same threshold noise for the 3 gains.
+ Left for backcompatability. Default [0, 0] to take the
+ following parameters.. Default: [0, 0]
+ --thresholds-noise-hard-hg int [int ...]
+ High-gain thresholds in absolute ADU terms for offset
+ deduced bad pixels. Default: [4, 20]
+ --thresholds-noise-hard-mg int [int ...]
+ Medium-gain thresholds in absolute ADU terms for
+ offset deduced bad pixels. Default: [4, 20]
+ --thresholds-noise-hard-lg int [int ...]
+ Low-gain thresholds in absolute ADU terms for offset
+ deduced bad pixels. Default: [4, 20]
+ --thresholds-gain-sigma float
+ Gain separation sigma threshold. Default: 5.0
+ --max-trains int Maximum number of trains to use for processing dark.
+ Set to 0 to process all available trains. 550 added
+ for ~500GB nodes to temporarely avoid memory issues..
+ Default: 550
+ --min-trains int Miniumum number of trains for processing dark. If run
+ folder has less than minimum trains, processing is
+ stopped.. Default: 1
+ --high-res-badpix-3d set this to True if you need high-resolution 3d bad
+ pixel plots. ~7mins extra time for 64 memory cells.
+ Default: False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+ --run-high int run number in which high gain data was recorded,
+ required. Default: None
+ --run-med int run number in which medium gain data was recorded,
+ required. Default: None
+ --run-low int run number in which low gain data was recorded,
+ required. Default: None
+
+
+Gain Characterization
++++++++++++++++++++++
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD FF --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--hist-file-template str] [--modules str [str ...]]
+ [--raw-folder str] [--proc-folder str] [--run int]
+ [--karabo-id str] [--karabo-da str [str ...]]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--karabo-da-control str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output]
+ [--peak-range int [int ...]]
+ [--peak-width-range int [int ...]]
+ [--peak-norm-range float [float ...]]
+ [--peak-lim int [int ...]] [--d0-lim int [int ...]]
+ [--peak-width-lim float [float ...]]
+ [--chi2-lim int [int ...]] [--intensity-lim int]
+ [--gain-lim float [float ...]]
+ [--cell-range int [int ...]]
+ [--pixel-range int [int ...]] [--max-bins int]
+ [--batch-size int [int ...]] [--fit-range int [int ...]]
+ [--n-peaks-fit int] [--fix-peaks | --no-fix-peaks]
+ [--do-minos | --no-do-minos] [--sigma-limit float]
+ [--mem-cells int] [--bias-voltage int]
+ [--acq-rate float] [--gain-setting int]
+ [--photon-energy float] [--integration-time int]
+ AGIPD FF
+
+ # Gain Characterization #
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ FF Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --hist-file-template str
+ the template to use to access histograms. Default:
+ hists_m{:02d}_sum.h5
+ --modules str [str ...]
+ modules to correct, set to -1 for all, range allowed.
+ Default: [10]
+ --raw-folder str Path to raw image data used to create histograms.
+ Default: /gpfs/exfel/exp/MID/202030/p900137/raw
+ --proc-folder str Path to corrected image data used to create
+ histograms. Default:
+ --run int of the run of image data used to create histograms.
+ Default: 449
+ --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --ctrl-source-template str
+ path to control information. Default: {}/MDL/FPGA_COMP
+ --karabo-id-control str
+ karabo-id for control device. Default:
+ MID_IRU_AGIPD1M1
+ --karabo-da-control str
+ karabo DA for control infromation. Default:
+ AGIPD1MCTRL00
+ --use-dir-creation-date
+ use the creation data of the input dir for database
+ queries. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8015#8045
+ --cal-db-timeout int in milli seconds. Default: 30000
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --peak-range int [int ...]
+ where to look for the peaks, [a0, b0, a1, b1, ...]
+ exactly 8 elements. Default: [-30, 30, 35, 70, 95,
+ 135, 145, 220]
+ --peak-width-range int [int ...]
+ fit limits on the peak widths, [a0, b0, a1, b1, ...]
+ exactly 8 elements. Default: [0, 30, 0, 35, 0, 40, 0,
+ 45]
+ --peak-norm-range float [float ...]
+ . Default: [0.0, -1, 0, -1, 0, -1, 0, -1]
+ --peak-lim int [int ...]
+ Limit of position of noise peak. Default: [-30, 30]
+ --d0-lim int [int ...]
+ hard limits for distance between noise and first peak.
+ Default: [10, 80]
+ --peak-width-lim float [float ...]
+ hard limits on the peak widths for first and second
+ peak, in units of the noise peak. 4 parameters..
+ Default: [0.9, 1.55, 0.95, 1.65]
+ --chi2-lim int [int ...]
+ Hard limit on chi2/nDOF value. Default: [0, 3.0]
+ --intensity-lim int Threshold on standard deviation of a histogram in ADU.
+ Contribute to BadPixel bit "No_Entry". Default: 15
+ --gain-lim float [float ...]
+ Threshold on gain in relative number. Contribute to
+ BadPixel bit "Gain_deviation". Default: [0.8, 1.2]
+ --cell-range int [int ...]
+ range of cell to be considered, [0,0] for all.
+ Default: [1, 3]
+ --pixel-range int [int ...]
+ range of pixels x1,y1,x2,y2 to consider [0,0,512,128]
+ for all. Default: [0, 0, 32, 32]
+ --max-bins int Maximum number of bins to consider, 0 for all bins.
+ Default: 0
+ --batch-size int [int ...]
+ batch size: [cell,x,y]. Default: [1, 8, 8]
+ --fit-range int [int ...]
+ range of a histogram considered for fitting in ADU.
+ Dynamically evaluated in case [0,0]. Default: [0, 0]
+ --n-peaks-fit int Number of gaussian peaks to fit including noise peak.
+ Default: 4
+ --fix-peaks Fix distance between photon peaks. Default: False
+ --no-fix-peaks Opposite of --fix-peaks (default: False)
+ --do-minos This is additional feature of minuit to evaluate
+ errors.. Default: False
+ --no-do-minos Opposite of --do-minos (default: False)
+ --sigma-limit float If >0, repeat fit keeping only bins within mu +-
+ sigma_limit*sigma. Default: 0.0
+ --mem-cells int number of memory cells used, negative values for auto-
+ detection.. Default: -1
+ --bias-voltage int Bias voltage.. Default: 300
+ --acq-rate float the detector acquisition rate, use 0 to try to auto-
+ determine.. Default: 0.0
+ --gain-setting int the gain setting, negative values for auto-detection..
+ Default: -1
+ --photon-energy float
+ photon energy in keV.. Default: 8.05
+ --integration-time int
+ integration time, negative values for auto-detection..
+ Default: -1
+
+ required arguments:
+ --in-folder str the folder to read histograms from, required. Default:
+ None
+ --out-folder str the folder to output to, required. Default: None
+
+
+Histogramming of AGIPD FF data
+++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 1.0
+
+Offline Calibration for the AGIPD Detector
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD FF_HISTS --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequences str [str ...]] [--modules str [str ...]]
+ --runs int [int ...] [--cells-list str [str ...]]
+ [--karabo-id str] [--karabo-da str [str ...]]
+ [--receiver-id str] [--path-template str] [--h5path str]
+ [--n-bins-adu int] [--h-range int [int ...]]
+ [--n-cells int] [--hist-std-tresh int]
+ [--hist-mean-tresh int] [--h-sums-tresh int]
+ [--n-cores-hists int] [--n-cores-files int]
+ AGIPD FF_HISTS
+
+ # Histogramming of AGIPD FF data
+ ,
+ Author: European XFEL Detector Group, Version: 1.0,
+ ,
+ Offline Calibration for the AGIPD Detector
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ FF_HISTS Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences str [str ...]
+ module to consider, set to -1 for all, range allowed.
+ Default: [-1]
+ --modules str [str ...]
+ module to consider, range allowed. Default: [5]
+ --cells-list str [str ...]
+ list of lists or any python expressions, which can be
+ converted to a list. E.g. 'range(0,15,5)'
+ 'list(range(0,5))+list(range(50,60))'. Default:
+ ['range(0,15)']
+ --karabo-id str karabo karabo_id. Default: MID_DET_AGIPD1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-id str inset for receiver devices. Default: {}CH0
+ --path-template str the template to use to access data. Default:
+ CORR-R{:04d}-AGIPD{:02d}-S{}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ INSTRUMENT/{}/DET/{}:xtdf/image/
+ --n-bins-adu int number of bins per ADU. Default: 1
+ --h-range int [int ...]
+ range of the histogram in ADU. Default: [-50, 450]
+ --n-cells int total number of memory cells (used to create summary
+ file). Default: 202
+ --hist-std-tresh int Threshold for histogram standard deviation. Default:
+ 10
+ --hist-mean-tresh int
+ Threshold for histogram mean. Default: 15
+ --h-sums-tresh int Threshold for number of entries in histograms.
+ Default: 100
+ --n-cores-hists int Number of processes (cores) to create histograms.
+ Default: 20
+ --n-cores-files int Number of processes (cores) to read files. Default: 10
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --runs int [int ...] list of run numbers, required. Default: None
+
+
+Characterize AGIPD Pulse Capacitor Data
++++++++++++++++++++++++++++++++++++++++
+
+Author: Detector Operations Group, Version 1.0
+
+The following code characterizes AGIPD gain via data take with the pulse
+capacitor source (PCS). The PCS allows scanning through the high and
+medium gains of AGIPD, by subsequently intecreasing the number of charge
+pulses from a on-ASIC capicitor, thus increasing the charge a pixel sees
+in a given integration time.
+
+Because induced charge does not originate from X-rays on the sensor, the
+gains evaluated here will later need to be rescaled with gains deduced
+from X-ray data.
+
+PCS data is organized into multiple runs, as the on-ASIC current source
+cannot supply all pixels of a given module with charge at the same time.
+Hence, only certain pixel rows will have seen charge for a given image.
+These rows then first need to be combined into single module images
+again. This script uses new style of merging, which does not support old
+data format.
+
+We then use a K-means clustering algorithm to identify components in the
+resulting per-pixel data series, matching to three general regions:
+
+- a high gain slope
+- a transition region, where gain switching occurs
+- a medium gain slope.
+
+The same regions are present in the gain-bit data and are used to deduce
+the switching threshold.
+
+The resulting slopes are then fitted with a linear function and a
+combination of a linear and exponential decay function to determine the
+relative gains of the pixels with respect to the module. Additionally,
+we deduce masks for bad pixels form the data.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate AGIPD PC --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str --runs str [str ...]
+ [--n-sequences int] [--modules str [str ...]]
+ [--karabo-da str [str ...]] [--karabo-da-control str]
+ [--karabo-id-control str] [--karabo-id str]
+ [--ctrl-source-template str]
+ [--instrument-source-template str]
+ [--receiver-template str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--delta-time int] [--cal-db-interface str]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--bias-voltage int]
+ [--mem-cells int] [--acq-rate float]
+ [--gain-setting int] [--integration-time int]
+ [--FF-gain float] [--fit-hook | --no-fit-hook]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ [--hg-range int [int ...]] [--mg-range int [int ...]]
+ [--sigma-dev-cut float]
+ AGIPD PC
+
+ # Characterize AGIPD Pulse Capacitor Data #
+ ,
+ Author: Detector Operations Group, Version 1.0,
+ ,
+ The following code characterizes AGIPD gain via data take with the pulse
+ capacitor source (PCS). The PCS allows scanning through the high and medium
+ gains of AGIPD, by subsequently intecreasing the number of charge pulses from
+ a on-ASIC capicitor, thus increasing the charge a pixel sees in a given
+ integration time.,
+ ,
+ Because induced charge does not originate from X-rays on the sensor, the
+ gains evaluated here will later need to be rescaled with gains deduced from
+ X-ray data.,
+ ,
+ PCS data is organized into multiple runs, as the on-ASIC current source
+ cannot supply all pixels of a given module with charge at the same time.
+ Hence, only certain pixel rows will have seen charge for a given image. These
+ rows then first need to be combined into single module images again. This
+ script uses new style of merging, which does not support old data format.,
+ ,
+ We then use a K-means clustering algorithm to identify components in the
+ resulting per-pixel data series, matching to three general regions:,
+ ,
+ * a high gain slope,
+ * a transition region, where gain switching occurs,
+ * a medium gain slope.,
+ ,
+ The same regions are present in the gain-bit data and are used to deduce the
+ switching threshold.
+ ,
+ The resulting slopes are then fitted with a linear function and a combination
+ of a linear and exponential decay function to determine the relative gains of
+ the pixels with respect to the module. Additionally, we deduce masks for bad
+ pixels form the data.
+
+ positional arguments:
+ AGIPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ PC Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ The ipcluster profile to use. Default: noDB
+ --n-sequences int number of sequence files, starting for 0 to evaluate.
+ Default: 5
+ --modules str [str ...]
+ modules to work on, required, range allowed. Default:
+ [-1]
+ --karabo-da str [str ...]
+ Default: ['all']
+ --karabo-da-control str
+ karabo DA for control infromation. Default:
+ AGIPD1MCTRL00
+ --karabo-id-control str
+ karabo-id for the control device e.g.
+ "MID_EXP_AGIPD1M1", or "SPB_IRU_AGIPD1M1". Default:
+ SPB_IRU_AGIPD1M1
+ --karabo-id str Default: SPB_DET_AGIPD1M-1
+ --ctrl-source-template str
+ path to control information. Default: {}/MDL/FPGA_COMP
+ --instrument-source-template str
+ path in the HDF5 file to images. Default:
+ {}/DET/{}:xtdf
+ --receiver-template str
+ inset for receiver devices. Default: {}CH0
+ --use-dir-creation-date
+ Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --delta-time int offset to the creation time (e.g. useful in case we
+ want to force the system to use diff. dark constants).
+ Default: 0
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8019
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --bias-voltage int detector bias voltage, negative values for auto-
+ detection.. Default: -1
+ --mem-cells int number of memory cells used, negative values for auto-
+ detection.. Default: -1
+ --acq-rate float the detector acquisition rate, negative values for
+ auto-detection.. Default: -1.0
+ --gain-setting int gain setting can have value 0 or 1, negative values
+ for auto-detection.. Default: -1
+ --integration-time int
+ integration time, negative values for auto-detection..
+ Default: -1
+ --FF-gain float ADU/keV, gain from FF to convert ADU to keV. Default:
+ 7.3
+ --fit-hook fit a hook function to medium gain slope --> run
+ without hook. Default: False
+ --no-fit-hook Opposite of --fit-hook (default: False)
+ --high-res-badpix-3d set this to True if you need high-resolution 3d bad
+ pixel plots. Runtime: ~ 1h. Default: False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+ --hg-range int [int ...]
+ [0,-150] range for linear fit. If upper edge is
+ negative use clustering result (bound_hg - 20).
+ Default: [30, 210]
+ --mg-range int [int ...]
+ [-350,600] range for linear fit. If lower edge is
+ negative use clustering result (bound_mg + 20).
+ Default: [-277, 600]
+ --sigma-dev-cut float
+ parameters outside the range median +-
+ sigma_dev_cut*MAD are replaced with the median.
+ Default: 5.0
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+ --runs str [str ...] runs to use, required, range allowed. Default: None
+
+
+DSSC
+----
+
+DSSC Offline Correction
++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 1.0
+
+Offline Calibration for the DSSC Detector
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate DSSC CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequences int [int ...]]
+ [--modules str [str ...]] --run int [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-id str]
+ [--path-template str] [--h5path str] [--h5path-idx str]
+ [--slow-data-pattern str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--mem-cells int] [--overwrite | --no-overwrite]
+ [--max-pulses int] [--bias-voltage int]
+ [--sequences-per-node int] [--chunk-size-idim int]
+ [--mask-noisy-asic float] [--mask-cold-asic float]
+ [--noisy-pix-threshold float] [--geo-file str]
+ [--dinstance str]
+ [--slow-data-aggregators int [int ...]]
+ [--slow-data-path str]
+ DSSC CORRECT
+
+ # DSSC Offline Correction #
+ ,
+ Author: European XFEL Detector Group, Version: 1.0,
+ ,
+ Offline Calibration for the DSSC Detector
+
+ positional arguments:
+ DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ The ipcluster profile to use. Default: noDB
+ --sequences int [int ...]
+ sequence files to evaluate.. Default: [-1]
+ --modules str [str ...]
+ modules to correct, set to -1 for all, range allowed.
+ Default: [-1]
+ --karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-id str inset for receiver devices. Default: {}CH0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{:05d}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ INSTRUMENT/{}/DET/{}:xtdf/image
+ --h5path-idx str path in the HDF5 file to images. Default:
+ /INDEX/{}/DET/{}:xtdf/image
+ --slow-data-pattern str
+ Default: RAW-R{}-DA{}-S00000.h5
+ --use-dir-creation-date
+ use the creation data of the input dir for database
+ queries. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8020#8025
+ --cal-db-timeout int in milli seconds. Default: 300000
+ --mem-cells int number of memory cells used, set to 0 to automatically
+ infer. Default: 0
+ --overwrite set to True if existing data should be overwritten.
+ Default: True
+ --no-overwrite Opposite of --overwrite (default: True)
+ --max-pulses int maximum number of pulses per train. Default: 800
+ --bias-voltage int detector bias voltage. Default: 100
+ --sequences-per-node int
+ number of sequence files per cluster node if run as
+ slurm job, set to 0 to not run SLURM parallel.
+ Default: 1
+ --chunk-size-idim int
+ chunking size of imaging dimension, adjust if user
+ software is sensitive to this.. Default: 1
+ --mask-noisy-asic float
+ set to a value other than 0 and below 1 to mask entire
+ ADC if fraction of noisy pixels is above. Default:
+ 0.25
+ --mask-cold-asic float
+ mask cold ASICS if number of pixels with negligable
+ standard deviation is larger than this fraction.
+ Default: 0.25
+ --noisy-pix-threshold float
+ threshold above which ap pixel is considered noisy..
+ Default: 1.0
+ --geo-file str detector geometry file. Default:
+ /gpfs/exfel/data/scratch/xcal/dssc_geo_june19.h5
+ --dinstance str Default: DSSC1M1
+ --slow-data-aggregators int [int ...]
+ quadrant/aggregator. Default: [1, 2, 3, 4]
+ --slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+ --run int runs to process, required. Default: None
+
+
+DSSC Characterize Dark Images
++++++++++++++++++++++++++++++
+
+Author: S. Hauf, Version: 0.1
+
+The following code analyzes a set of dark images taken with the DSSC
+detector to deduce detector offsets and noise. Data for the detector is
+presented in one run and don’t acquire multiple gain stages.
+
+The notebook explicitely does what pyDetLib provides in its offset
+calculation method for streaming data.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate DSSC DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequences int [int ...]]
+ [--modules int [int ...]] --run int [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-id str]
+ [--path-template str] [--h5path str] [--h5path-idx str]
+ [--slow-data-pattern str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--mem-cells int]
+ [--bias-voltage int] [--rawversion int]
+ [--thresholds-offset-sigma float]
+ [--thresholds-offset-hard int [int ...]]
+ [--thresholds-noise-sigma float]
+ [--thresholds-noise-hard float [float ...]]
+ [--offset-numpy-algorithm str]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ [--slow-data-aggregators int [int ...]]
+ [--slow-data-path str] [--operation-mode str]
+ DSSC DARK
+
+ # DSSC Characterize Dark Images #
+ ,
+ Author: S. Hauf, Version: 0.1,
+ ,
+ The following code analyzes a set of dark images taken with the DSSC detector
+ to deduce detector offsets and noise. Data for the detector is presented in
+ "one run and dont acquire multiple gain stages. ",
+ ,
+ The notebook explicitely does what pyDetLib provides in its offset
+ calculation method for streaming data.
+
+ positional arguments:
+ DSSC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ The ipcluster profile to use. Default: noDB
+ --sequences int [int ...]
+ sequence files to evaluate.. Default: [0]
+ --modules int [int ...]
+ modules to run for. Default: [-1]
+ --karabo-id str karabo karabo_id. Default: SQS_DET_DSSC1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-id str inset for receiver devices. Default: {}CH0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{:05d}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ /INSTRUMENT/{}/DET/{}:xtdf/image
+ --h5path-idx str path in the HDF5 file to images. Default:
+ /INDEX/{}/DET/{}:xtdf/image
+ --slow-data-pattern str
+ Default: RAW-R{}-DA{}-S00000.h5
+ --use-dir-creation-date
+ use the dir creation date for determining the creation
+ time. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8020
+ --cal-db-timeout int timeout on caldb requests". Default: 3000000
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --mem-cells int number of memory cells used, set to 0 to automatically
+ infer. Default: 0
+ --bias-voltage int detector bias voltage. Default: 100
+ --rawversion int RAW file format version. Default: 2
+ --thresholds-offset-sigma float
+ thresholds in terms of n sigma noise for offset
+ deduced bad pixels. Default: 3.0
+ --thresholds-offset-hard int [int ...]
+ thresholds in absolute ADU terms for offset deduced
+ bad pixels,. Default: [4, 125]
+ --thresholds-noise-sigma float
+ thresholds in terms of n sigma noise for offset
+ deduced bad pixels. Default: 3.0
+ --thresholds-noise-hard float [float ...]
+ thresholds in absolute ADU terms for offset deduced
+ bad pixels. Default: [0.001, 3]
+ --offset-numpy-algorithm str
+ Default: mean
+ --high-res-badpix-3d set this to True if you need high-resolution 3d bad
+ pixel plots. Runtime: ~ 1h. Default: False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+ --slow-data-aggregators int [int ...]
+ quadrant/aggregator. Default: [1, 2, 3, 4]
+ --slow-data-path str Default: SQS_NQS_DSSC/FPGA/PPT_Q
+ --operation-mode str Detector operation mode, optional. Default:
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+ --run int run number in which data was recorded, required.
+ Default: None
+
+
+EPIX100
+-------
+
+ePix100 Data Correction
++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+The following notebook provides data correction of images acquired with
+the ePix100 detector.
+
+The sequence of correction applied are: Offset –> Common Mode Noise –>
+Relative Gain –> Charge Sharing –> Absolute Gain.
+
+Offset, common mode and gain corrected data is saved to
+/data/image/pixels in the CORR files.
+
+If pattern classification is applied (charge sharing correction), this
+data will be saved to /data/image/pixels_classified, while the
+corresponding patterns will be saved to /data/image/patterns in the CORR
+files.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate EPIX100 CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequences str [str ...]] [--sequences-per-node int]
+ --run int [--karabo-id str] [--karabo-da str]
+ [--db-module str] [--receiver-template str]
+ [--path-template str] [--instrument-source-template str]
+ [--chunk-size-idim int] [--limit-images int]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--creation-time str] [--bias-voltage int]
+ [--in-vacuum | --no-in-vacuum] [--integration-time int]
+ [--fix-temperature int] [--gain-photon-energy float]
+ [--photon-energy float]
+ [--pattern-classification | --no-pattern-classification]
+ [--relative-gain | --no-relative-gain]
+ [--absolute-gain | --no-absolute-gain]
+ [--common-mode | --no-common-mode] [--cm-min-frac float]
+ [--cm-noise-sigma float]
+ [--split-evt-primary-threshold float]
+ [--split-evt-secondary-threshold float]
+ [--split-evt-mip-threshold float]
+ EPIX100 CORRECT
+
+ # ePix100 Data Correction
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ The following notebook provides data correction of images acquired with the
+ ePix100 detector.
+ ,
+ The sequence of correction applied are:,
+ Offset --> Common Mode Noise --> Relative Gain --> Charge Sharing -->
+ Absolute Gain.,
+ ,
+ Offset, common mode and gain corrected data is saved to /data/image/pixels in
+ the CORR files.,
+ ,
+ If pattern classification is applied (charge sharing correction), this data
+ will be saved to /data/image/pixels_classified, while the corresponding
+ patterns will be saved to /data/image/patterns in the CORR files.
+
+ positional arguments:
+ EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --sequences-per-node int
+ number of sequence files per cluster node if run as
+ slurm job, set to 0 to not run SLURM parallel.
+ Default: 1
+ --karabo-id str karabo karabo_id. Default: HED_IA1_EPX100-1
+ --karabo-da str data aggregators. Default: EPIX01
+ --db-module str module id in the database. Default:
+ --receiver-template str
+ detector receiver template for accessing raw data
+ files. Default: RECEIVER
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --instrument-source-template str
+ instrument detector data source in h5files. Default:
+ {}/DET/{}:daqOutput
+ --chunk-size-idim int
+ H5 chunking size of output data. Default: 1
+ --limit-images int ONLY FOR TESTING. process only first N images, 0 -
+ process all.. Default: 0
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8015#8025
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --creation-time str The timestamp to use with Calibration DBe. Required
+ Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
+ 11:02:41. Default:
+ --bias-voltage int bias voltage. Default: 200
+ --in-vacuum detector operated in vacuum. Default: False
+ --no-in-vacuum Opposite of --in-vacuum (default: False)
+ --integration-time int
+ Detector integration time, Default value -1 to use the
+ value from the slow data.. Default: -1
+ --fix-temperature int
+ fixed temperature value in Kelvin, Default value -1 to
+ use the value from files.. Default: -1
+ --gain-photon-energy float
+ Photon energy used for gain calibration. Default:
+ 8.048
+ --photon-energy float
+ Photon energy to calibrate in number of photons, 0 for
+ calibration in keV. Default: 0.0
+ --pattern-classification
+ do clustering.. Default: True
+ --no-pattern-classification
+ Opposite of --pattern-classification (default: True)
+ --relative-gain Apply relative gain correction.. Default: True
+ --no-relative-gain Opposite of --relative-gain (default: True)
+ --absolute-gain Apply absolute gain correction (implies relative
+ gain).. Default: True
+ --no-absolute-gain Opposite of --absolute-gain (default: True)
+ --common-mode Apply common mode correction.. Default: True
+ --no-common-mode Opposite of --common-mode (default: True)
+ --cm-min-frac float No CM correction is performed if after masking the
+ ratio of good pixels falls below this. Default: 0.25
+ --cm-noise-sigma float
+ CM correction noise standard deviation. Default: 5.0
+ --split-evt-primary-threshold float
+ primary threshold for split event correction. Default:
+ 7.0
+ --split-evt-secondary-threshold float
+ secondary threshold for split event correction.
+ Default: 5.0
+ --split-evt-mip-threshold float
+ minimum ionizing particle threshold. Default: 1000.0
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+ePix100 Dark Characterization
++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+The following notebook provides dark image analysis and calibration
+constants of the ePix100 detector.
+
+Dark characterization evaluates offset and noise of the detector and
+gives information about bad pixels.
+
+Noise and bad pixels maps are calculated independently for each of the 4
+ASICs of ePix100, since their noise behaviour can be significantly
+different.
+
+Common mode correction can be applied to increase sensitivity to noise
+related bad pixels. Common mode correction is achieved by subtracting
+the median of all pixels that are read out at the same time along a
+row/column. This is done in an iterative process, by which a new bad
+pixels map is calculated and used to mask data as the common mode values
+across the rows/columns is updated.
+
+Resulting maps are saved as .h5 files for a later use and injected to
+calibration DB.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate EPIX100 DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequence int] --run int [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-template str]
+ [--path-template str] [--instrument-source-template str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output]
+ [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
+ [--fix-integration-time int] [--fix-temperature int]
+ [--temp-limits int] [--badpixel-threshold-sigma float]
+ [--CM-N-iterations int] [--min-trains int]
+ [--max-trains int] [--operation-mode str]
+ [--db-module str]
+ EPIX100 DARK
+
+ # ePix100 Dark Characterization
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ The following notebook provides dark image analysis and calibration constants
+ of the ePix100 detector.,
+ ,
+ Dark characterization evaluates offset and noise of the detector and gives
+ information about bad pixels.
+ ,
+ Noise and bad pixels maps are calculated independently for each of the 4
+ ASICs of ePix100, since their noise behaviour can be significantly different.,
+ ,
+ Common mode correction can be applied to increase sensitivity to noise
+ related bad pixels. Common mode correction is achieved by subtracting the
+ median of all pixels that are read out at the same time along a row/column.
+ This is done in an iterative process, by which a new bad pixels map is
+ calculated and used to mask data as the common mode values across the
+ rows/columns is updated.,
+ ,
+ Resulting maps are saved as .h5 files for a later use and injected to
+ calibration DB.
+
+ positional arguments:
+ EPIX100 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequence int sequence file to use. Default: 0
+ --karabo-id str karabo karabo_id. Default: HED_IA1_EPX100-1
+ --karabo-da str [str ...]
+ data aggregators. Default: ['EPIX01']
+ --receiver-template str
+ detector receiver template for accessing raw data
+ files. Default: RECEIVER
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --instrument-source-template str
+ instrument detector data source in h5files. Default:
+ {}/DET/{}:daqOutput
+ --use-dir-creation-date
+ Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8020
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --db-output Output constants to the calibration database. Default:
+ False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output Output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --bias-voltage int Bias voltage. Default: 200
+ --in-vacuum Detector operated in vacuum. Default: False
+ --no-in-vacuum Opposite of --in-vacuum (default: False)
+ --fix-integration-time int
+ Integration time. Set to -1 to read from .h5 file.
+ Default: -1
+ --fix-temperature int
+ Fixed temperature in Kelvin. Set to -1 to read from
+ .h5 file. Default: -1
+ --temp-limits int Limit for parameter Operational temperature. Default:
+ 5
+ --badpixel-threshold-sigma float
+ Bad pixels defined by values outside n times this std
+ from median. Default: 5. Default: 5.0
+ --CM-N-iterations int
+ Number of iterations for common mode correction. Set
+ to 0 to skip it. Default: 2
+ --min-trains int Minimum number of trains that should be available to
+ process dark constants. Default 1.. Default: 1
+ --max-trains int Maximum number of trains to use for processing dark
+ constants. Set to 0 to use all available trains..
+ Default: 1000
+ --operation-mode str Detector operation mode, optional. Default:
+ --db-module str ID of module in calibration database, this parameter
+ is ignore in the notebook. TODO: remove from
+ calibration_configurations.. Default:
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+EPIX10K
+-------
+
+ePIX10K Data Correction
++++++++++++++++++++++++
+
+Authors: M. Karnevskiy, S. Hauf, Version 1.0
+
+The following notebook provides Offset correction of images acquired
+with the ePix10K detector.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate EPIX10K CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequences str [str ...]] --run
+ int [--karabo-id str] [--karabo-da str]
+ [--receiver-id str] [--path-template str] [--h5path str]
+ [--h5path-t str] [--h5path-cntrl str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--cpuCores int] [--chunk-size-idim int]
+ [--overwrite | --no-overwrite] [--limit-images int]
+ [--db-module str] [--sequences-per-node int]
+ [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
+ [--fix-temperature float]
+ [--split-evt-primary-threshold float]
+ [--split-evt-secondary-threshold float]
+ [--split-evt-mip-threshold float]
+ EPIX10K CORRECT
+
+ # ePIX10K Data Correction #
+ ,
+ Authors: M. Karnevskiy, S. Hauf, Version 1.0,
+ ,
+ The following notebook provides Offset correction of images acquired with the
+ ePix10K detector.
+
+ positional arguments:
+ EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ ipcluster profile to use. Default: noDB
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1
+ --karabo-da str data aggregators. Default: EPIX03
+ --receiver-id str inset for receiver devices. Default: RECEIVER
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ /INSTRUMENT/{}/DET/{}:daqOutput/data/image
+ --h5path-t str path to find temperature at. Default:
+ /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
+ --h5path-cntrl str path to control data. Default: /CONTROL/{}/DET
+ --use-dir-creation-date
+ date constants injected before directory creation
+ time. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8015#8025
+ --cal-db-timeout int timeout on caldb requests. Default: 30000000
+ --cpuCores int Specifies the number of running cpu cores. Default: 4
+ --chunk-size-idim int
+ H5 chunking size of output data. Default: 1
+ --overwrite overwrite output folder. Default: True
+ --no-overwrite Opposite of --overwrite (default: True)
+ --limit-images int process only first N images, 0 - process all. Default:
+ 0
+ --db-module str module id in the database. Default: ePix10K_M43
+ --sequences-per-node int
+ number of sequence files per cluster node if run as
+ slurm job, set to 0 to not run SLURM parallel.
+ Default: 1
+ --bias-voltage int bias voltage. Default: 200
+ --in-vacuum detector operated in vacuum. Default: False
+ --no-in-vacuum Opposite of --in-vacuum (default: False)
+ --fix-temperature float
+ fix temperature to this value. Default: 290.0
+ --split-evt-primary-threshold float
+ primary threshold for split event correction. Default:
+ 7.0
+ --split-evt-secondary-threshold float
+ secondary threshold for split event correction.
+ Default: 5.0
+ --split-evt-mip-threshold float
+ minimum ionizing particle threshold. Default: 1000.0
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+ePix10K Dark Characterization
++++++++++++++++++++++++++++++
+
+Author: M. Karnevskiy, Version 1.0
+
+The following notebook provides dark image analysis of the ePix10K
+detector.
+
+Dark characterization evaluates offset and noise of the detector and
+gives information about bad pixels. Resulting maps are saved as .h5
+files for a latter use and injected to the calibration DB.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate EPIX10K DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequence int] --run int
+ [--karabo-id str] [--karabo-da str [str ...]]
+ [--receiver-id str] [--path-template str] [--h5path str]
+ [--h5path-t str] [--h5path-cntrl str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output] [--temp-limits int]
+ [--number-dark-frames int] [--db-module str]
+ [--bias-voltage int] [--in-vacuum | --no-in-vacuum]
+ [--fix-temperature float] [--operation-mode str]
+ EPIX10K DARK
+
+ # ePix10K Dark Characterization #
+ ,
+ Author: M. Karnevskiy, Version 1.0,
+ ,
+ The following notebook provides dark image analysis of the ePix10K detector.,
+ ,
+ Dark characterization evaluates offset and noise of the detector and gives
+ information about bad pixels. Resulting maps are saved as .h5 files for a
+ latter use and injected to the calibration DB.
+
+ positional arguments:
+ EPIX10K The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ ipcluster profile to use. Default: noDB
+ --sequence int sequence file to use. Default: 0
+ --karabo-id str karabo karabo_id. Default: HED_IA1_EPIX10K-1
+ --karabo-da str [str ...]
+ data aggregators. Default: ['EPIX03']
+ --receiver-id str inset for receiver devices. Default: RECEIVER
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ /INSTRUMENT/{}/DET/{}:daqOutput/data/image/pixels
+ --h5path-t str path to find temperature at. Default:
+ /INSTRUMENT/{}/DET/{}:daqOutput/data/backTemp
+ --h5path-cntrl str path to control data. Default: /CONTROL/{}/DET
+ --use-dir-creation-date
+ Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8020
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --db-output Output constants to the calibration database. Default:
+ False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --temp-limits int limit for parameter Operational temperature. Default:
+ 5
+ --number-dark-frames int
+ number of images to be used, if set to 0 all available
+ images are used. Default: 0
+ --db-module str detector karabo_id. Default: ePix10K_M43
+ --bias-voltage int bias voltage. Default: 200
+ --in-vacuum detector operated in vacuum. Default: False
+ --no-in-vacuum Opposite of --in-vacuum (default: False)
+ --fix-temperature float
+ fix temperature to this value. Default: 290.0
+ --operation-mode str Detector operation mode, optional. Default:
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+FASTCCD
+-------
+
+FastCCD Data Correction
++++++++++++++++++++++++
+
+Authors: I. KlaÄková, S. Hauf, Version 1.0
+
+The following notebook provides correction of images acquired with the
+FastCCD.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate FASTCCD CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequences str [str ...]]
+ [--run int] [--karabo-da str] [--karabo-id str]
+ [--receiver-id str] [--path-template str] [--h5path str]
+ [--h5path-t str] [--h5path-cntrl str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--cpuCores int] [--operation-mode str]
+ [--split-evt-primary-threshold float]
+ [--split-evt-secondary-threshold float]
+ [--split-evt-mip-threshold float]
+ [--chunk-size-idim int] [--overwrite | --no-overwrite]
+ [--sequences-per-node int] [--limit-images int]
+ [--time-offset-days int]
+ [--photon-energy-gain-map float]
+ [--fix-temperature float]
+ [--flipped-between str [str ...]] [--temp-limits int]
+ [--commonModeAxis int]
+ [--only-offset | --no-only-offset]
+ [--cti-corr | --no-cti-corr]
+ [--relgain-corr | --no-relgain-corr]
+ [--common-mode-corr | --no-common-mode-corr]
+ [--correct-offset-drift | --no-correct-offset-drift]
+ [--do-pattern-classification | --no-do-pattern-classification]
+ FASTCCD CORRECT
+
+ # FastCCD Data Correction
+ ,
+ Authors: I. KlaÄková, S. Hauf, Version 1.0,
+ ,
+ The following notebook provides correction of images acquired with the
+ FastCCD.
+
+ positional arguments:
+ FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ ipcluster profile to use. Default: noDB
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --run int run number. Default: 277
+ --karabo-da str data aggregators. Default: DA05
+ --karabo-id str karabo prefix of PNCCD devices. Default:
+ SCS_CDIDET_FCCD2M
+ --receiver-id str inset for receiver devices. Default: FCCD
+ --path-template str path template in hdf5 file. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --h5path str path in HDF5 file. Default:
+ /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image
+ --h5path-t str temperature path in HDF5 file. Default:
+ /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
+ --h5path-cntrl str path to control data. Default: /RUN/{}/DET/FCCD
+ --use-dir-creation-date
+ use dir creation data for calDB queries. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8015#8025
+ --cal-db-timeout int timeout on caldb requests. Default: 300000000
+ --cpuCores int Specifies the number of running cpu cores. Default: 16
+ --operation-mode str FS stands for frame-store and FF for full-frame
+ opeartion. Default: FF
+ --split-evt-primary-threshold float
+ primary threshold for split event classification in
+ terms of n sigma noise. Default: 7.0
+ --split-evt-secondary-threshold float
+ secondary threshold for split event classification in
+ terms of n sigma noise. Default: 4.0
+ --split-evt-mip-threshold float
+ MIP threshold for event classification. Default:
+ 1000.0
+ --chunk-size-idim int
+ H5 chunking size of output data. Default: 1
+ --overwrite overwrite existing files. Default: True
+ --no-overwrite Opposite of --overwrite (default: True)
+ --sequences-per-node int
+ sequences to correct per node. Default: 1
+ --limit-images int limit images per file. Default: 0
+ --time-offset-days int
+ offset in days for calibration parameters. Default: 0
+ --photon-energy-gain-map float
+ energy in keV. Default: 5.9
+ --fix-temperature float
+ fix temperature to this value, set to 0 to use slow
+ control value. Default: 0.0
+ --flipped-between str [str ...]
+ detector was flipped during this timespan. Default:
+ ['2019-02-01', '2019-04-02']
+ --temp-limits int limits within which temperature is considered the
+ same. Default: 5
+ --commonModeAxis int Axis along which common mode will be calculated (0:
+ along rows, 1: along columns). Default: 1
+ --only-offset Only apply offset correction. Default: False
+ --no-only-offset Opposite of --only-offset (default: False)
+ --cti-corr Apply CTI correction. Default: False
+ --no-cti-corr Opposite of --cti-corr (default: False)
+ --relgain-corr Apply relative gain correction. Default: True
+ --no-relgain-corr Opposite of --relgain-corr (default: True)
+ --common-mode-corr Apply commonMode correction. Default: False
+ --no-common-mode-corr
+ Opposite of --common-mode-corr (default: False)
+ --correct-offset-drift
+ correct for offset drifts. Default: False
+ --no-correct-offset-drift
+ Opposite of --correct-offset-drift (default: False)
+ --do-pattern-classification
+ classify split events. Default: True
+ --no-do-pattern-classification
+ Opposite of --do-pattern-classification (default:
+ True)
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+
+
+FastCCD Dark Characterization
++++++++++++++++++++++++++++++
+
+Author: I. KlaÄková, S. Hauf, K. Setoodehnia and M. Cascella
+
+The following notebook provides dark image analysis of the FastCCD
+detector.
+
+Dark characterization evaluates offset and noise of the FastCCD
+detector, corrects the noise for Common Mode (CM), and defines bad
+pixels relative to offset and CM corrected noise. Bad pixels are then
+excluded and CM corrected noise is recalculated excluding the bad
+pixels. Resulting offset and CM corrected noise maps, as well as the bad
+pixel map are sent to the calibration database.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate FASTCCD DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--sequence int] --run int
+ [--karabo-da str [str ...]] [--karabo-id str]
+ [--receiver-id str] [--path-template str] [--h5path str]
+ [--h5path-t str] [--h5path-cntrl str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output]
+ [--number-dark-frames int] [--operation-mode str]
+ [--temp-limits int] [--bad-pixel-offset-sigma float]
+ [--bad-pixel-noise-sigma float] [--sigmaNoise float]
+ [--fix-temperature float] [--chunkSize int]
+ [--cpuCores int] [--commonModeAxis int]
+ [--run-parallel | --no-run-parallel]
+ [--ADU-to-electron-upper-hg float]
+ [--ADU-to-electron-lower-hg float]
+ [--ADU-to-electron-upper-mg float]
+ [--ADU-to-electron-lower-mg float]
+ [--ADU-to-electron-upper-lg float]
+ [--ADU-to-electron-lower-lg float]
+ FASTCCD DARK
+
+ # FastCCD Dark Characterization
+ ,
+ Author: I. KlaÄková, S. Hauf, K. Setoodehnia and M. Cascella,
+ ,
+ The following notebook provides dark image analysis of the FastCCD detector.,
+ ,
+ Dark characterization evaluates offset and noise of the FastCCD detector,
+ corrects the noise for Common Mode (CM), and defines bad pixels relative to
+ offset and CM corrected noise. Bad pixels are then excluded and CM corrected
+ noise is recalculated excluding the bad pixels. Resulting offset and CM
+ corrected noise maps, as well as the bad pixel map are sent to the
+ calibration database.
+
+ positional arguments:
+ FASTCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ ipcluster profile to use. Default: noDB
+ --sequence int sequence file to use. Default: 0
+ --karabo-da str [str ...]
+ data aggregators. Default: ['DA05']
+ --karabo-id str karabo prefix of PNCCD devices. Default:
+ SCS_CDIDET_FCCD2M
+ --receiver-id str inset for receiver devices. Default: FCCD
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --h5path str path to the data in the HDF5 file. Default:
+ /INSTRUMENT/{}/DAQ/{}:daqOutput/data/image/pixels
+ --h5path-t str path to find temperature. Default:
+ /CONTROL/{}/CTRL/LSLAN/inputA/crdg/value
+ --h5path-cntrl str path to find control data. Default: /RUN/{}/DET/FCCD
+ --use-dir-creation-date
+ To be used to retrieve calibration constants later on
+ (for database time derivation). Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the calibration database interface to use. Default:
+ tcp://max-exfl016:8020
+ --cal-db-timeout int timeout on calibration database requests. Default:
+ 300000
+ --db-output Output constants to the calibration database. Default:
+ False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --number-dark-frames int
+ number of images to be used, if set to 0 all available
+ images are used. Default: 0
+ --operation-mode str FS stands for frame-store and FF for full-frame
+ operation.. Default: FF
+ --temp-limits int to find calibration constants later on, the sensor
+ temperature is allowed to vary by 5 units. Default: 5
+ --bad-pixel-offset-sigma float
+ Any pixel whose offset is beyond 5 standard
+ deviations, is a bad pixel. Default: 5.0
+ --bad-pixel-noise-sigma float
+ Any pixel whose noise is beyond 5 standard deviations,
+ is a bad pixel. Default: 5.0
+ --sigmaNoise float Any pixel whose signal exceeds 'sigmaNoise'*noiseCM
+ (common mode corrected noise) will be masked. Default:
+ 5.0
+ --fix-temperature float
+ Fixed operation temperature in Kelvins. If set to 0,
+ mean value of the data file's temperature is. Default:
+ 0.0
+ --chunkSize int Number of images to read per chunk. Default: 100
+ --cpuCores int Specifies the number of running cpu cores. Default: 40
+ --commonModeAxis int Axis along which common mode will be calculated (0:
+ along rows, 1: along columns). Default: 1
+ --run-parallel For parallel computation. Default: True
+ --no-run-parallel Opposite of --run-parallel (default: True)
+ --ADU-to-electron-upper-hg float
+ for upper hemisphere and high gain. Default: 6.3
+ --ADU-to-electron-lower-hg float
+ for lower hemisphere and high gain. Default: 6.4
+ --ADU-to-electron-upper-mg float
+ for upper hemisphere and medium gain. Default: 23.4
+ --ADU-to-electron-lower-mg float
+ for lower hemisphere and medium gain. Default: 23.4
+ --ADU-to-electron-upper-lg float
+ for upper hemisphere and low gain. Default: 49.3
+ --ADU-to-electron-lower-lg float
+ for lower hemisphere and low gain. Default: 47.3
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+GENERIC
+-------
+
+Constants from DB to HDF5
++++++++++++++++++++++++++
+
+Version 0.1, Author: S. Hauf
+
+Currently available instances are LPD1M1 and AGIPD1M1
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate GENERIC DB_TO_H5 --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --detector-instance str --out-file
+ str [--valid-at str] [--cal-db-interface str]
+ [--modules str [str ...]]
+ GENERIC DB_TO_H5
+
+ # Constants from DB to HDF5 #
+ ,
+ Version 0.1, Author: S. Hauf,
+ ,
+ Currently available instances are LPD1M1 and AGIPD1M1
+
+ positional arguments:
+ GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DB_TO_H5 Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --valid-at str ISO formatted date for which constants shoudl be
+ valid. Leave empty to get most current ones. Default:
+ --cal-db-interface str
+ Default: tcp://max-exfl015:5005
+ --modules str [str ...]
+ modules to get data from, in terms of numerical
+ quadrant indices, range allowed. Default: [-1]
+
+ required arguments:
+ --detector-instance str
+ the detector instance to get constants for e.g.
+ LPD1M1, required. Default: None
+ --out-file str HDF5 file to output constants into, required. Default:
+ None
+
+
+Statistical analysis of calibration factors
++++++++++++++++++++++++++++++++++++++++++++
+
+Author: Mikhail Karnevskiy, Version 0.2
+
+Plot calibration constants retrieved from the cal. DB.
+
+To be visualized, calibration constants are averaged per group of
+pixels. Plots shows calibration constant over time for each constant.
+
+Values shown in plots are saved in h5 files.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate GENERIC STATS_FROM_DB --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--start-date str] [--end-date str]
+ [--dclass str] [--modules str [str ...]]
+ [--submodules str [str ...]] [--constants str [str ...]]
+ [--nconstants int] [--max-time int] [--nMemToShow int]
+ [--gain-setting int [int ...]]
+ [--bias-voltage int [int ...]]
+ [--temperature int [int ...]]
+ [--integration-time float [float ...]]
+ [--pixels-x int [int ...]] [--pixels-y int [int ...]]
+ [--in-vacuum int [int ...]]
+ [--memory-cells int [int ...]]
+ [--acquisition-rate float [float ...]]
+ [--parameter-names str [str ...]]
+ [--separate-plot str [str ...]]
+ [--x-labels str [str ...]] [--photon-energy float]
+ [--out-folder str] [--use-existing str]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--plot-range int] [--spShape int [int ...]]
+ [--sp-name str] [--gain-titles str [str ...]]
+ GENERIC STATS_FROM_DB
+
+ # Statistical analysis of calibration factors#
+ ,
+ Author: Mikhail Karnevskiy, Version 0.2,
+ ,
+ Plot calibration constants retrieved from the cal. DB.,
+ ,
+ To be visualized, calibration constants are averaged per group of pixels.
+ Plots shows calibration constant over time for each constant.,
+ ,
+ Values shown in plots are saved in h5 files.
+
+ positional arguments:
+ GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ STATS_FROM_DB Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --start-date str date to start investigation interval from. Default:
+ 2019-01-01
+ --end-date str date to end investigation interval at, can be "now".
+ Default: NOW
+ --dclass str Detector class. Default: jungfrau
+ --modules str [str ...]
+ detector entry in the DB to investigate. Default:
+ ['Jungfrau_M035']
+ --submodules str [str ...]
+ module index of a modular detector (1 for Q1M1 of
+ AGIPD), range allowed. Default: [2]
+ --constants str [str ...]
+ constants to plot. Default: ['RelativeGain']
+ --nconstants int Number of time stamps to plot. If not 0, overcome
+ start_date.. Default: 20
+ --max-time int max time margin in minutes to match bad pixels.
+ Default: 15
+ --nMemToShow int Number of memory cells to be shown in plots. Default:
+ 16
+ --gain-setting int [int ...]
+ gain stages. Default: [0, 1, 2]
+ --bias-voltage int [int ...]
+ Bias voltage. Default: [90, 180]
+ --temperature int [int ...]
+ Operation temperature. Default: [291]
+ --integration-time float [float ...]
+ Integration time. Default: [4.96, 10, 50, 250]
+ --pixels-x int [int ...]
+ number of pixels along X axis. Default: [1024]
+ --pixels-y int [int ...]
+ number of pixels along Y axis. Default: [512]
+ --in-vacuum int [int ...]
+ 0 if detector is operated in room pressure. Default:
+ [0]
+ --memory-cells int [int ...]
+ number of memory cells. Default: [1, 16]
+ --acquisition-rate float [float ...]
+ aquisition rate. Default: [1.1]
+ --parameter-names str [str ...]
+ names of parameters. Default: ['bias_voltage',
+ 'integration_time', 'pixels_x', 'pixels_y',
+ 'temperature', 'memory_cells']
+ --separate-plot str [str ...]
+ Plot on separate plots. Default: ['gain_setting',
+ 'memory_cells', 'integration_time']
+ --x-labels str [str ...]
+ parameters to be shown on X axis: Acquisition rate,
+ Memory cells, Sensor Temperature, Integration Time.
+ Default: ['Sensor Temperature', 'Integration Time']
+ --photon-energy float
+ Photon energy of the beam. Default: 9.2
+ --out-folder str output folder. Default:
+ /gpfs/exfel/data/scratch/karnem/test_bla4/
+ --use-existing str If not empty, constants stored in given folder will be
+ used. Default:
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8016
+ --cal-db-timeout int timeout on caldb requests",. Default: 180000
+ --plot-range int range for plotting in units of median absolute
+ deviations. Default: 3
+ --spShape int [int ...]
+ Shape of superpixel. Default: [256, 64]
+ --sp-name str name of superpixel. Default: ASIC IDs
+ --gain-titles str [str ...]
+ Title inset related to gain. Default: ['High gain',
+ 'Medium gain', 'Low gain']
+
+
+Statistical analysis of calibration factors
++++++++++++++++++++++++++++++++++++++++++++
+
+Author: Mikhail Karnevskiy, Version 0.2
+
+Plot calibration constants retrieved from the cal. DB.
+
+To be visualized, calibration constants are averaged per group of
+pixels. Plots shows calibration constant over time for each constant.
+
+Values shown in plots are saved in h5 files.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate GENERIC STATS_FROM_DB2 --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--start-date str] [--end-date str]
+ [--dclass str] [--modules str [str ...]]
+ [--submodules str [str ...]] [--constants str [str ...]]
+ [--nconstants int] [--max-time int] [--nMemToShow int]
+ [--gain-setting int [int ...]]
+ [--bias-voltage int [int ...]]
+ [--temperature int [int ...]]
+ [--integration-time float [float ...]]
+ [--pixels-x int [int ...]] [--pixels-y int [int ...]]
+ [--in-vacuum int [int ...]]
+ [--memory-cells int [int ...]]
+ [--acquisition-rate float [float ...]]
+ [--parameter-names str [str ...]]
+ [--separate-plot str [str ...]]
+ [--x-labels str [str ...]] [--photon-energy float]
+ [--out-folder str] [--use-existing str]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--plot-range int] [--spShape int [int ...]]
+ [--sp-name str] [--gain-titles str [str ...]]
+ GENERIC STATS_FROM_DB2
+
+ # Statistical analysis of calibration factors#
+ ,
+ Author: Mikhail Karnevskiy, Version 0.2,
+ ,
+ Plot calibration constants retrieved from the cal. DB.,
+ ,
+ To be visualized, calibration constants are averaged per group of pixels.
+ Plots shows calibration constant over time for each constant.,
+ ,
+ Values shown in plots are saved in h5 files.
+
+ positional arguments:
+ GENERIC The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ STATS_FROM_DB2 Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --start-date str date to start investigation interval from. Default:
+ 2019-01-01
+ --end-date str date to end investigation interval at, can be "now".
+ Default: NOW
+ --dclass str Detector class. Default: jungfrau
+ --modules str [str ...]
+ detector entry in the DB to investigate. Default:
+ ['Jungfrau_M035']
+ --submodules str [str ...]
+ module index of a modular detector (1 for Q1M1 of
+ AGIPD), range allowed. Default: [2]
+ --constants str [str ...]
+ constants to plot. Default: ['RelativeGain']
+ --nconstants int Number of time stamps to plot. If not 0, overcome
+ start_date.. Default: 20
+ --max-time int max time margin in minutes to match bad pixels.
+ Default: 15
+ --nMemToShow int Number of memory cells to be shown in plots. Default:
+ 16
+ --gain-setting int [int ...]
+ gain stages. Default: [0, 1, 2]
+ --bias-voltage int [int ...]
+ Bias voltage. Default: [90, 180]
+ --temperature int [int ...]
+ Operation temperature. Default: [291]
+ --integration-time float [float ...]
+ Integration time. Default: [4.96, 10, 50, 250]
+ --pixels-x int [int ...]
+ number of pixels along X axis. Default: [1024]
+ --pixels-y int [int ...]
+ number of pixels along Y axis. Default: [512]
+ --in-vacuum int [int ...]
+ 0 if detector is operated in room pressure. Default:
+ [0]
+ --memory-cells int [int ...]
+ number of memory cells. Default: [1, 16]
+ --acquisition-rate float [float ...]
+ aquisition rate. Default: [1.1]
+ --parameter-names str [str ...]
+ names of parameters. Default: ['bias_voltage',
+ 'integration_time', 'pixels_x', 'pixels_y',
+ 'temperature', 'memory_cells']
+ --separate-plot str [str ...]
+ Plot on separate plots. Default: ['gain_setting',
+ 'memory_cells', 'integration_time']
+ --x-labels str [str ...]
+ parameters to be shown on X axis: Acquisition rate,
+ Memory cells, Sensor Temperature, Integration Time.
+ Default: ['Sensor Temperature', 'Integration Time']
+ --photon-energy float
+ Photon energy of the beam. Default: 9.2
+ --out-folder str output folder. Default:
+ /gpfs/exfel/data/scratch/karnem/test_bla4/
+ --use-existing str If not empty, constants stored in given folder will be
+ used. Default:
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8016
+ --cal-db-timeout int timeout on caldb requests",. Default: 180000
+ --plot-range int range for plotting in units of median absolute
+ deviations. Default: 3
+ --spShape int [int ...]
+ Shape of superpixel. Default: [256, 64]
+ --sp-name str name of superpixel. Default: ASIC IDs
+ --gain-titles str [str ...]
+ Title inset related to gain. Default: ['High gain',
+ 'Medium gain', 'Low gain']
+
+
+GOTTHARD2
+---------
+
+Gotthard2 Offline Correction
+++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 1.0
+
+Offline Calibration for the Gothard2 Detector
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate GOTTHARD2 CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run int [--sequences str [str ...]]
+ [--sequences-per-node int] [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-template str]
+ [--control-template str]
+ [--instrument-source-template str]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--overwrite-creation-time str] [--constants-file str]
+ [--offset-correction | --no-offset-correction]
+ [--gain-correction | --no-gain-correction]
+ [--bias-voltage int] [--exposure-time float]
+ [--exposure-period float] [--acquisition-rate float]
+ [--single-photon int] [--skip-plots | --no-skip-plots]
+ [--pulse-idx-preview int]
+ GOTTHARD2 CORRECT
+
+ # Gotthard2 Offline Correction #
+ ,
+ Author: European XFEL Detector Group, Version: 1.0,
+ ,
+ Offline Calibration for the Gothard2 Detector
+
+ positional arguments:
+ GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences str [str ...]
+ sequences to correct, set to [-1] for all, range
+ allowed. Default: [-1]
+ --sequences-per-node int
+ number of sequence files per node if notebook executed
+ through xfel-calibrate, set to 0 to not run SLURM
+ parallel. Default: 1
+ --karabo-id str karabo prefix of Gotthard-II devices. Default:
+ FXE_XAD_G2XES
+ --karabo-da str [str ...]
+ data aggregators. Default: ['GH201']
+ --receiver-template str
+ receiver template used to read INSTRUMENT keys..
+ Default: RECEIVER
+ --control-template str
+ control template used to read CONTROL keys.. Default:
+ CONTROL
+ --instrument-source-template str
+ template for source name (filled with karabo_id &
+ receiver_id). e.g.
+ 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
+ {}/DET/{}:daqOutput
+ --ctrl-source-template str
+ template for control source name (filled with
+ karabo_id_control). Default: {}/DET/{}
+ --karabo-id-control str
+ Control karabo ID. Set to empty string to use the
+ karabo-id. Default:
+ --use-dir-creation-date
+ use the creation data of the input dir for database
+ queries.. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use.. Default: tcp://max-
+ exfl016:8016#8025
+ --cal-db-timeout int timeout on caldb requests.. Default: 180000
+ --overwrite-creation-time str
+ To overwrite the measured creation_time. Required
+ Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
+ 13:00:00.00". Default:
+ --constants-file str Use constants in given constant file path. /gpfs/exfel
+ /data/scratch/ahmedk/dont_remove/gotthard2/constants/c
+ alibration_constants_GH2.h5. Default:
+ --offset-correction apply offset correction. This can be disabled to only
+ apply LUT or apply LUT and gain correction for non-
+ linear differential results.. Default: True
+ --no-offset-correction
+ Opposite of --offset-correction (default: True)
+ --gain-correction apply gain correction.. Default: True
+ --no-gain-correction Opposite of --gain-correction (default: True)
+ --bias-voltage int Detector bias voltage, set to -1 to use value in raw
+ file.. Default: -1
+ --exposure-time float
+ Detector exposure time, set to -1 to use value in raw
+ file.. Default: -1.0
+ --exposure-period float
+ Detector exposure period, set to -1 to use value in
+ raw file.. Default: -1.0
+ --acquisition-rate float
+ Detector acquisition rate (1.1/4.5), set to -1 to use
+ value in raw file.. Default: -1.0
+ --single-photon int Detector single photon mode (High/Low CDS), set to -1
+ to use value in raw file.. Default: -1
+ --skip-plots exit after writing corrected files. Default: False
+ --no-skip-plots Opposite of --skip-plots (default: False)
+ --pulse-idx-preview int
+ pulse index to preview. The following even/odd pulse
+ index is used for preview. TODO: update to pulseId
+ preview.. Default: 3
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --run int run to process, required. Default: None
+
+
+Gotthard2 Dark Image Characterization
++++++++++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 1.0
+
+The following is a processing for offset, noise, and Badpixels maps
+using dark images taken with Gotthard2 detector. All constants are
+evaluated per strip, per pulse, and per memory cell. The maps are
+calculated for each gain stage that is acquired in 3 separate runs.
+
+The three maps (calibration constants) can be injected to the database
+and stored locally.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate GOTTHARD2 DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run-high int --run-med int --run-low int
+ [--karabo-id str] [--karabo-da str [str ...]]
+ [--receiver-template str] [--control-template str]
+ [--instrument-source-template str]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--overwrite-creation-time str]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output]
+ [--bias-voltage int] [--exposure-time float]
+ [--exposure-period float] [--acquisition-rate float]
+ [--single-photon int] [--min-trains int]
+ [--max-trains int] [--badpixel-threshold-sigma float]
+ [--operation-mode str]
+ GOTTHARD2 DARK
+
+ # Gotthard2 Dark Image Characterization #
+ ,
+ Author: European XFEL Detector Group, Version: 1.0,
+ ,
+ The following is a processing for offset, noise, and Badpixels maps using
+ dark images taken with Gotthard2 detector.,
+ All constants are evaluated per strip, per pulse, and per memory cell. The
+ maps are calculated for each gain stage that is acquired in 3 separate runs.,
+ ,
+ The three maps (calibration constants) can be injected to the database and
+ stored locally.
+
+ positional arguments:
+ GOTTHARD2 The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --karabo-id str karabo prefix of Gotthard-II devices. Default:
+ FXE_XAD_G2XES
+ --karabo-da str [str ...]
+ data aggregators. Default: ['GH201']
+ --receiver-template str
+ receiver template used to read INSTRUMENT keys..
+ Default: RECEIVER
+ --control-template str
+ control template used to read CONTROL keys.. Default:
+ CONTROL
+ --instrument-source-template str
+ template for source name (filled with karabo_id &
+ receiver_id). e.g.
+ 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput' noqa. Default:
+ {}/DET/{}:daqOutput
+ --ctrl-source-template str
+ template for control source name (filled with
+ karabo_id_control). Default: {}/DET/{}
+ --karabo-id-control str
+ Control karabo ID. Set to empty string to use the
+ karabo-id. Default:
+ --use-dir-creation-date
+ Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8020
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --overwrite-creation-time str
+ To overwrite the measured creation_time. Required
+ Format: YYYY-MM-DD HR:MN:SC.00 e.g. "2022-06-28
+ 13:00:00.00". Default:
+ --db-output Output constants to the calibration database. Default:
+ False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output Output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --bias-voltage int Detector bias voltage, set to -1 to use value in raw
+ file.. Default: -1
+ --exposure-time float
+ Detector exposure time, set to -1 to use value in raw
+ file.. Default: -1.0
+ --exposure-period float
+ Detector exposure period, set to -1 to use value in
+ raw file.. Default: -1.0
+ --acquisition-rate float
+ Detector acquisition rate (1.1/4.5), set to -1 to use
+ value in raw file.. Default: -1.0
+ --single-photon int Detector single photon mode (High/Low CDS), set to -1
+ to use value in raw file.. Default: -1
+ --min-trains int Minimum number of trains that should be available to
+ process dark constants. Default 1.. Default: 1
+ --max-trains int Maximum number of trains to use for processing dark
+ constants. Set to 0 to use all available trains..
+ Default: 1000
+ --badpixel-threshold-sigma float
+ bad pixels defined by values outside n times this std
+ from median. Default: 5.0
+ --operation-mode str Detector dark run acquiring operation mode, optional.
+ Default:
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --run-high int run number for G0 dark run, required. Default: None
+ --run-med int run number for G1 dark run, required. Default: None
+ --run-low int run number for G2 dark run, required. Default: None
+
+
+JUNGFRAU
+--------
+
+Jungfrau Offline Correction
++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+Offline Calibration for the Jungfrau Detector
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate JUNGFRAU CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run int [--sequences str [str ...]]
+ [--sequences-per-node int] [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-template str]
+ [--instrument-source-template str]
+ [--ctrl-source-template str] [--karabo-id-control str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--relative-gain | --no-relative-gain]
+ [--strixel-sensor | --no-strixel-sensor]
+ [--strixel-double-norm float] [--limit-trains int]
+ [--chunks-ids int] [--chunks-data int]
+ [--manual-slow-data | --no-manual-slow-data]
+ [--integration-time float] [--gain-setting int]
+ [--gain-mode int] [--mem-cells int] [--bias-voltage int]
+ [--skip-plots | --no-skip-plots] [--plot-trains int]
+ [--cell-id-preview int]
+ [--roi-definitions int [int ...]]
+ JUNGFRAU CORRECT
+
+ # Jungfrau Offline Correction #
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ Offline Calibration for the Jungfrau Detector
+
+ positional arguments:
+ JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences str [str ...]
+ sequences to correct, set to [-1] for all, range
+ allowed. Default: [-1]
+ --sequences-per-node int
+ number of sequence files per cluster node if run as
+ slurm job, set to 0 to not run SLURM parallel.
+ Default: 1
+ --karabo-id str karabo prefix of Jungfrau devices. Default:
+ SPB_IRDA_JF4M
+ --karabo-da str [str ...]
+ data aggregators. Default: ['JNGFR01', 'JNGFR02',
+ 'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
+ 'JNGFR08']
+ --receiver-template str
+ Detector receiver template for accessing raw data
+ files. e.g. "JNGFR{:02d}". Default: JNGFR{:02d}
+ --instrument-source-template str
+ template for source name (filled with karabo_id &
+ receiver_id). e.g.
+ 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
+ {}/DET/{}:daqOutput
+ --ctrl-source-template str
+ template for control source name (filled with
+ karabo_id_control). Default: {}/DET/CONTROL
+ --karabo-id-control str
+ if control is on a different ID, set to empty string
+ if it is the same a karabo-id. Default:
+ --use-dir-creation-date
+ use the creation data of the input dir for database
+ queries. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8017#8025
+ --cal-db-timeout int timeout on caldb requests. Default: 180000
+ --relative-gain do relative gain correction.. Default: True
+ --no-relative-gain Opposite of --relative-gain (default: True)
+ --strixel-sensor reordering for strixel detector layout.. Default:
+ False
+ --no-strixel-sensor Opposite of --strixel-sensor (default: False)
+ --strixel-double-norm float
+ normalization to use for double-size pixels, only
+ applied for strixel sensors.. Default: 2.0
+ --limit-trains int ONLY FOR TESTING. process only first N trains, Use 0
+ to process all.. Default: 0
+ --chunks-ids int HDF chunk size for memoryCell and frameNumber..
+ Default: 32
+ --chunks-data int HDF chunk size for pixel data in number of frames..
+ Default: 1
+ --manual-slow-data if true, use manually entered bias_voltage,
+ integration_time, gain_setting, and gain_mode values.
+ Default: False
+ --no-manual-slow-data
+ Opposite of --manual-slow-data (default: False)
+ --integration-time float
+ integration time in us, will be overwritten by value
+ in file. Default: 4.96
+ --gain-setting int 0 for dynamic gain, 1 for dynamic HG0, will be
+ overwritten by value in file. Default: 0
+ --gain-mode int 0 for runs with dynamic gain setting, 1 for fixgain.
+ It will be overwritten by value in file, if
+ manual_slow_data is set to True.. Default: 0
+ --mem-cells int Set mem_cells to -1 to automatically use the value
+ stored in RAW data.. Default: -1
+ --bias-voltage int will be overwritten by value in file. Default: 180
+ --skip-plots exit after writing corrected files. Default: False
+ --no-skip-plots Opposite of --skip-plots (default: False)
+ --plot-trains int Number of trains to plot for RAW and CORRECTED plots.
+ Set to -1 to automatically plot all trains.. Default:
+ 500
+ --cell-id-preview int
+ cell Id used for preview in single-shot plots.
+ Default: 15
+ --roi-definitions int [int ...]
+ List with groups of 6 values defining ROIs, e.g. [3,
+ 120, 180, 200, 550, -2] for module 3 (JNGFR03), slice
+ 120:180, 200:550, average along axis -2 (slow scan, or
+ -1 for fast scan). Default: [-1]
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --run int run to process, required. Default: None
+
+
+Jungfrau Dark Image Characterization
+++++++++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 2.0
+
+Analyzes Jungfrau dark image data to deduce offset, noise and resulting
+bad pixel maps
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate JUNGFRAU DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run-high int --run-med int --run-low int
+ [--karabo-da str [str ...]] [--karabo-id str]
+ [--karabo-id-control str] [--receiver-template str]
+ [--instrument-source-template str]
+ [--ctrl-source-template str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output]
+ [--badpixel-threshold-sigma float]
+ [--offset-abs-threshold-low int [int ...]]
+ [--offset-abs-threshold-high int [int ...]]
+ [--max-trains int] [--min-trains int]
+ [--manual-slow-data | --no-manual-slow-data]
+ [--time-limits float] [--integration-time int]
+ [--gain-setting int] [--gain-mode int]
+ [--bias-voltage int] [--memory-cells int]
+ [--detailed-report | --no-detailed-report]
+ [--operation-mode str]
+ JUNGFRAU DARK
+
+ # Jungfrau Dark Image Characterization #
+ ,
+ Author: European XFEL Detector Group, Version: 2.0,
+ ,
+ Analyzes Jungfrau dark image data to deduce offset, noise and resulting bad
+ pixel maps
+
+ positional arguments:
+ JUNGFRAU The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --karabo-da str [str ...]
+ list of data aggregators, which corresponds to
+ different JF modules. Default: ['JNGFR01', 'JNGFR02',
+ 'JNGFR03', 'JNGFR04', 'JNGFR05', 'JNGFR06', 'JNGFR07',
+ 'JNGFR08']
+ --karabo-id str karabo_id (detector identifier) prefix of Jungfrau
+ detector to process.. Default: SPB_IRDA_JF4M
+ --karabo-id-control str
+ if control is on a different ID, set to empty string
+ if it is the same a karabo-id. Default:
+ --receiver-template str
+ inset for receiver devices. Default: JNGFR{:02}
+ --instrument-source-template str
+ template for instrument source name (filled with
+ karabo_id & receiver_id). e.g.
+ 'SPB_IRDA_JF4M/DET/JNGFR01:daqOutput'. Default:
+ {}/DET/{}:daqOutput
+ --ctrl-source-template str
+ template for control source name (filled with
+ karabo_id_control). Default: {}/DET/CONTROL
+ --use-dir-creation-date
+ use dir creation date. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibrate db interface to connect to. Default:
+ tcp://max-exfl016:8016#8045
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --badpixel-threshold-sigma float
+ bad pixels defined by values outside n times this std
+ from median. Default: 5.0
+ --offset-abs-threshold-low int [int ...]
+ absolute bad pixel threshold in terms of offset, lower
+ values. Default: [1000, 10000, 10000]
+ --offset-abs-threshold-high int [int ...]
+ absolute bad pixel threshold in terms of offset, upper
+ values. Default: [8000, 15000, 15000]
+ --max-trains int Maximum trains to process darks. Set to 0 to process
+ all available train images. 1000 trains is enough
+ resolution to create the dark constants. Default: 1000
+ --min-trains int Minimum number of trains to process dark constants.
+ Raise a warning if the run has fewer trains.. Default:
+ 100
+ --manual-slow-data if true, use manually entered bias_voltage and
+ integration_time values. Default: False
+ --no-manual-slow-data
+ Opposite of --manual-slow-data (default: False)
+ --time-limits float to find calibration constants later on, the
+ integration time is allowed to vary by 0.5 us.
+ Default: 0.025
+ --integration-time int
+ integration time in us, will be overwritten by value
+ in file. Default: 1000
+ --gain-setting int 0 for dynamic, forceswitchg1, forceswitchg2, 1 for
+ dynamichg0, fixgain1, fixgain2. Will be overwritten by
+ value in file. Default: 0
+ --gain-mode int 1 if medium and low runs are fixgain1 and fixgain2,
+ otherwise 0. It will be overwritten by value in file,
+ if manual_slow_data. Default: 0
+ --bias-voltage int sensor bias voltage in V, will be overwritten by value
+ in file. Default: 90
+ --memory-cells int number of memory cells. Default: 16
+ --detailed-report Default: False
+ --no-detailed-report Opposite of --detailed-report (default: False)
+ --operation-mode str Detector operation mode, optional. Default:
+ ADAPTIVE_GAIN
+
+ required arguments:
+ --in-folder str folder under which runs are located, required.
+ Default: None
+ --out-folder str path to place reports at, required. Default: None
+ --run-high int run number for G0 dark run, required. Default: None
+ --run-med int run number for G1 dark run, required. Default: None
+ --run-low int run number for G2 dark run, required. Default: None
+
+
+LPD
+---
+
+LPD Offline Correction
+++++++++++++++++++++++
+
+Author: European XFEL Data Analysis Group
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequences int [int ...]] [--modules int [int ...]]
+ [--karabo-da str [str ...]] --run int [--karabo-id str]
+ [--input-source str] [--output-source str]
+ [--creation-time str] [--cal-db-interface str]
+ [--cal-db-timeout int] [--cal-db-root str]
+ [--mem-cells int] [--bias-voltage float]
+ [--capacitor str] [--photon-energy float]
+ [--category int]
+ [--use-cell-order | --no-use-cell-order]
+ [--offset-corr | --no-offset-corr]
+ [--rel-gain | --no-rel-gain] [--ff-map | --no-ff-map]
+ [--gain-amp-map | --no-gain-amp-map]
+ [--overwrite | --no-overwrite] [--chunks-data int]
+ [--chunks-ids int] [--create-virtual-cxi-in str]
+ [--sequences-per-node int] [--max-nodes int]
+ [--num-workers int] [--num-threads-per-worker int]
+ LPD CORRECT
+
+ # LPD Offline Correction #
+ Author: European XFEL Data Analysis Group
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences int [int ...]
+ Sequences to correct, use [-1] for all. Default: [-1]
+ --modules int [int ...]
+ Modules indices to correct, use [-1] for all, only
+ used when karabo_da is empty. Default: [-1]
+ --karabo-da str [str ...]
+ Data aggregators names to correct, use [''] for all.
+ Default: ['']
+ --karabo-id str Karabo domain for detector.. Default: FXE_DET_LPD1M-1
+ --input-source str Input fast data source.. Default:
+ {karabo_id}/DET/{module_index}CH0:xtdf
+ --output-source str Output fast data source, empty to use same as input..
+ Default:
+ --creation-time str The timestamp to use with Calibration DB. Required
+ Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
+ 11:02:41. Default:
+ --cal-db-interface str
+ Not needed, compatibility with current webservice..
+ Default:
+ --cal-db-timeout int Not needed, compatbility with current webservice..
+ Default: 0
+ --cal-db-root str Default: /gpfs/exfel/d/cal/caldb_store
+ --mem-cells int Memory cells, LPD constants are always taken with 512
+ cells.. Default: 512
+ --bias-voltage float Detector bias voltage.. Default: 250.0
+ --capacitor str Capacitor setting: 5pF or 50pF. Default: 5pF
+ --photon-energy float
+ Photon energy in keV.. Default: 9.2
+ --category int Whom to blame.. Default: 0
+ --use-cell-order Whether to use memory cell order as a detector
+ condition (not stored for older constants). Default:
+ False
+ --no-use-cell-order Opposite of --use-cell-order (default: False)
+ --offset-corr Offset correction.. Default: True
+ --no-offset-corr Opposite of --offset-corr (default: True)
+ --rel-gain Gain correction based on RelativeGain constant..
+ Default: True
+ --no-rel-gain Opposite of --rel-gain (default: True)
+ --ff-map Gain correction based on FFMap constant.. Default:
+ True
+ --no-ff-map Opposite of --ff-map (default: True)
+ --gain-amp-map Gain correction based on GainAmpMap constant..
+ Default: True
+ --no-gain-amp-map Opposite of --gain-amp-map (default: True)
+ --overwrite set to True if existing data should be overwritten.
+ Default: True
+ --no-overwrite Opposite of --overwrite (default: True)
+ --chunks-data int HDF chunk size for pixel data in number of frames..
+ Default: 1
+ --chunks-ids int HDF chunk size for cellId and pulseId datasets..
+ Default: 32
+ --create-virtual-cxi-in str
+ Folder to create virtual CXI files in (for each
+ sequence).. Default:
+ --sequences-per-node int
+ Sequence files to process per node. Default: 1
+ --max-nodes int Maximum number of SLURM jobs to split correction work
+ into. Default: 8
+ --num-workers int Worker processes per node, 8 is safe on 768G nodes but
+ won't work on 512G.. Default: 8
+ --num-threads-per-worker int
+ Number of threads per worker.. Default: 32
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default: None
+ --out-folder str the folder to output to, required. Default: None
+ --run int run to process, required. Default: None
+
+
+LPD Offset, Noise and Dead Pixels Characterization
+++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Author: M. Karnevskiy, S. Hauf
+
+This notebook performs re-characterize of dark images to derive offset,
+noise and bad-pixel maps. All three types of constants are evaluated
+per-pixel and per-memory cell.
+
+The notebook will correctly handle veto settings, but note that if you
+veto cells you will not be able to use these offsets for runs with
+different veto settings - vetoed cells will have zero offset.
+
+The evaluated calibration constants are stored locally and injected in
+the calibration data base.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ [--sequence int] [--modules str [str ...]] --run-high
+ int --run-med int --run-low int [--karabo-id str]
+ [--karabo-da str [str ...]] [--receiver-id str]
+ [--path-template str] [--h5path str] [--h5path-idx str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--capacitor-setting int]
+ [--mem-cells int] [--bias-voltage int]
+ [--thresholds-offset-sigma float]
+ [--thresholds-offset-hard int [int ...]]
+ [--thresholds-noise-sigma float]
+ [--thresholds-noise-hard int [int ...]]
+ [--skip-first-ntrains int]
+ [--skip-plots | --no-skip-plots] [--instrument str]
+ [--ntrains int]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ [--test-for-normality | --no-test-for-normality]
+ [--inject-cell-order | --no-inject-cell-order]
+ [--operation-mode str]
+ LPD DARK
+
+ # LPD Offset, Noise and Dead Pixels Characterization #
+ ,
+ Author: M. Karnevskiy, S. Hauf,
+ ,
+ This notebook performs re-characterize of dark images to derive offset, noise
+ and bad-pixel maps. All three types of constants are evaluated per-pixel and
+ per-memory cell.,
+ ,
+ The notebook will correctly handle veto settings, but note that if you veto
+ cells you will not be able to use these offsets for runs with different veto
+ settings - vetoed cells will have zero offset.,
+ ,
+ The evaluated calibration constants are stored locally and injected in the
+ calibration data base.,
+ ,
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequence int sequence files to evaluate. Default: 0
+ --modules str [str ...]
+ list of modules to evaluate, RANGE ALLOWED. Default:
+ [-1]
+ --karabo-id str karabo karabo_id. Default: FXE_DET_LPD1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for
+ selecting all data aggregators. Default: ['-1']
+ --receiver-id str inset for receiver devices. Default: {}CH0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{:05d}.h5
+ --h5path str path in the HDF5 file to images. Default:
+ /INSTRUMENT/{}/DET/{}:xtdf/image
+ --h5path-idx str path in the HDF5 file to images. Default:
+ /INDEX/{}/DET/{}:xtdf/image
+ --use-dir-creation-date
+ use the creation date of the directory for database
+ time derivation. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8015#8025
+ --cal-db-timeout int timeout on caldb requests". Default: 300000
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --capacitor-setting int
+ capacitor_setting for which data was taken. Default: 5
+ --mem-cells int number of memory cells used. Default: 512
+ --bias-voltage int detector bias voltage. Default: 250
+ --thresholds-offset-sigma float
+ bad pixel relative threshold in terms of n sigma
+ offset. Default: 3.0
+ --thresholds-offset-hard int [int ...]
+ bad pixel hard threshold. Default: [400, 1500]
+ --thresholds-noise-sigma float
+ bad pixel relative threshold in terms of n sigma
+ noise. Default: 7.0
+ --thresholds-noise-hard int [int ...]
+ bad pixel hard threshold. Default: [1, 35]
+ --skip-first-ntrains int
+ Number of first trains to skip. Default: 10
+ --skip-plots exit after writing corrected files. Default: False
+ --no-skip-plots Opposite of --skip-plots (default: False)
+ --instrument str instrument name. Default: FXE
+ --ntrains int number of trains to use. Default: 100
+ --high-res-badpix-3d plot bad-pixel summary in high resolution. Default:
+ False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+ --test-for-normality permorm normality test. Default: False
+ --no-test-for-normality
+ Opposite of --test-for-normality (default: False)
+ --inject-cell-order Include memory cell order as part of the detector
+ condition. Default: False
+ --no-inject-cell-order
+ Opposite of --inject-cell-order (default: False)
+ --operation-mode str Detector operation mode, optional. Default:
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+ --run-high int run number in which high gain data was recorded,
+ required. Default: None
+ --run-med int run number in which medium gain data was recorded,
+ required. Default: None
+ --run-low int run number in which low gain data was recorded,
+ required. Default: None
+
+
+LPD Radial X-ray Gain Evaluation
+++++++++++++++++++++++++++++++++
+
+Author: S. Hauf, Version 0.5
+
+Taking proper flat field for LPD can be difficult, as air scattering
+will always be present. Additionally, the large detector mandates a
+large distance to the source, in order to reduce :math:`1/r` effects.
+
+Because of this a radial evaluation method is used, which assumes that
+pixels a the same radial distance :math:`r` should on average have the
+same signal :math:`S(r)`.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD FF --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run int [--sequences int [int ...]]
+ [--capacitor-setting int] [--mem-cells int]
+ [--local-output | --no-local-output]
+ [--db-output | --no-db-output] [--bias-voltage int]
+ [--cal-db-interface str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--instrument str] [--limit-trains int]
+ [--geometry-file str]
+ [--beam-center-offset float [float ...]]
+ [--allowed-gain-thresholds float [float ...]]
+ [--badpix-entire-asic-threshold float]
+ [--photon-energy float] [--max-images int]
+ LPD FF
+
+ # LPD Radial X-ray Gain Evaluation #
+ ,
+ Author: S. Hauf, Version 0.5,
+ ,
+ Taking proper flat field for LPD can be difficult, as air scattering will
+ always be present. Additionally, the large detector mandates a large distance
+ to the source, in order to reduce $1/r$ effects.
+ ,
+ Because of this a radial evaluation method is used, which assumes that pixels
+ a the same radial distance $r$ should on average have the same signal $S(r)$.
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ FF Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sequences int [int ...]
+ which sequence files to use. Default: [0]
+ --capacitor-setting int
+ capacitor_setting for which data was taken. Default: 5
+ --mem-cells int number of memory cells used. Default: 512
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --bias-voltage int detector bias voltage. Default: 250
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl015:5005
+ --use-dir-creation-date
+ use the creation date of the directory for database
+ time derivation. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --instrument str Default: FXE
+ --limit-trains int limit the number of train for the evaluation. Default:
+ 10
+ --geometry-file str the geometry file to use, MAR 2018. Default:
+ /gpfs/exfel/d/cal/exchange/lpdMF_00.h5
+ --beam-center-offset float [float ...]
+ offset from the beam center, MAR 2018. Default: [1.5,
+ 1]
+ --allowed-gain-thresholds float [float ...]
+ relative gain values within these bounds are valid.
+ Default: [0.5, 1.5]
+ --badpix-entire-asic-threshold float
+ if more than this fraction of pixels on an ASIC are
+ "bad" the entire ASIC is flagged. Default: 0.25
+ --photon-energy float
+ photon enery in keV. Default: 9.2
+ --max-images int maximum number of images to use in evaluation.
+ Default: 1024
+
+ required arguments:
+ --in-folder str path to already corrected input data, required.
+ Default: None
+ --out-folder str path to output to, required. Default: None
+ --run int runs to process, required. Default: None
+
+
+Injecting calibration constant data to the database
++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+Author: European XFEL Detector Group, Version: 1.0
+
+Reading h5files of calibration constants to inject them to the database.
+Used for LPD
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD INJECT_CONSTANTS --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --constant-names str [str ...]
+ --in-folder str --out-folder str [--proposal str]
+ [--runs str [str ...]] [--karabo-id str]
+ [--karabo-da str [str ...]] [--cal-db-interface str]
+ [--memory-cells int] [--bias-voltage int]
+ [--capacitor int] [--category int]
+ [--photon-energy float] --creation-time str
+ LPD INJECT_CONSTANTS
+
+ # Injecting calibration constant data to the database #
+ ,
+ Author: European XFEL Detector Group, Version: 1.0,
+ ,
+ Reading h5files of calibration constants to inject them to the database. Used
+ for LPD
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ INJECT_CONSTANTS Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --proposal str Add proposal number to be sent to the database as a
+ part of Raw data location.. Default:
+ --runs str [str ...] Add list of runs to be sent to the database as a part
+ of Raw data location.. Default: ['']
+ --karabo-id str detector identifier.. Default: FXE_DET_LPD1M-1
+ --karabo-da str [str ...]
+ karabo data aggregators. default "all" for all 16
+ karabo data aggregator names.. Default: ['all']
+ --cal-db-interface str
+ calibration DB zmq address.. Default: tcp://max-
+ exfl016:8015#8045
+ --memory-cells int Number of memory cells. Used for constant conditions..
+ Default: 512
+ --bias-voltage int bias voltage value. Used for constant conditions..
+ Default: 250
+ --capacitor int capacitor value. Used for constant conditions..
+ Default: 5
+ --category int calibration constant source category, 0 for European
+ XFEl and 1 for RAL. Used for constant conditions..
+ Default: 0
+ --photon-energy float
+ calibration constant photon energy. Used for constant
+ conditions.. Default: 9.2
+
+ required arguments:
+ --constant-names str [str ...]
+ calibration constant name, required.. Default: None
+ --in-folder str calibration constants folder, required.. Default: None
+ --out-folder str output folder to store report path in case the
+ notebook is executed by CLI, required.. Default: None
+ --creation-time str creation time for the injected constants. required
+ format '2019-01-20T14:12:06'. Default: None
+
+
+LPD Gain Characterization (Charge Injection)
+++++++++++++++++++++++++++++++++++++++++++++
+
+Author: S. Hauf, Version: 0.5
+
+The following code characterizes the gain of the LPD detector from
+charge injection data, i.e. data with charge injected into the
+amplifiers, bypassing the sensor. The data needs to fulfil the following
+requirements:
+
+- each file should represent one scan point for one mudle, defined by
+ detector gain setting
+ and charge injections setting
+- settings need to overlap at at least one point for two neighboring
+ gain ranges
+- 100 samples or more per pixel and memory cell should be present for
+ each setting.
+
+The data is then analyzed by calcualting the per-pixel, per memory cell
+mean of the samples for each setting. These means are then normalized to
+the median peak position of a all means of the first module. Overlapping
+settings in neighboring gain ranges are used to deduce the slopes of the
+different gains with respect to the high gain setting.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD PC --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str] --in-folder
+ str --out-folder str [--offset-store str]
+ [--modules int [int ...]] [--capacitor-setting int]
+ [--mem-cells int] [--local-output | --no-local-output]
+ [--db-output | --no-db-output]
+ [--db-input | --no-db-input] [--bias-voltage int]
+ [--cal-db-interface str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--instrument str]
+ [--high-res-badpix-3d | --no-high-res-badpix-3d]
+ LPD PC
+
+ # LPD Gain Characterization (Charge Injection) #
+ ,
+ Author: S. Hauf, Version: 0.5,
+ ,
+ The following code characterizes the gain of the LPD detector from charge
+ injection data, i.e. data with charge injected into the amplifiers, bypassing
+ the sensor. The data needs to fulfil the following requirements:,
+ ,
+ * each file should represent one scan point for one mudle, defined by
+ detector gain setting
+ and charge injections setting,
+ * settings need to overlap at at least one point for two neighboring gain
+ ranges,
+ * 100 samples or more per pixel and memory cell should be present for each
+ setting.,
+ ,
+ The data is then analyzed by calcualting the per-pixel, per memory cell mean
+ of the samples for each setting. These means are then normalized to the
+ median peak position of a all means of the first module. Overlapping settings
+ in neighboring gain ranges are used to deduce the slopes of the different
+ gains with respect to the high gain setting.
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ PC Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ The ipcluster profile to use. Default: noDB
+ --offset-store str path to offset store, overwrite use_db_input. Default:
+ --modules int [int ...]
+ Default: [-1]
+ --capacitor-setting int
+ capacitor_setting for which data was taken. Default: 5
+ --mem-cells int number of memory cells used. Default: 512
+ --local-output output constants locally. Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --db-output output constants to database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --db-input Default: True
+ --no-db-input Opposite of --db-input (default: True)
+ --bias-voltage int detector bias voltage. Default: 300
+ --cal-db-interface str
+ the database interface to use. Default: tcp://max-
+ exfl016:8015
+ --use-dir-creation-date
+ use the creation date of the directory for database
+ time derivation. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --instrument str Default: FXE
+ --high-res-badpix-3d Default: False
+ --no-high-res-badpix-3d
+ Opposite of --high-res-badpix-3d (default: False)
+
+ required arguments:
+ --in-folder str path to input data, required. Default: None
+ --out-folder str path to output to, required. Default: None
+
+
+Mine XGM Data and LPD Profile
++++++++++++++++++++++++++++++
+
+Author: S. Hauf, Version: 0.1
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate LPD XGM_MINE --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--dc float [float ...]]
+ [--geometry-file str] [--proposal-folder str]
+ [--xgm-folder str] [--adc-folder str]
+ [--file-prefix str] [--raw-prefix str]
+ [--xgm-prefix str] [--adc-prefix str] [--data-path str]
+ [--gain-path str] [--xgm-path str] [--xgm-path-td str]
+ [--adc-root str] [--adc-paths str [str ...]]
+ [--temp-out-folder str] [--cluster-profile str]
+ [--chunk-size int] [--runs str [str ...]] [--method str]
+ [--sequences str [str ...]]
+ LPD XGM_MINE
+
+ # Mine XGM Data and LPD Profile #
+ Author: S. Hauf, Version: 0.1
+
+ positional arguments:
+ LPD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ XGM_MINE Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --dc float [float ...]
+ center offset. Default: [1.5, 1]
+ --geometry-file str the geometry file to use, MAR 2018. Default:
+ /gpfs/exfel/d/cal/exchange/lpdMF_00.h5
+ --proposal-folder str
+ folder under which runs can be found. Default:
+ /gpfs/exfel/exp/FXE/201701/p002045/proc/
+ --xgm-folder str folder under which runs can be found. Default:
+ /gpfs/exfel/exp/FXE/201701/p002045/raw/
+ --adc-folder str folder under which runs can be found. Default:
+ /gpfs/exfel/exp/FXE/201701/p002045/raw/
+ --file-prefix str prefix of each data file - should be a template to
+ replace run number. Default: CORR-R{:04d}-LPD
+ --raw-prefix str prefix of each data file - should be a template to
+ replace run number. Default: RAW-R{:04d}-LPD
+ --xgm-prefix str prefix for XGM file - should be a template to replace
+ run number. Default: RAW-R{:04d}-DA03
+ --adc-prefix str prefix for ADC file - should be a template to replace
+ run number. Default: RAW-R{:04d}-DA01
+ --data-path str path to data in file. Default:
+ INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/data
+ --gain-path str path to data in file. Default:
+ INSTRUMENT/FXE_DET_LPD1M-1/DET/{}CH0:xtdf/image/gain
+ --xgm-path str path to XGM data. Default: /INSTRUMENT/SA1_XTD2_XGM/DO
+ OCS/MAIN:output/data/intensityTD
+ --xgm-path-td str path to XGM AUX data. Default: /INSTRUMENT/SA1_XTD2_XG
+ M/DOOCS/MAIN:output/data/intensityAUXTD
+ --adc-root str root path to ADC data. Default:
+ /INSTRUMENT/FXE_RR_DAQ/ADC/1:network/digitizers
+ --adc-paths str [str ...]
+ Default: ['channel_1_A/apd/pulseIntegral',
+ 'channel_1_B/apd/pulseIntegral',
+ 'channel_1_C/apd/pulseIntegral',
+ 'channel_1_D/apd/pulseIntegral']
+ --temp-out-folder str
+ Default: /gpfs/exfel/data/scratch/haufs/test/tempout6
+ --cluster-profile str
+ Default: noDB
+ --chunk-size int read this amount of data at once. Default: 512
+ --runs str [str ...] Default: 69-72
+ --method str method to use for evaluation of images: radial,
+ average. Default: average
+ --sequences str [str ...]
+ range allowed. Default: [-1]
+
+
+PNCCD
+-----
+
+pnCCD Data Correction
++++++++++++++++++++++
+
+Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0
+
+The following notebook provides offset, common mode, relative gain,
+split events and pattern classification corrections of images acquired
+with the pnCCD. This notebook *does not* yet correct for charge transfer
+inefficiency.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate PNCCD CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--in-folder str] [--out-folder str]
+ [--run int] [--sequences str [str ...]]
+ [--sequences-per-node int] [--karabo-da str]
+ [--karabo-id str] [--receiver-id str]
+ [--path-template str] [--instrument-source-template str]
+ [--commonModeAxis int]
+ [--commonModeBlockSize int [int ...]]
+ [--split-evt-primary-threshold float]
+ [--split-evt-secondary-threshold float]
+ [--saturated-threshold float]
+ [--fix-temperature-top float]
+ [--fix-temperature-bot float] [--gain int]
+ [--bias-voltage float] [--integration-time int]
+ [--photon-energy float] [--cal-db-interface str]
+ [--cal-db-timeout int] [--creation-time str]
+ [--only-offset | --no-only-offset]
+ [--common-mode | --no-common-mode]
+ [--relgain | --no-relgain]
+ [--pattern-classification | --no-pattern-classification]
+ [--chunk-size-idim int] [--limit-images int]
+ PNCCD CORRECT
+
+ # pnCCD Data Correction #
+ ,
+ Authors: DET Group, Modified by Kiana Setoodehnia - Version 5.0,
+ ,
+ The following notebook provides offset, common mode, relative gain, split
+ events and pattern classification corrections of images acquired with the
+ pnCCD. This notebook *does not* yet correct for charge transfer inefficiency.
+
+ positional arguments:
+ PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --in-folder str input folder. Default:
+ /gpfs/exfel/exp/SQS/202031/p900166/raw
+ --out-folder str output folder. Default: /gpfs/exfel/data/scratch/ahmed
+ k/test/remove/pnccd_correct
+ --run int which run to read data from. Default: 347
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --sequences-per-node int
+ number of sequences running on the same slurm node..
+ Default: 1
+ --karabo-da str data aggregators. Default: PNCCD01
+ --karabo-id str karabo prefix of PNCCD devices. Default:
+ SQS_NQS_PNCCD1MP
+ --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{:05d}.h5
+ --instrument-source-template str
+ template for data source name, will be filled with
+ karabo_id and receiver_id.. Default: {}/CAL/{}:output
+ --commonModeAxis int axis along which common mode will be calculated, 0 =
+ row, and 1 = column. Default: 0
+ --commonModeBlockSize int [int ...]
+ size of the detector in pixels for common mode
+ calculations. Default: [512, 512]
+ --split-evt-primary-threshold float
+ primary threshold for split event classification in
+ terms of n sigma noise. Default: 4.0
+ --split-evt-secondary-threshold float
+ secondary threshold for split event classification in
+ terms of n sigma noise. Default: 3.0
+ --saturated-threshold float
+ full well capacity in ADU. Default: 32000.0
+ --fix-temperature-top float
+ fix temperature for top sensor in K, set to 0. to use
+ value from slow data.. Default: 0.0
+ --fix-temperature-bot float
+ fix temperature for bottom senspr in K, set to 0. to
+ use value from slow data.. Default: 0.0
+ --gain int the detector's gain setting. Set to -1 to use the
+ value from the slow data.. Default: -1
+ --bias-voltage float the detector's bias voltage. set to 0. to use value
+ from slow data.. Default: 0.0
+ --integration-time int
+ detector's integration time. Default: 70
+ --photon-energy float
+ Al fluorescence in keV. Default: 1.6
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8015
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --creation-time str The timestamp to use with Calibration DB. Required
+ Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04
+ 11:02:41. Default:
+ --only-offset Only, apply offset.. Default: False
+ --no-only-offset Opposite of --only-offset (default: False)
+ --common-mode Apply common mode correction. Default: True
+ --no-common-mode Opposite of --common-mode (default: True)
+ --relgain Apply relative gain correction. Default: True
+ --no-relgain Opposite of --relgain (default: True)
+ --pattern-classification
+ classify split events. Default: True
+ --no-pattern-classification
+ Opposite of --pattern-classification (default: True)
+ --chunk-size-idim int
+ H5 chunking size of output data. Default: 1
+ --limit-images int this parameter is used for limiting number of images
+ to correct from a sequence file. ONLY FOR TESTING..
+ Default: 0
+
+
+pnCCD Dark Characterization
++++++++++++++++++++++++++++
+
+Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0
+
+The following notebook provides dark image analysis of the pnCCD
+detector. Dark characterization evaluates offset and noise of the
+detector and gives information about bad pixels.
+
+On the first iteration, the offset and noise maps are generated. Initial
+bad pixels map is obtained based on the offset and initial noise maps.
+Edge pixels are also added to the bad pixels map.
+
+On the second iteration, the noise map is corrected for common mode. A
+second bad pixel map is generated based on the offset map and
+offset-and-common-mode-corrected noise map. Then, the hole in the center
+of the CCD is added to the second bad pixel map.
+
+On the third and final iteration, the pixels that show up on the
+abovementioned bad pixels map are masked. Possible events due to cosmic
+rays are found and masked. The data are then again offset and common
+mode corrected and a new final noise and bad pixels maps are generated.
+
+These latter resulting maps together with the offset map are saved as
+.h5 files to a local path for a later use. These dark constants are not
+automatically sent to the database.
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate PNCCD DARK --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --in-folder str --out-folder str
+ --run int [--karabo-da str [str ...]] [--karabo-id str]
+ [--receiver-id str] [--path-template str]
+ [--instrument-source-template str]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output]
+ [--creation-time str] [--fix-temperature-top float]
+ [--fix-temperature-bot float] [--temp-limits int]
+ [--gain int] [--bias-voltage float]
+ [--integration-time int] [--commonModeAxis int]
+ [--commonModeBlockSize int [int ...]]
+ [--sigmaNoise float] [--bad-pixel-offset-sigma float]
+ [--bad-pixel-noise-sigma float] [--max-trains int]
+ [--min-trains int] [--operation-mode str]
+ [--db-module str]
+ PNCCD DARK
+
+ # pnCCD Dark Characterization
+ ,
+ Author: DET Group, modified by Kiana Setoodehnia, Version: 5.0,
+ ,
+ The following notebook provides dark image analysis of the pnCCD detector.
+ Dark characterization evaluates offset and noise of the detector and gives
+ information about bad pixels.
+ ,
+ On the first iteration, the offset and noise maps are generated. Initial bad
+ pixels map is obtained based on the offset and initial noise maps. Edge
+ pixels are also added to the bad pixels map.,
+ ,
+ On the second iteration, the noise map is corrected for common mode. A second
+ bad pixel map is generated based on the offset map and
+ offset-and-common-mode-corrected noise map. Then, the hole in the center of
+ the CCD is added to the second bad pixel map.,
+ ,
+ On the third and final iteration, the pixels that show up on the
+ abovementioned bad pixels map are masked. Possible events due to cosmic rays
+ are found and masked. The data are then again offset and common mode
+ corrected and a new final noise and bad pixels maps are generated.,
+ ,
+ These latter resulting maps together with the offset map are saved as .h5
+ files to a local path for a later use. These dark constants are not
+ automatically sent to the database.
+
+ positional arguments:
+ PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ DARK Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --karabo-da str [str ...]
+ data aggregators. Default: ['PNCCD01']
+ --karabo-id str karabo prefix of PNCCD devices. Default:
+ SQS_NQS_PNCCD1MP
+ --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
+ --path-template str the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --instrument-source-template str
+ data source path in h5file. Template filled with
+ karabo_id and receiver_id. Default: {}/CAL/{}:output
+ --use-dir-creation-date
+ use dir creation date as data production reference
+ date. Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8021
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --db-output if True, the notebook sends dark constants to the
+ calibration database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output if True, the notebook saves dark constants locally.
+ Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --creation-time str To overwrite the measured creation_time. Required
+ Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
+ 11:02:41.00. Default:
+ --fix-temperature-top float
+ fix temperature of top pnCCD sensor in K. Set to 0, to
+ use the value from slow data. Default: 0.0
+ --fix-temperature-bot float
+ fix temperature of bottom pnCCD sensor in K. Set to 0,
+ to use the value from slow data. Default: 0.0
+ --temp-limits int temperature limits in which calibration parameters are
+ considered equal. Default: 5
+ --gain int the detector's gain setting. Set to -1 to use the
+ value from the slow data.. Default: -1
+ --bias-voltage float the detector's bias voltage. set to 0. to use the
+ value from slow data.. Default: 0.0
+ --integration-time int
+ detector's integration time.. Default: 70
+ --commonModeAxis int axis along which common mode will be calculated (0:
+ along rows, 1: along columns). Default: 0
+ --commonModeBlockSize int [int ...]
+ size of the detector in pixels for common mode
+ calculations. Default: [512, 512]
+ --sigmaNoise float pixels whose signal value exceeds sigmaNoise*noise
+ will be considered as cosmics and are masked. Default:
+ 10.0
+ --bad-pixel-offset-sigma float
+ any pixel whose offset beyond this standard deviations
+ is a bad pixel. Default: 4.0
+ --bad-pixel-noise-sigma float
+ any pixel whose noise beyond this standard deviations
+ is a bad pixel. Default: 4.0
+ --max-trains int Maximum number of trains to use for dark processing.
+ Set to 0 to process all trains.. Default: 500
+ --min-trains int Minimum number of trains to proceed with dark
+ processing.. Default: 1
+ --operation-mode str Detector operation mode, optional. Default:
+ --db-module str the device name for pnCCD detector. Default:
+
+ required arguments:
+ --in-folder str input folder, required. Default: None
+ --out-folder str output folder, required. Default: None
+ --run int which run to read data from, required. Default: None
+
+
+pnCCD Gain Characterization
++++++++++++++++++++++++++++
+
+Authors: DET Group, modified by Kiana Setoodehnia on December 2020 -
+Version 4.0
+
+The following notebook provides gain characterization for the pnCCD. It
+relies on data which are previously corrected using the Meta Data
+Catalog web service interface or by running the Correct_pnCCD_NBC.ipynb
+notebook. Prior to running this notebook, the corrections which should
+be applied by the web service or the aforementioned notebook are as
+follows:
+
+- offset correction
+- common mode correction
+- split pattern classification
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate PNCCD RELGAIN --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--cluster-profile str]
+ [--in-folder str] [--out-folder str] [--run int]
+ [--sequences str [str ...]] [--db-module str]
+ [--karabo-da str] [--karabo-da-control str]
+ [--karabo-id str] [--receiver-id str]
+ [--path-template str] [--path-template-ctrl str]
+ [--path-template-seqs str] [--h5path str]
+ [--h5path-ctrl str] [--cpuCores int]
+ [--use-dir-creation-date | --no-use-dir-creation-date]
+ [--sequences-per-node int] [--chunkSize int]
+ [--run-parallel | --no-run-parallel]
+ [--db-output | --no-db-output]
+ [--local-output | --no-local-output]
+ [--cal-db-interface str] [--cal-db-timeout int]
+ [--creation-time str] [--fix-temperature-top float]
+ [--fix-temperature-bot float] [--gain float]
+ [--bias-voltage float] [--integration-time int]
+ [--photon-energy float]
+ PNCCD RELGAIN
+
+ # pnCCD Gain Characterization #
+ ,
+ Authors: DET Group, modified by Kiana Setoodehnia on December 2020 - Version
+ 4.0,
+ ,
+ The following notebook provides gain characterization for the pnCCD. It
+ relies on data which are previously corrected using the Meta Data Catalog
+ web service interface or by running the Correct_pnCCD_NBC.ipynb notebook.
+ Prior to running this notebook, the corrections which should be applied by
+ the web service or the aforementioned notebook are as follows:,
+ ,
+ - offset correction,
+ - common mode correction,
+ - split pattern classification
+
+ positional arguments:
+ PNCCD The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ RELGAIN Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --cluster-profile str
+ ipcluster profile to use. Default: noDB
+ --in-folder str input folder for the raw data. Default:
+ /gpfs/exfel/exp/SQS/202031/p900166/raw
+ --out-folder str output folder. Default:
+ /gpfs/exfel/data/scratch/setoodeh/Test
+ --run int which run to read data from. Default: 347
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range
+ allowed. Default: [-1]
+ --db-module str Default: pnCCD_M205_M206
+ --karabo-da str data aggregators. Default: PNCCD01
+ --karabo-da-control str
+ file inset for control data. Default: PNCCD02
+ --karabo-id str karabo prefix of PNCCD devices. Default:
+ SQS_NQS_PNCCD1MP
+ --receiver-id str inset for receiver devices. Default: PNCCD_FMT-0
+ --path-template str the template to use to access data. Default:
+ CORR-R{:04d}-PNCCD01-S{{:05d}}.h5
+ --path-template-ctrl str
+ the template to use to access data. Default:
+ RAW-R{:04d}-{}-S{{:05d}}.h5
+ --path-template-seqs str
+ Default: {}/r{:04d}/*PNCCD01-S*.h5
+ --h5path str path to data in the HDF5 file. Default:
+ /INSTRUMENT/{}/CAL/{}:output/data/
+ --h5path-ctrl str Default: /CONTROL/{}/CTRL/TCTRL
+ --cpuCores int specifies the number of running cpu cores. Default: 40
+ --use-dir-creation-date
+ this is needed to obtain creation time of the run.
+ Default: True
+ --no-use-dir-creation-date
+ Opposite of --use-dir-creation-date (default: True)
+ --sequences-per-node int
+ Default: 1
+ --chunkSize int number of images to read per chunk. Default: 100
+ --run-parallel Default: True
+ --no-run-parallel Opposite of --run-parallel (default: True)
+ --db-output if True, the notebook injects dark constants into the
+ calibration database. Default: False
+ --no-db-output Opposite of --db-output (default: False)
+ --local-output if True, the notebook saves dark constants locally.
+ Default: True
+ --no-local-output Opposite of --local-output (default: True)
+ --cal-db-interface str
+ calibration DB interface to use. Default: tcp://max-
+ exfl016:8015
+ --cal-db-timeout int timeout on caldb requests. Default: 300000
+ --creation-time str To overwrite the measured creation_time. Required
+ Format: YYYY-MM-DD HR:MN:SC.00 e.g. 2019-07-04
+ 11:02:41.00. Default:
+ --fix-temperature-top float
+ fix temperature of top pnCCD sensor in K, set to 0. to
+ use the value from slow data. Default: 0.0
+ --fix-temperature-bot float
+ fix temperature of bottom pnCCD sensor in K, set to 0.
+ to use the value from slow data. Default: 0.0
+ --gain float the detector's gain setting. Set to 0, to use the
+ value from slow data. Default: 0.1
+ --bias-voltage float the detector's bias voltage. set to 0. to use the
+ value from slow data.. Default: 0.0
+ --integration-time int
+ detector's integration time. Default: 70
+ --photon-energy float
+ Al fluorescence in keV. Default: 1.6
+
+
+REMI
+----
+
+Transformation parameters
++++++++++++++++++++++++++
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate REMI CORRECT --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] [--run int] [--in-folder str]
+ [--out-folder str] [--calib-config-path str]
+ [--cycle str] [--cal-db-timeout int]
+ [--cal-db-interface str] [--karabo-da str]
+ [--karabo-id str] [--proposal str]
+ [--out-aggregator str] [--out-seq-len int]
+ [--det-device-id str] [--det-output-key str]
+ [--save-raw-triggers | --no-save-raw-triggers]
+ [--save-raw-edges | --no-save-raw-edges]
+ [--save-rec-signals | --no-save-rec-signals]
+ [--save-rec-hits | --no-save-rec-hits]
+ [--chunks-triggers int [int ...]]
+ [--chunks-edges int [int ...]]
+ [--chunks-hits int [int ...]]
+ [--chunks-signals int [int ...]]
+ [--dataset-compression str]
+ [--dataset-compression-opts int]
+ [--quad-anode | --no-quad-anode] [--ppt-source str]
+ [--ignore-fel | --no-ignore-fel]
+ [--ignore-ppl | --no-ignore-ppl] [--ppl-offset int]
+ [--laser-ppt-mask int] [--instrument-sase int]
+ [--first-pulse-offset int] [--single-pulse-length int]
+ [--mp-find-triggers float] [--mp-find-edges float]
+ [--mt-avg-trace int] [--mp-rec-hits float]
+ REMI CORRECT
+
+ # Transformation parameters
+
+ positional arguments:
+ REMI The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ CORRECT Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --run int Run ID.. Default: 104
+ --in-folder str Partial input path appended with run ID.. Default:
+ /gpfs/exfel/exp/SQS/202101/p002535/raw
+ --out-folder str Full path to output folder.. Default:
+ /gpfs/exfel/exp/SQS/202101/p002535/scratch/cal_test
+ --calib-config-path str
+ Path to correction and transform configuration.
+ Default: /gpfs/exfel/exp/SQS/202101/p002535/usr/config
+ _board2+4.yaml
+ --cycle str Proposal cycle, currently not used.. Default:
+ --cal-db-timeout int Calibration DB timeout, currently not used.. Default:
+ 0
+ --cal-db-interface str
+ Calibration DB interface, currently not used..
+ Default: foo
+ --karabo-da str Karabo data aggregator name, currently not used.
+ Default: bar
+ --karabo-id str Karabo device ID root for virtual output device..
+ Default: SQS_REMI_DLD6
+ --proposal str Proposal, leave empty for auto detection based on
+ in_folder. Default:
+ --out-aggregator str Aggregator name for output files.. Default: REMI01
+ --out-seq-len int Number of trains per sequence file in output..
+ Default: 5000
+ --det-device-id str Karabo device ID for virtual output device.. Default:
+ {karabo_id}/DET/{det_name}
+ --det-output-key str Pipeline name for fast data output.. Default: output
+ --save-raw-triggers Whether to save trigger position in files.. Default:
+ True
+ --no-save-raw-triggers
+ Opposite of --save-raw-triggers (default: True)
+ --save-raw-edges Whether to save digitized edge positions in files..
+ Default: True
+ --no-save-raw-edges Opposite of --save-raw-edges (default: True)
+ --save-rec-signals Whether to save reconstructed signals (u1-w2, mcp) in
+ files.. Default: True
+ --no-save-rec-signals
+ Opposite of --save-rec-signals (default: True)
+ --save-rec-hits Whether to save reoncstructed hits (x,y,t,m) in
+ files.. Default: True
+ --no-save-rec-hits Opposite of --save-rec-hits (default: True)
+ --chunks-triggers int [int ...]
+ HDF chunk size for triggers.. Default: [500]
+ --chunks-edges int [int ...]
+ HDF chunk size for edges.. Default: [500, 7, 50]
+ --chunks-hits int [int ...]
+ HDF chunk size for hits.. Default: [50, 50]
+ --chunks-signals int [int ...]
+ HDF chunk size for signals.. Default: [50, 50]
+ --dataset-compression str
+ HDF compression method.. Default: gzip
+ --dataset-compression-opts int
+ HDF GZIP compression level.. Default: 3
+ --quad-anode Reconstruction assumes a hex anode by default, change
+ for quad anodes.. Default: False
+ --no-quad-anode Opposite of --quad-anode (default: False)
+ --ppt-source str Default: SQS_RR_UTC/TSYS/TIMESERVER:outputBunchPattern
+ --ignore-fel Ignore any FEL entries in the PPT.. Default: False
+ --no-ignore-fel Opposite of --ignore-fel (default: False)
+ --ignore-ppl Ignore any PPL entries in the PPT.. Default: False
+ --no-ignore-ppl Opposite of --ignore-ppl (default: False)
+ --ppl-offset int In units of the PPT.. Default: 0
+ --laser-ppt-mask int Bit mask for used laser, negative to auto-detect from
+ instrument.. Default: -1
+ --instrument-sase int
+ Default: 3
+ --first-pulse-offset int
+ Default: 1000
+ --single-pulse-length int
+ Default: 25000
+ --mp-find-triggers float
+ Parallelization for finding triggers.. Default: 0.5
+ --mp-find-edges float
+ Parallelization for digitizing analog signal..
+ Default: 0.5
+ --mt-avg-trace int Parallelization for trace averaging.. Default: 2
+ --mp-rec-hits float Parallelization for hit reconstruction.. Default: 1.0
+
+
+TUTORIAL
+--------
+
+Tutorial Calculation
+++++++++++++++++++++
+
+Author: Astrid Muennich
+
+Version: 0.1
+
+A small example how to adapt a notebook to run with the offline
+calibration package “pycalibationâ€.
+
+The first cell contains all parameters that should be exposed to the
+command line.
+
+To run this notebook with different input parameters in parallel by
+submitting multiple SLURM jobs, for example for various random seed we
+can do the following:
+
+xfel-calibrate TUTORIAL TEST –random-seed 1,2,3,4
+
+or
+
+xfel-calibrate TUTORIAL TEST –random-seed 1-5
+
+will produce 4 processing jobs
+
+To invoke this notebook and display help use:
+
+.. code-block:: bash
+
+ xfel-calibrate TUTORIAL TEST --help
+
+The full parameter list of this notebook (with defaults is):
+
+.. code-block:: bash
+
+ usage: xfel-calibrate [-h] [--no-cluster-job] [--prepare-only]
+ [--report-to str] [--not-reproducible] [--skip-report]
+ [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int]
+ [--slurm-name str] [--slurm-scheduling int]
+ [--request-time str] [--slurm-partition str]
+ [--reservation str] --out-folder str
+ [--sensor-size int [int ...]]
+ [--random-seed str [str ...]] [--runs int]
+ TUTORIAL TEST
+
+ # Tutorial Calculation #
+ ,
+ Author: Astrid Muennich,
+ ,
+ Version: 0.1,
+ ,
+ A small example how to adapt a notebook to run with the offline calibration
+ package "pycalibation".,
+ ,
+ The first cell contains all parameters that should be exposed to the command
+ line.,
+ ,
+ To run this notebook with different input parameters in parallel by
+ submitting multiple SLURM jobs, for example for various random seed we can do
+ the following:,
+ ,
+ xfel-calibrate TUTORIAL TEST --random-seed 1,2,3,4,
+ ,
+ or,
+ ,
+ xfel-calibrate TUTORIAL TEST --random-seed 1-5,
+ ,
+ will produce 4 processing jobs
+
+ positional arguments:
+ TUTORIAL The detector to calibrate: AGIPD, LPD, PNCCD, GENERIC,
+ TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100,
+ EPIX10K, DSSC, REMI, TEST, TEST-RAISES-ERRORS
+ TEST Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --no-cluster-job Do not run as a cluster job (default: False)
+ --prepare-only Prepare notebooks but don't run them (default: False)
+ --report-to str Filename (and optionally path) for output report
+ (default: None)
+ --not-reproducible Disable checks to allow the processing result to not
+ be reproducible based on its metadata. (default:
+ False)
+ --skip-report Skip report generation in finalize step. (default:
+ False)
+ --skip-env-freeze Skip recording the Python environment for
+ reproducibility purposes, requires --not-reproducible
+ to run. (default: False)
+ --concurrency-par str
+ Name of concurrency parameter.If not given, it is
+ taken from configuration. (default: None)
+ --constants-from str Path to a calibration-metadata.yml file. If given,
+ retrieved-constants will be copied to use for a new
+ correction. (default: None)
+ --vector-figs Use vector graphics for figures in the report.
+ (default: False)
+ --slurm-mem int Requested node RAM in GB (default: 500)
+ --slurm-name str Name of slurm job (default: xfel_calibrate)
+ --slurm-scheduling int
+ Change scheduling priority for a slurm job +-
+ 2147483645 (negative value increases priority)
+ (default: 0)
+ --request-time str Time of request to process notebook. Iso format
+ (default: Now)
+ --slurm-partition str
+ Submit jobs in this Slurm partition (default: )
+ --reservation str Submit jobs in this Slurm reservation, overriding
+ --slurm-partition if both are set (default: )
+ --sensor-size int [int ...]
+ defining the picture size. Default: [10, 30]
+ --random-seed str [str ...]
+ random seed for filling of fake data array. Change it
+ to produce different results, range allowed. Default:
+ [1, 2]
+ --runs int how may iterations to fill histograms. Default: 500
+
+ required arguments:
+ --out-folder str output folder, required. Default: None
+
+
diff --git a/docs/cal_tools.md b/docs/cal_tools.md
new file mode 100644
index 0000000000000000000000000000000000000000..c1df5b3b0e600916947895c0ac837f86b077a0e5
--- /dev/null
+++ b/docs/cal_tools.md
@@ -0,0 +1,11 @@
+# Calibration Tools
+
+## Data File
+
+::: cal_tools.files
+
+## Step Timer
+
+::: cal_tools.step_timing
+
+
diff --git a/docs/cal_tools_algorithms.md b/docs/cal_tools_algorithms.md
new file mode 100644
index 0000000000000000000000000000000000000000..5bd185e3fadce8f7fca2f2603a6ebea4760d7036
--- /dev/null
+++ b/docs/cal_tools_algorithms.md
@@ -0,0 +1,5 @@
+# Detector correction functions
+
+## AGIPD Correction
+
+::: cal_tools.agipdlib
diff --git a/docs/calcat_interface.md b/docs/calcat_interface.md
new file mode 100644
index 0000000000000000000000000000000000000000..716203745a3f7b69170cee4106576857d9daf0a0
--- /dev/null
+++ b/docs/calcat_interface.md
@@ -0,0 +1,38 @@
+# Contacting the database
+
+## CALCAT interface
+
+To connect to the CALCAT database and retrieve calibration constants.
+You can use the provided interface to use the calibration client API
+provided by ITDM to retrieve the desired calibration constants and
+retrieve related information e.g. detector mapping.
+
+
+::: cal_tools.calcat_interface.CalibrationData
+ options:
+ members:
+ - metadata
+ - ndarray
+ - ndarray_map
+ show_source: false
+
+::: cal_tools.calcat_interface.SplitConditionCalibrationData
+
+::: cal_tools.calcat_interface.LPD_CalibrationData
+
+::: cal_tools.calcat_interface.DSSC_CalibrationData
+
+::: cal_tools.calcat_interface.JUNGFRAU_CalibrationData
+
+::: cal_tools.calcat_interface.PNCCD_CalibrationData
+
+::: cal_tools.calcat_interface.JUNGFRAU_CalibrationData
+
+::: cal_tools.calcat_interface.EPIX100_CalibrationData
+
+::: cal_tools.calcat_interface.GOTTHARD2_CalibrationData
+
+::: cal_tools.calcat_interface.CalCatError
+
+This is the current error type for this module. And it means that the
+used API faced an Exception like no constants for the selected modules.
diff --git a/docs/calibration_database.md b/docs/calibration_database.md
new file mode 100644
index 0000000000000000000000000000000000000000..b0922a69b52b61a4b2021f141354bc5dbdd3b376
--- /dev/null
+++ b/docs/calibration_database.md
@@ -0,0 +1,122 @@
+# Calibration database
+
+This is the main database where all calibration constants are stored.
+The calibration database is based on CALCAT as the main web interface
+and the GPFS file system where all calibration constant files are
+stored.
+
+## CALCAT
+
+
+
+CALCAT can be accessed through <https://in.xfel.eu/calibration>. It
+consists of multiple definitions and metadata related to the stored
+calibration constants. the definitions for the available detector types
+(AGIPD, JUNGFRAU, ePix100, ...), the available instruments, the
+available detectors in all instrument, the mapping of each detector to
+it's connected physical detector units, and all calibration constants.
+
+All of these content ensure retrieving the correct calibration constant
+any point in time.
+
+Through CALCAT page one can check all stored calibration constants,
+where the files are stored, and their metadata. Additionally, there are
+different ways to edit various parameters like the detector to PDU
+mapping \<TODO: add a definition link\>, adding new calibrations, and
+new parameter conditions.
+
+In order to proceed with the relevant parts of CALCAT, it is critical to
+first establish clear definitions for the terms used in both CALCAT and
+pyCalibration within the context of the Calibration database.
+
+- Calibrations: These are the types of calibration available that can
+ be applied to RAW data. For example Offset, Noise, and BadPixelsDark
+ are some calibrations available in CALCAT.
+- Calibration Constants (CC): Calibration Constants are a list of
+ calibration constant versions with the same parameter conditions.
+ For example 3 Offsets injected in with 3 different creation time for
+ AGIPD and the same parameter conditions are in the same CC list.
+
+
+
+- Calibration Constant Version (CCV): CCVs are injected through time
+ into the database. Each CCV must have a list of parameter
+ conditions, an attached timestamp of when it was created, the
+ physical detector unit, the file path, and the dataset name to
+ access its data.
+
+
+
+- Parameters conditions: The CC parameter conditions are decided by
+ detector experts because the CC\'s data would differ based on it.
+ Some examples are memory cells, integration time, and bias voltage.
+
+
+- PDU: Physical detector unit is the physical name given by the manufacturer or detector experts to the detector module's hardware. This representation is very important to calibration, as it is only trace of the physical movement for the detector modules.
+
+
+
+- Detector: A detector for CALCAT is based on a detector identifier
+ name, which was based on the Karabo ID give for detector karabo
+ devices. A detector can consist of one module like
+ HED_IA1_JF500K1, or multiple modules like MID_DET_AGIPD1M-1,
+ FXE_DET_LPD1M-1, and SPB_IRDA_JF4M.
+
+
+
+- Reports: pyCalibration's PDF reports are the main data
+ visualization source for any calibration processing, either
+ correcting raw data or generating calibration constants. For CALCAT
+ the only relevant reports are the reports for generating new
+ calibration constants. The paths for these reports are one injected
+ metadata along with the calibration constant.
+
+
+
+
+## Detector Mapping
+
+Detector mapping is the operation of associating physical detector units
+(modules) with its data sources (raw data, data aggregators and calibration
+constants files)
+
+<figure markdown>
+ { align=left width="500" }
+ <figcaption>AGIPD PDUs [AGIPD Nr. 315 = AGIPD_SIV1_AGIPDV11_M315]</figcaption>
+</figure>
+
+<figure markdown>
+ { align=left width="500" }
+ <figcaption>JUNGFRAU PDU [Jungfrau Nr. 035 = Jungfrau_M035]</figcaption>
+</figure>
+
+
+PDUs can be transferred between detectors and instruments or replaced. To correct raw data, the calibration pipeline produces calibration constants, which are specific to the module they were generated from. As a result, it's crucial to map the hardware to the corresponding software modules.
+
+As the detector is
+
+
+### Modifying detector mapping
+
+The detector experts are the most up-to date with the PDU changes. They are able to update and track the PDU names to software module using CALCAT.
+
+Updating the mapping immediately after making any hardware modifications is crucial before acquiring RAW data for calibration. This is because during offline calibration, the timestamp of the acquired RAW data serves as a reference point to retrieve the correct mapping from the database.
+
+### History of Detector mapping
+
+{ align=right width=200 height=350}
+
+
+In the beginning, detector mapping was designed to be hard coded in [iCalibrationDB](https://git.xfel.eu/detectors/cal_db_interactive). It is an interface package to [CalibrationDBRemote](https://git.xfel.eu/karaboDevices/calibrationDBRemote) and CALCAT.
+
+This screenshot shows the last updated hard coded physical detector units for AGIPD detector at SPB.
+
+The of course led with error-prone to not be updated in time after there were changes reported when PDUs where replaced or moved.
+
+## Database snapshot
+
+A database snapshot is a point-in-time copy of a database that provides a read-only, consistent view of the data as it existed at the time the snapshot was created. It's a static picture of the database and includes all the data that was in the database at the time of the snapshot's creation. Database snapshots are useful for performing data analysis or creating reports without interfering with ongoing database transactions. They can also be used for backup and recovery purposes or to provide a consistent view of the data for testing and development purposes.
+
+CALCAT snapshot feature is used for obtaining the correct detector mapping. This is done by using the RAW data timestamp (creation_time) as the snapshot timestamp.
+
+This is why it is crucial to update the detector mapping immediately after any hardware changes.
\ No newline at end of file
diff --git a/docs/configuration.md b/docs/configuration.md
new file mode 100644
index 0000000000000000000000000000000000000000..51b8dcf52b45a82efe601f7ec5385b370765b1ea
--- /dev/null
+++ b/docs/configuration.md
@@ -0,0 +1,104 @@
+Configuration
+=============
+
+The European XFEL Offline Calibration is configured through
+[settings.py]{.title-ref} and [notebooks.py]{.title-ref} files. Both can
+be found in [xfel\_calibrate]{.title-ref} source directory. The
+[notebook.py]{.title-ref} file exposes and configures the notebooks
+by the [xfel-calibrate]{.title-ref} command line interface.
+
+Settings
+--------
+
+The [settings.py]{.title-ref} is a python configuration file, which
+configures the tool\'s environment.:
+
+ # path into which temporary files from each run are placed
+ temp_path = "{}/temp/".format(os.getcwd())
+
+ # Path to use for calling Python. If the environment is correctly set, simply the command
+ python_path = "python"
+
+ # Path to store reports in
+ report_path = "{}/calibration_reports/".format(os.getcwd())
+
+ # Also try to output the report to an out_folder defined by the notebook
+ try_report_to_output = True
+
+ # the command to run this concurrently. It is prepended to the actual call
+ launcher_command = "sbatch -p exfel -t 24:00:00 --mem 500G --mail-type END --requeue --output {temp_path}/slurm-%j.out"
+
+A comment is given for the meaning of each configuration parameter.
+
+Notebooks
+---------
+
+The [xfel-calibrate]{.title-ref} tool exposes configured notebooks to
+the command line by automatically parsing the parameters given in the
+notebooks first cell. The configuration is given in the form of a python
+dictionary:
+
+ notebooks = {
+ "AGIPD": {
+ "DARK": {
+ "notebook": "AGIPD/Characterize_AGIPD_Gain_Darks_NBC.ipynb",
+ "concurrency": {"parameter": "modules",
+ "default concurrency": 16,
+ "cluster cores": 16},
+ },
+ "PC": {
+ "notebook": "AGIPD/Chracterize_AGIPD_Gain_PC_NBC.ipynb",
+ "concurrency": "parameter": "modules",
+ "default concurrency": 16,
+ "cluster cores": 16},
+ },
+ "CORRECT": {
+ "notebook": "notebooks/AGIPD/AGIPD_Correct_and_Verify.ipynb",
+ "concurrency": {"parameter": "sequences",
+ "use function": "balance_sequences",
+ "default concurrency": [-1],
+ "cluster cores": 32},
+ ...
+ }
+ }
+
+The first key is the detector, e.g. AGIPD. The second key is the
+calibration type name, e.g. DARK or PC. A dictionary is expected for
+each calibration type with a notebook path and concurrency
+configuration. For the concurrency three values are expected. Key
+[parameter]{.title-ref} with a value name of type list, which is defined
+in the first notebook cell. The key [default concurrency]{.title-ref} to
+define the range of values for [parameter]{.title-ref} in each
+concurrent notebook, if it is not defined by the user. e.g. [\"default
+concurrency\": 16]{.title-ref} leads to running 16 concurrent jobs, each
+processing one module with values of \[0,1,2,\...,15\]. Finally, a hint
+for the number of [cluster cores]{.title-ref} that is used if the
+notebook is using ipcluster parallelization, only. This value should be
+derived e.g. by profiling memory usage per core, run times, etc.
+
+ {.note}
+ {.admonition-title}
+Note
+
+
+It is good practice to name command line enabled notebooks with an
+[\_NBC]{.title-ref} suffix as shown in the above example.
+
+
+The AGIPD [CORRECT]{.title-ref} notebook (last notebook in the example)
+makes use of a concurrency generating function by setting the [use
+function]{.title-ref} parameter. This function must be defined in the
+first cell in the notebook its given arguments should be named as the
+first cell notebook parameters. It is expected to return a list of
+parameters to concurrently run notebooks. Above the used function is
+`xfel_calibrate.calibrate.balance_sequences`{.interpreted-text
+role="func"}.
+
+ {.note}
+ {.admonition-title}
+Note
+
+
+The function only needs to be defined, but not executed within the
+notebook context itself.
+
diff --git a/docs/css/custom.css b/docs/css/custom.css
new file mode 100644
index 0000000000000000000000000000000000000000..2a2f37f47f3aa6a3008c2ec8bd53f25e10578671
--- /dev/null
+++ b/docs/css/custom.css
@@ -0,0 +1,87 @@
+div.autodoc-docstring {
+ padding-left: 20px;
+ margin-bottom: 30px;
+ border-left: 5px solid rgba(230, 230, 230);
+ }
+
+ div.autodoc-members {
+ padding-left: 20px;
+ margin-bottom: 15px;
+ }
+
+
+/* :root > * {
+ --md-primary-fg-color: #152066;
+ --md-primary-fg-color--light:#3c3b72;
+ --md-primary-fg-color--dark: #000020;
+ --md-primary-bg-color: #ffffff;
+ --md-primary-bg-color--light:#B2B2B2;
+ --md-footer-bg-color: #000020;
+
+ --md-accent-fg-color: #f39200;
+ --md-accent-fg-color--transparent: #f3920085;
+ --md-accent-bg-color: #ffffff;
+ --md-accent-bg-color--light: #ffffff;
+} */
+
+[data-md-color-scheme="light"] {
+
+ /* // Default color shades */
+ --md-primary-fg-color: #152066;
+ --md-primary-fg-color--light:#3c3b72;
+ --md-primary-fg-color--dark: #000020;
+ --md-primary-bg-color: #ffffff;
+ --md-primary-bg-color--light:#B2B2B2;
+ --md-footer-bg-color: #000020;
+
+ --md-accent-fg-color: #f39200;
+ --md-accent-fg-color--transparent: #f3920085;
+ --md-accent-bg-color: #ffffff;
+ --md-accent-bg-color--light: #ffffff;
+
+ --md-typeset-a-color: #2840dd;
+
+}
+
+[data-md-color-scheme="slate"] {
+
+ --md-primary-fg-color: #f39200;
+ --md-primary-fg-color--light:#f3920085;
+ --md-primary-fg-color--dark: #da996f;
+ --md-primary-bg-color: #ffffff;
+ --md-primary-bg-color--light:#B2B2B2;
+ --md-footer-bg-color: #000020;
+
+ --md-accent-fg-color: #152066;
+ --md-accent-fg-color--transparent: #3c3b72;
+ --md-accent-bg-color: #ffffff;
+ --md-accent-bg-color--light: #ffffff;
+
+ /* // Default color shades */
+ --md-default-fg-color: hsla(0, 0%, 100%, 1);
+ --md-default-fg-color--light: hsla(0, 0%, 100%, 0.87);
+ --md-default-fg-color--lighter: hsla(0, 0%, 100%, 0.32);
+ --md-default-fg-color--lightest: hsla(0, 0%, 100%, 0.12);
+ --md-default-bg-color: hsla(232, 15%, 21%, 1);
+ --md-default-bg-color--light: hsla(232, 15%, 21%, 0.54);
+ --md-default-bg-color--lighter: hsla(232, 15%, 21%, 0.26);
+ --md-default-bg-color--lightest: hsla(232, 15%, 21%, 0.07);
+
+ /* // Code color shades */
+ --md-code-bg-color: hsla(232, 15%, 18%, 1);
+ --md-code-fg-color: hsla(60, 30%, 96%, 1);
+
+ /* // Text color shades */
+ --md-text-color: var(--md-default-fg-color--light);
+ --md-text-link-color: var(--md-primary-fg-color);
+
+ /* // Admonition color shades */
+ --md-admonition-bg-color: hsla(0, 0%, 100%, 0.025);
+ --md-admonition-fg-color: var(--md-default-fg-color);
+
+ /* // Footer color shades */
+ --md-footer-bg-color: hsla(230, 9%, 13%, 0.87);
+ --md-footer-bg-color--dark: hsla(232, 15%, 10%, 1);
+
+ --md-typeset-a-color: #e99a25;
+}
\ No newline at end of file
diff --git a/docs/how_to_write_xfel_calibrate_notebook_NBC.md b/docs/how_to_write_xfel_calibrate_notebook_NBC.md
new file mode 100644
index 0000000000000000000000000000000000000000..39766789fd61945882306661ad6bdecd77dff4bb
--- /dev/null
+++ b/docs/how_to_write_xfel_calibrate_notebook_NBC.md
@@ -0,0 +1,257 @@
+# How to start writing new calibration notebook
+
+Author: European XFEL Detector Group, Version 0.1
+
+This is an example notebook to point to some common practices used in
+production notebooks. This notebook is using ePix100 detector RAW data
+to apply offset and gain correction.
+
+This is meant to be a starting point on how to write calibration
+notebooks that can run in production using `xfel-calibrate` CLI.
+However, it is recommended to have a look on some production notebooks
+residing in `/notebooks/` directory which can have more advanced
+practices that can help you during your notebook development.
+
+```python
+# This first code cell must always contain the global notebook parameters.
+# The parameters are parsed as input arguments for `xfel-calibration` command line interface.
+# It is very important to have a comment for each parameter. The comments are not only helpful within the notebook,
+# but they are used the as parameter description when `xfel-calibrate DETECTOR CALIBRATION --help` is used.
+
+in_folder = "/gpfs/exfel/exp/CALLAB/202130/p900203/raw/" # directory to read data from, required
+out_folder = "/gpfs/exfel/exp/CALLAB/202130/p900203/scratch/how_write_xfel_calibrate_NBC" # directory to output to, required
+# Adding `required` at the comment here forces the user to add the parameter through the command line,
+# ignoring the default value and only using it as an indication of the expected type.
+run = 9046 # runs to process, required
+
+# Parameters for accessing the raw data.
+karabo_da = "EPIX01" # Data aggregator names. For multi-modular detectors like AGIPD and LPD, this is a list.
+# To access the correct data files and calibration constants. The karabo_id string is used as a detector identifier.
+karabo_id = "HED_IA1_EPX100-1" # Detector karabo_id name
+
+# Boolean parameter can be set to False from xfel-calibrate by adding `no-` at the beginning of the boolean parameter name.
+gain_correction = True # Proceed with gain correction.
+
+# Parameters for the calibration database.
+creation_time = "" # The timestamp to use with Calibration DB. Required Format: "YYYY-MM-DD hh:mm:ss" e.g. 2019-07-04 11:02:41
+# It is preferred if operating conditions are read from RAW data instead of passed as an input argument.
+bias_voltage = 200 # RAW data bias voltage.
+in_vacuum = False # Detector operated in vacuum
+photon_energy = 8.048 # Photon energy used for gain calibration
+fix_temperature = 290 # fixed temperature value in Kelvin.
+
+# Parameters affecting writing corrected data.
+chunk_size_idim = 1 # H5 chunking size of output data
+```
+
+```python
+from pathlib import Path
+
+# Write after the first python notebook cell. It is a good practice to import all needed libraries and modules.
+# Same as we do in a normal python module.
+import matplotlib.pyplot as plt
+
+import numpy as np
+
+# To access data `extra_data` is used to read RAW/CORR data.
+from extra_data import RunDirectory # https://extra-data.readthedocs.io/en/latest/
+
+from extra_geom import Epix100Geometry # https://extra-geom.readthedocs.io/en/latest/
+
+# For parallelization with a notebook it is suggested to use multiprocessing.
+import multiprocessing # or
+import pasha as psh # https://github.com/European-XFEL/pasha
+# This library uses multiprocessing and provide tight integration with extra_data
+
+# `cal_tools` directory consists of multiple useful functions that are used in many notebooks.
+import cal_tools.restful_config as rest_cfg
+# `calcat_interface` is the main module with functions to retrieve calibration constants from CALCAT.
+from cal_tools.calcat_interface import EPIX100_CalibrationData
+from cal_tools.epix100 import epix100lib
+# `cal_tools.files` is recommended to write corrected data.
+from cal_tools.files import DataFile
+# An internal class to record computation time.
+from cal_tools.step_timing import StepTimer
+# `tools` consists for various number of functions to read files, wrappers for iCalibrationDB, etc ...
+from cal_tools.tools import (
+ calcat_creation_time,
+)
+```
+
+Prepare global variables.
+-------------------------
+
+In the following cells it is a common practice to start assigning global
+variables, like converting in\_folder and out\_folder to Path objects or
+initializing step\_timer object.
+
+```python
+# Convert main folders to Paths.
+in_folder = Path(in_folder)
+out_folder = Path(out_folder)
+# This is only needed in case of running the notebook interactively. Otherwise, the machinery take care of this.
+out_folder.mkdir(parents=True, exist_ok=True)
+run_folder = in_folder / f"r{run:04d}"
+
+# Initiate the main Run data collection.
+run_dc = RunDirectory(
+ run_folder, include="*S00000*").select(f"*{karabo_id}*", require_all=True)
+
+print(f"The available source to correct for {karabo_id} are {list(run_dc.all_sources)}")
+
+step_timer = StepTimer()
+```
+
+```python
+The available source to correct for HED_IA1_EPX100-1 are ['HED_IA1_EPX100-1/DET/RECEIVER', 'HED_IA1_EPX100-1/DET/RECEIVER:daqOutput',
+'HED_IA1_EPX100-1/DET/CONTROL']
+```
+Read operating conditions from RAW data.
+----------------------------------------
+
+It is recommended to read the calibration constants\' operating
+conditions directly from RAW data. To avoid wrong given values from the
+notebook\'s input argument. Unfortunately, there is the possibility that
+these conditions are not stored in RAW data because the detector is in
+its early operation stages.
+
+Below we give an example of reading the integration time of the data.
+There are multiple functions and similar class as epix100Ctrl for other
+detectors that are used for the same purpose.
+
+```python
+# Read control data.
+data_source = "HED_IA1_EPX100-1/DET/RECEIVER:daqOutput"
+
+ctrl_data = epix100lib.epix100Ctrl(
+ run_dc=run_dc,
+ instrument_src=data_source,
+ ctrl_src=f"{karabo_id}/DET/CONTROL",
+ )
+
+integration_time = ctrl_data.get_integration_time()
+```
+
+Retrieve needed calibration constants
+-------------------------------------
+
+Usually there is a cell when we retrieve calibration constants before
+correction and sometimes before processing new calibration constants.
+
+In this example we use `EPIX100_CalibrationData` class to initialize an
+object with the necessary operating conditions and creation time.
+
+Below the operating conditions values like integration\_time and
+sensor\_temperature are hard coded to specific value. In production
+notebooks this is done differently.
+
+```python
+# Run creation time is important to get the correct calibration constant versions.
+creation_time = calcat_creation_time(in_folder, run, creation_time)
+print(f"Using {creation_time.isoformat()} as creation time")
+
+epix_cal = EPIX100_CalibrationData(
+ detector_name=karabo_id,
+ sensor_bias_voltage=bias_voltage,
+ integration_time=integration_time,
+ sensor_temperature=fix_temperature,
+ in_vacuum=in_vacuum,
+ source_energy=photon_energy,
+ event_at=creation_time,
+ client=rest_cfg.calibration_client(),
+)
+
+const_data = epix_cal.ndarray_map()[karabo_da]
+
+print(f"Retrieved calibrations for {karabo_id}: {list(const_data.keys())}")
+```
+
+```python
+Reading creation_date from input files metadata [INDEX/timestamp] Using 2021-09-19T14:39:26.744069+00:00 as
+creation time Retrieved calibrations for HED_IA1_EPX100-1:
+['BadPixelsDarkEPix100', 'NoiseEPix100', 'OffsetEPix100', 'RelativeGainEPix100']
+```
+Correcting Raw data.
+--------------------
+
+```python
+data_key = "data.image.pixels"
+raw_data = run_dc[data_source, data_key].ndarray()
+dshape = raw_data.shape # Raw input data shape.
+
+print(f"Number of trains to correct is {len(run_dc.train_ids)}")
+
+def correct_train(wid, index, d):
+ """Correct one train for ePix100 detector."""
+ d -= const_data["OffsetEPix100"][..., 0]
+ if gain_correction:
+ d /= const_data["RelativeGainEPix100"]
+
+step_timer.start()
+context = psh.context.ThreadContext(num_workers=10)
+corr_data = context.alloc(shape=dshape, dtype=np.float32)
+
+context.map(correct_train, raw_data.astype(np.float32))
+step_timer.done_step('Correcting data')
+```
+
+```python
+Number of trains to correct is 1000 Correcting data: 1.0 s
+```
+
+Writing corrected data.
+-----------------------
+
+```python
+# Storing data.
+out_file = out_folder / "CORR-R9046-EPIX01-S00000.h5"
+instrument_source = "HED_IA1_EPX100-1/DET/RECEIVER:daqOutput"
+
+image_counts = run_dc[instrument_source, "data.image.pixels"].data_counts(labelled=False)
+
+step_timer.start()
+
+with DataFile(out_file, "w") as ofile:
+ # Create INDEX datasets.
+ ofile.create_index(run_dc.train_ids, from_file=run_dc.files[0])
+
+ # Create METDATA datasets
+ ofile.create_metadata(
+ like=run_dc,
+ sequence=run_dc.run_metadata()["sequenceNumber"],
+ instrument_channels=(f"{instrument_source}/data",)
+ )
+ # Create Instrument section to later add corrected datasets.
+ outp_source = ofile.create_instrument_source(instrument_source)
+
+ # Create count/first datasets at INDEX source.
+ outp_source.create_index(data=image_counts)
+
+ # Add main corrected `data.image.pixels` dataset and store corrected data.
+ outp_source.create_key(
+ "data.image.pixels", data=corr_data, chunks=((chunk_size_idim,) + dshape[1:]))
+step_timer.done_step('Writing corrected data')
+```
+
+```python
+Writing corrected data: 1.2 s
+```
+
+Plotting results
+================
+
+```python
+geom = Epix100Geometry.from_origin()
+
+fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(25, 10))
+
+# Plotting mean data for RAW and CORRECTED across trains
+geom.plot_data(np.mean(raw_data, axis=0), ax=ax1)
+ax1.set_title("Mean RAW across trains")
+geom.plot_data(np.mean(corr_data, axis=0), ax=ax2)
+ax2.set_title("Mean CORR across trains")
+
+plt.show()
+```
+
+
diff --git a/docs/index.md b/docs/index.md
new file mode 100644
index 0000000000000000000000000000000000000000..40916c2f766b284aec94973c5c2257475061d542
--- /dev/null
+++ b/docs/index.md
@@ -0,0 +1,22 @@
+European XFEL Offline Calibration
+=================================
+
+The European XFEL Offline Calibration (pyCalibration) is a python
+package that consists of different services, responsible for applying
+most of the offline calibration and characterization for the detectors.
+
+
+
+Documentation contents:
+
+- [Installation](installation.md) How to install pycalibration
+
+- [Configuration](configuration.md) xfel-calibrate CLI configuration
+
+- [Development](workflow.md) How to start developing offline calibration notebooks
+
+- [How to write a notebook](how_to_write_xfel_calibrate_notebook_NBC.md) A guide on how to arrange notebook's cells.
+
+- [available_notebooks](available_notebooks.md) The current available production notebooks.
+
+- [calibration_database](calibration_database.md) The calibration database.
\ No newline at end of file
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 0000000000000000000000000000000000000000..9d533db32799f8b05531eb89f6a21f81a5d3f574
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,100 @@
+# Installation
+
+It's recommended to install the offline calibration (pycalibration)
+package on maxwell, using the anaconda/3 environment.
+
+The following instructions clone from the EuXFEL GitLab instance using
+SSH remote URLs, this assumes that you have set up SSH keys for use with
+GitLab already. If you have not then read the appendix section on [SSH
+Key Setup for GitLab](#ssh-key-setup-for-gitlab) for instructions on how
+to do this.
+
+## Installation using python virtual environment - recommended
+
+[pycalibration] uses the same version of Python as Karabo,
+which in June 2021 updated to use Python 3.8. Currently, the default
+python installation on Maxwell is still Python 3.6.8, so Python 3.8
+needs to be loaded from a different location.
+
+One option is to use the Maxwell Spack installation, running [module
+load maxwell] will activate the test Spack instance from
+DESY, then you can use [module load
+python-3.8.6-gcc-10.2.0-622qtxd] to Python 3.8. Note that
+this Spack instance is currently a trial phase and may not be stable.
+
+Another option is to use [pyenv], we provide a pyenv
+installation at [/gpfs/exfel/sw/calsoft/.pyenv] which we use
+to manage different versions of python. This can be activated with
+`source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate`
+
+A quick setup would be:
+
+1. `source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate`
+2. `git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration` -
+ clone the offline calibration package from EuXFEL GitLab
+3. `pyenv shell 3.8.11` - load required version of python
+4. `python3 -m venv .venv` - create the virtual environment
+5. `source .venv/bin/activate` - activate the virtual environment
+6. `python3 -m pip install --upgrade pip` - upgrade version of pip
+7. `python3 -m pip install .` - install the pycalibration package (add
+ `-e` flag for editable development installation)
+
+Copy/paste script:
+
+```bash
+source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
+git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
+cd pycalibration
+pyenv shell 3.8.11
+python3 -m venv .venv
+source .venv/bin/activate
+python3 -m pip install --upgrade pip
+python3 -m pip install . # `-e` flag for editable install, e.g. `pip install -e .`
+```
+
+## Creating an ipython kernel for virtual environments
+
+To create an ipython kernel with pycalibration available you should (if
+using a venv) activate the virtual environment first, and then run:
+
+```bash
+python3 -m pip install ipykernel # If not using a venv add `--user` flag
+python3 -m ipykernel install --user --name pycalibration --display-name "pycalibration" # If not using a venv pick different name
+```
+
+This can be useful for Jupyter notebook tools as
+<https://max-jhub.desy.de/hub/login>
+
+## SSH Key Setup for GitLab
+
+It is highly recommended to setup SSH keys for access to GitLab as this
+simplifies the setup process for all of our internal software present on
+GitLab.
+
+To set up the keys:
+
+1. Connect to Maxwell
+2. Generate a new keypair with `ssh-keygen -o -a 100 -t ed25519`, you
+ can either leave this in the default location (`~/.ssh/id_ed25519`)
+ or place it into a separate directory to make management of keys
+ easier if you already have multiple ones. If you are using a
+ password for your keys please check this page to learn how to manage
+ them:
+ <https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent>
+3. Add the public key (`id_ed25519.pub`) to your account on GitLab:
+ <https://git.xfel.eu/gitlab/profile/keys>
+4. Add the following to your `~/.ssh/config` file
+
+```bash
+# Special flags for gitlab over SSH
+Host git.xfel.eu
+ User git
+ Port 10022
+ ForwardX11 no
+ IdentityFile ~/.ssh/id_ed25519
+```
+
+Once this is done you can clone repositories you have access to from
+GitLab without having to enter your password each time. As
+`pycalibration` requirements are installed from SSH remote URLs having
+SSH keys set up is a requirement for installing pycalibration.
diff --git a/docs/myMDC.md b/docs/myMDC.md
new file mode 100644
index 0000000000000000000000000000000000000000..7eb3a8be15bfd141a66b6b41389079ec7ffaca75
--- /dev/null
+++ b/docs/myMDC.md
@@ -0,0 +1,37 @@
+# Metadata Catalouge
+
+
+European XFEL Data Portal is provided by myMdC, allowing users to access the metadata of scientific data taken at the European XFEL facility. Datasets are linked to experiment proposals, annotated by data collection types, samples and are assigned DOIs to be referenced in publications. myMdC helps data management activities and allows users to find open and own proposals data based on the metadata.
+
+Through myMDC users can request to correct migrated runs. Beside correction myMDC is working as the main entry to generate dark calibration constants for detector data.
+
+myMDC stands for Metadata Catalog. It arranges the facility's data into proposals. Each proposal consists of a number of acquired runs. Offline calibration only works with runs set to Good `Run Quality`, which means it was migrated from the online cluster to `gpfs`.
+
+
+## Proposals
+
+Proposals are created for commissioning or user experiments. There can be a proposal per instrument. Every proposal has a number of team members who have access to interact with the proposal's runs.
+
+proposal's RAW data location can be found at: `/gpfs/exfel/exp/<instrument>/<cycle>/<proposal>/raw`
+
+## Runs
+
+Runs have multiple properties to modify through myMDC.
+
+The most relevant properties are:
+
+- Run Quality: Currently only migrated runs are allowed to be offline calibrated.
+- Run Types: A run type can be relevant in accepting a correction request or not. e.g. Correcting dark runs can be skipped on purpose.
+
+
+
+## Offline correction
+
+
+## Generating calibration constant
+
+{ align=right width=200 height=350}
+
+
+Currently, only dark calibration constants generation can be triggered through myMDC.
+
diff --git a/docs/operation.md b/docs/operation.md
new file mode 100644
index 0000000000000000000000000000000000000000..ddf0c5b7eb05909fbae06f31b0ffe81948d1b54b
--- /dev/null
+++ b/docs/operation.md
@@ -0,0 +1 @@
+# Operation page
\ No newline at end of file
diff --git a/docs/troubleshooting_calibration_generation.md b/docs/troubleshooting_calibration_generation.md
new file mode 100644
index 0000000000000000000000000000000000000000..b4e5c58c3e4c12fb698773653afa021f2bfbe992
--- /dev/null
+++ b/docs/troubleshooting_calibration_generation.md
@@ -0,0 +1 @@
+# Troubleshooting Calibration constants generation
\ No newline at end of file
diff --git a/docs/troubleshooting_correction.md b/docs/troubleshooting_correction.md
new file mode 100644
index 0000000000000000000000000000000000000000..6d12aacb313b0962432e2887a686f2463d831a0c
--- /dev/null
+++ b/docs/troubleshooting_correction.md
@@ -0,0 +1 @@
+# Troubleshooting correction
\ No newline at end of file
diff --git a/docs/workflow.md b/docs/workflow.md
new file mode 100644
index 0000000000000000000000000000000000000000..bbe1d0a8d83b23ed2a1c264fb5a65b28b7e0b272
--- /dev/null
+++ b/docs/workflow.md
@@ -0,0 +1,323 @@
+# Development Workflow
+
+Contributing is encouraged if you have calibration notebooks that you would like to add to the pipeline
+or if you have a specific algorithm which is beneficial for the pipeline.
+
+To guide you through this process, this section collects the main points to consider during development.
+These points should help in both the development and review process of the new feature.
+
+
+## Developing a notebook from scratch
+
+When you are starting a blank notebook from scratch you should first
+answer few questions:
+
+* How will the notebook be executed:
+
+ - Is it executed interactively by the user?
+ - Should it execute the same as the production notebooks through xfel-calibrate CLI.
+ - Is it a headless task or a report is needed at the end to see the results.
+
+* Do you need to run the notebook's processing concurrently?
+
+ - Is internal concurrency using multiprocessing sufficient, or processing using host level
+ cluster computing via SLURM is essential for higher performance.
+
+If `xfel-calibrate` CLI is essential, you need to follow the guidelines in where and how to
+write the variables in the first notebook cell and how to include it as one of the CLI
+calibration options to execute.
+
+If a report is needed you should make sure to provide sufficient guidance and textual details
+using markdown cells and clear prints within the code. You should also structure the notebook cells
+it into appropriate subsections.
+
+If SLURM concurrency is needed, you need to identify the variable that the notebook will be
+replicated based on to split the processing.
+
+Once you've clarified the above points, you should create a new notebook,
+either in an existing detector folder or into a new folder with the new detector's name.
+Give it a suffix `_NBC` to denote that it is enabled for the tool chain.
+
+You should then start writing your code following the guidelines
+at [How to start writing calibration notebook](how_to_write_xfel_calibrate_notebook_NBC.md)
+
+## Title and Author Information in `markdown` [First Cell]
+
+Especially for report generation the notebook should have a proper title
+author and version. These should be given in a leading `markdown` cell in
+the form::
+
+ # My Fancy Calculation
+
+ Author](Jane Doe, Version 0.1
+
+ A description of the notebook.
+
+Information in this format allows automatic parsing of author and version.
+
+
+## Exposing Parameters to the `xfel-calibrate` Command Line [SECOND CELL]
+
+The European XFEL Offline Calibration toolkit automatically deduces
+command line arguments from Jupyter notebooks. It does this with an
+extended version of nbparameterise_, originally written by Thomas
+Kluyver.
+
+Parameter deduction tries to parse all variables defined in the first
+code cell of a notebook. The following variable types are supported:
+
+* Numbers](INT or FLOAT
+* Booleans
+* Strings
+* Lists of any of the above
+
+You should avoid having `import` statements in this cell. Line comments
+can be used to define the help text provided by the command line interface,
+and to signify if lists can be constructed from ranges and if parameters are
+required::
+
+ in_folder = "" # directory to read data from, required
+ out_folder = "" # directory to output to, required
+ metadata_folder = "" # directory containing calibration metadata file when run by xfel-calibrate
+ run = [820, ] # runs to use, required, range allowed
+ sequences = [0, 1, 2, 3, 4] # sequences files to use, range allowed
+ modules = [0] # modules to work on, required, range allowed
+
+ karabo_id = "MID_DET_AGIPD1M-1" # Detector karabo_id name
+ karabo_da = [""] # a list of data aggregators names, Default [-1] for selecting all data aggregators
+
+ skip-plots = False # exit after writing corrected files and metadata
+
+The above are some examples of parameters from AGIPD correction notebook.
+
+- Here, `in_folder` and `out_folder` are set as `required` string values.
+
+Values for required parameters have to be given when executing from the command line.
+This means that any defaults given in the first cell of the code are ignored
+(they are only used to derive the type of the parameter).
+
+- `modules` and `sequences` are lists of integers, which from the command line could also be assigned using a range expression,
+e.g. `5-10,12,13,18-21`, which would translate to `5,6,7,8,9,12,13,18,19,20`.
+
+.. note::
+
+ nbparameterise_ can only parse the mentioned subset of variable types. An expression
+ that evaluates to such a type will not be recognized](e.g. `a = list(range(3))` will
+ not work!
+
+- `karabo_id` is a string value indicating the detector to be processed.
+- `karabo_da` is a list of strings to indicate the detector's modules to be processed. As `karabo_da` and `modules`
+ are two different variables pointing to the same physical parameter. In the later notebook cells both parameters are synced
+ before usage.
+
+- `skip-plots` is a boolean for skipping the notebook plots to save time and deliver the report as soon as the data are processed.
+ to set `skip-plots` to False from the command line. `--no-skip-plots` is used.
+
+The code would lead to the following parameters being exposed via the command line::
+
+ % xfel-calibrate AGIPD FF --help
+ usage](xfel-calibrate [-h] [--no-cluster-job] [--prepare-only] [--report-to str] [--not-reproducible] [--skip-report] [--skip-env-freeze] [--concurrency-par str]
+ [--constants-from str] [--vector-figs] [--slurm-mem int] [--slurm-name str] [--slurm-scheduling int] [--request-time str]
+ [--slurm-partition str] [--reservation str] --in-folder str --out-folder str [--sequences str [str ...]] [--modules str [str ...]]
+ [--train-ids str [str ...]] --run int [--karabo-id str] [--karabo-da str [str ...]] [--receiver-template str] [--path-template str]
+ [--instrument-source-template str] [--index-source-template str] [--ctrl-source-template str] [--karabo-id-control str]
+ [--slopes-ff-from-files str] [--use-dir-creation-date | --no-use-dir-creation-date] [--cal-db-interface str] [--cal-db-timeout int]
+ [--creation-date-offset str] [--mem-cells int] [--bias-voltage int] [--acq-rate float] [--gain-setting int] [--gain-mode int]
+ [--overwrite | --no-overwrite] [--max-pulses int [int ...]] [--mem-cells-db int] [--integration-time int] [--blc-noise-threshold int]
+ [--cm-dark-fraction float] [--cm-dark-range float [float ...]] [--cm-n-itr int] [--hg-hard-threshold int] [--mg-hard-threshold int]
+ [--noisy-adc-threshold float] [--ff-gain float] [--photon-energy float] [--only-offset | --no-only-offset] [--rel-gain | --no-rel-gain]
+ [--xray-gain | --no-xray-gain] [--blc-noise | --no-blc-noise] [--blc-stripes | --no-blc-stripes] [--blc-hmatch | --no-blc-hmatch]
+ [--match-asics | --no-match-asics] [--adjust-mg-baseline | --no-adjust-mg-baseline] [--zero-nans | --no-zero-nans]
+ [--zero-orange | --no-zero-orange] [--blc-set-min | --no-blc-set-min] [--corr-asic-diag | --no-corr-asic-diag]
+ [--force-hg-if-below | --no-force-hg-if-below] [--force-mg-if-below | --no-force-mg-if-below] [--mask-noisy-adc | --no-mask-noisy-adc]
+ [--common-mode | --no-common-mode] [--melt-snow | --no-melt-snow] [--mask-zero-std | --no-mask-zero-std]
+ [--low-medium-gap | --no-low-medium-gap] [--round-photons | --no-round-photons] [--use-ppu-device str] [--ppu-train-offset int]
+ [--use-litframe-finder str] [--litframe-device-id str] [--energy-threshold int] [--use-super-selection str] [--use-xgm-device str]
+ [--recast-image-data str] [--compress-fields str [str ...]] [--skip-plots | --no-skip-plots] [--cell-id-preview int] [--chunk-size int]
+ [--n-cores-correct int] [--n-cores-files int] [--sequences-per-node int] [--max-nodes int] [--max-tasks-per-worker int]
+ DETECTOR TYPE
+
+ # AGIPD Offline Correction #
+ ,
+ Author](European XFEL Detector Group, Version](2.0,
+ ,
+ Offline Calibration for the AGIPD Detector
+
+ positional arguments:
+ DETECTOR The detector to calibrate](AGIPD, LPD, PNCCD, GENERIC, TUTORIAL, FASTCCD, JUNGFRAU, GOTTHARD2, EPIX100, EPIX10K, DSSC, REMI, TEST, TEST-
+ RAISES-ERRORS
+ TYPE Type of calibration.
+
+ optional arguments:
+ -h, --help show this help message and exit
+ --sequences str [str ...]
+ sequences to correct, set to -1 for all, range allowed. Default]([-1]
+ --modules str [str ...]
+ modules to correct, set to -1 for all, range allowed. Default]([-1]
+ --train-ids str [str ...]
+ train IDs to correct, set to -1 for all, range allowed. Default]([-1]
+ --karabo-id str karabo karabo_id. Default](MID_DET_AGIPD1M-1
+ --karabo-da str [str ...]
+ a list of data aggregators names, Default [-1] for selecting all data aggregators. Default](['-1']
+ --karabo-id-control str
+ karabo-id for control device. Default](MID_EXP_AGIPD1M1
+ ...
+
+ required arguments:
+ --in-folder str the folder to read data from, required. Default](None
+ --out-folder str the folder to output to, required. Default](None
+ --run int runs to process, required. Default](None
+
+
+The following table contains a list of suggested names for certain parameters, allowing
+to stay consistent amongst all notebooks.
+
+
+| Parameter name | To be used for | Special purpose |
+| ----------- | --------------------------------------------------------------------- |----------------------------|
+| `in_folder` | the input path data resides in, usually without a run number. ||
+| `out_folder` | path to write data out to, usually without a run number. | reports can be placed here |
+| `metadata_folder` | directory path for calibration metadata file with local constants. ||
+| `run(s)` | which XFEL DAQ runs to use, often ranges are allowed. ||
+| `karabo_id` | detector karabo name to access detector files and constants. ||
+| `karabo_da` | refers to detector's modules data aggregator names to process. ||
+| `modules` | refers to the detector's modules indices to process, ranges often ok. ||
+| `sequences` | sequence files for the XFEL DAQ system, ranges are often ok. ||
+| `local_output` | write calibration constant from file, not database. ||
+| `db_output` | write calibration constant from database, not file. |saves the database from unintentional constant |
+| `injections` | developments or testing. | |
+
+
+
+## Best Coding Practices
+
+In principle there a not restrictions other than that parameters that are exposed to the
+command line need to be defined in the first code cell of the notebook.
+
+However, a few guidelines should be observed to make notebook useful for display as
+reports and usage by others.
+
+### External Libraries
+
+You may use a wide variety of libraries available in Python, but keep in mind that others
+wanting to run the tool will need to install these requirements as well. Thus,
+
+* Do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
+ be created using matplotlib_ and numerical processing should be done in numpy_.
+
+* Keep runtime and library requirements in mind. A library doing its own parallelism either
+ needs to programmatically be able to set this up, or automatically do so. If you need to
+ start something from the command line first, things might be tricky as you will likely
+ need to run this via `POpen` commands with appropriate environment variable.
+
+* Reading out RAW data should be done using extra_data_. It helps in accessing the HDF5 data
+ structures efficiently. It reduces the complexity of accessing the RAW or CORRECTED datasets,
+ and it provides different methods to select and filter the trains, cells, or pixels of interest.
+
+### Writing out data
+
+
+If your notebook produces output data, consider writing data out as early as possible,
+such that it is available as soon as possible. Detailed plotting and inspection can
+be done later on in the notebook.
+
+Also use HDF5 via h5py_ as your output format. If you correct or calibrate
+input data, which adheres to the XFEL naming convention, you should maintain the convention
+in your output data. You should not touch any data that you do not actively work on and
+should assure that the `INDEX` and identifier entries are synchronized with respect to
+your output data. E.g. if you remove pulses from a train, the `INDEX/.../count` section
+should reflect this.
+
+
+### Plotting
+
+When creating plots, make sure that the plot is either self-explanatory or add markdown
+comments with adequate description. Do not add "free-floating" plots, always put them into
+a context. Make sure to label your axes.
+
+Also make sure the plots are readable on an A4-sized PDF page; this is the format the notebook
+will be rendered to for report outputs. Specifically, this means that figure sizes should not
+exceed approx 15x15 inches.
+
+The report will contain 150 dpi png images of your plots. If you need higher quality output
+of individual plot files you should save these separately, e.g. via `fig.savefig(...)` yourself.
+
+## Running a calibration
+
+It utilizes tools such as
+[nbconvert](https://github.com/jupyter/nbconvert) and
+[nbparameterise](https://github.com/takluyver/nbparameterise) to expose
+[Jupyter](http://jupyter.org/) notebooks to a command line interface. In
+the process reports are generated from these notebooks.
+
+The general interface is:
+
+ % xfel-calibrate DETECTOR TYPE
+
+where [DETECTOR] and [TYPE] specify the task to
+be performed.
+
+Additionally, it leverages the DESY/XFEL Maxwell cluster to run these
+jobs in parallel via [SLURM](https://slurm.schedmd.com).
+
+Here is a list of `available_notebooks`.
+See the `advanced_topics` if you are
+looking for details on how to use as detector group staff.
+
+If you would like to integrate additional notebooks please see the
+`development_workflow`.
+
+## Interaction with Calibration Database
+
+During development, it is advised to work with local constant files first before injecting
+any calibration constants to the production database. After the notebook's algorithms arguments
+matured one can switch over to the test database and then production database.
+The reason for this is to avoid injecting wrong constants that can affect production calibration.
+And to avoid unnecessary intervention to disable wrong or unused injected calibration constants.
+
+Additionally, the calibration database is limited to XFEL networks, so independent development improves
+the workflow.
+
+
+## Testing
+
+The most important test is that your notebook completes flawlessly outside any special
+tool chain feature. After all, the tool chain will only replace parameters, and then
+launch a concurrent job and generate a report out of notebook. If it fails to run in the
+normal Jupyter notebook environment, it will certainly fail in the tool chain environment.
+
+Once you are satisfied with your current state of initial development, you can add it
+to the list of notebooks as mentioned in the :ref:`configuration` section.
+
+Any changes you now make in the notebook will be automatically propagated to the command line.
+Specifically, you should verify that all arguments are parsed correctly, e.g. by calling::
+
+ xfel-calibrate DETECTOR NOTEBOOK_TYPE --help
+
+From then on, check include if parallel slurm jobs are executed correctly and if a report
+is generated at the end.
+
+Finally, you should verify that the report contains the information you'd like to convey and
+is intelligible to people other than you.
+
+???+ note
+
+ You can run the `xfel-calibrate` command without starting a SLURM cluster job, giving
+ you direct access to console output, by adding the `--no-cluster-job` option.
+
+## Documenting
+
+Most documentation should be done in the notebook itself. Any notebooks specified in the
+`notebook.py` file will automatically show up in the [Available Notebooks](available_notebooks.md) section of this
+documentation.
+
+
+- [nbparameterise](https://github.com/takluyver/nbparameterise)
+- [ipcluster](https://ipyparallel.readthedocs.io/en/latest/)
+- [matplotlib](https://matplotlib.org/)
+- [numpy](http://www.numpy.org/)
+- [h5py](https://www.h5py.org/)
+- [iCalibrationDB](https://git.xfel.eu/detectors/cal_db_interactive)
+- [extra_data](https://extra-data.readthedocs.io/en/latest/)
+- [extra-geom](https://extra-geom.readthedocs.io/en/latest/)
+- [pasha](https://github.com/European-XFEL/pasha)
\ No newline at end of file
diff --git a/docs/xfel_calibrate_conf.md b/docs/xfel_calibrate_conf.md
new file mode 100644
index 0000000000000000000000000000000000000000..a24b14789d676e271a98839ab120165d30afb9cc
--- /dev/null
+++ b/docs/xfel_calibrate_conf.md
@@ -0,0 +1,6 @@
+# xfel-calibrate
+
+
+::: xfel_calibrate.calibrate.balance_sequences
+ :docstring: balance_sequences
+ :members: asasdads
\ No newline at end of file
diff --git a/mkdocs.yml b/mkdocs.yml
new file mode 100644
index 0000000000000000000000000000000000000000..f6f6323b0538cf276ad16e7876358be452d8e9fd
--- /dev/null
+++ b/mkdocs.yml
@@ -0,0 +1,92 @@
+site_name: EuXFEL Offline calibration
+
+theme:
+ name: "material"
+ features:
+ - navigation.tabs
+ - navigation.sections
+ - toc.integrate
+ - navigation.top
+ - search.suggest
+ - search.highlight
+ - content.tabs.link
+ - content.code.annotation
+ - content.code.copy
+ language: en
+ palette:
+ - scheme: light
+ toggle:
+ icon: material/lightbulb
+ name: Switch to dark mode
+ - scheme: slate
+ toggle:
+ icon: material/lightbulb-outline
+ name: Switch to light mode
+
+markdown_extensions:
+ - pymdownx.highlight:
+ anchor_linenums: true
+ - pymdownx.inlinehilite
+ - pymdownx.snippets
+ - pymdownx.tasklist
+ - pymdownx.arithmatex:
+ generic: true
+ - pymdownx.tabbed:
+ alternate_style: true
+ - pymdownx.details
+ - pymdownx.superfences
+ - pymdownx.mark
+ - pymdownx.emoji:
+ emoji_index: !!python/name:materialx.emoji.twemoji
+ emoji_generator: !!python/name:materialx.emoji.to_svg
+ - attr_list
+ - def_list
+ - footnotes
+ - md_in_html
+ - toc:
+ permalink: "%"
+ permalink: True
+ - admonition
+ - tables
+ - codehilite
+
+
+
+extra_css:
+ - css/custom.css
+
+plugins:
+ - glightbox
+ - literate-nav
+ - mkdocstrings:
+ custom_templates: templates
+ default_handler: python
+ - search
+ - autorefs
+
+repo_url: https://git.xfel.eu/calibration/pycalibration
+
+
+nav:
+ - index.md
+ - Operation:
+ - Calibration database: calibration_database.md
+ - myMDC: myMDC.md
+ - Troubleshooting:
+ - Correcting detector RAW data: troubleshooting_correction.md
+ - Calibration constant generation: troubleshooting_calibration_generation.md
+ - Development:
+ - Installation: installation.md
+ - Workflow: workflow.md
+ - How to write a notebook: how_to_write_xfel_calibrate_notebook_NBC.md
+ - Advanced topics: advanced.md
+ - Code Reference:
+ - CALCAT Interface: calcat_interface.md
+ - Calibration Tools: cal_tools.md
+ - Reference:
+ - FAQ: references/faq.md
+ - Changelog: references/changelog.md
+ - Glossary: glossary.md
+
+copyright: |
+ © 2018 <a href="https://www.xfel.eu/" target="_blank" rel="noopener">European XFEL</a>
\ No newline at end of file
diff --git a/setup.py b/setup.py
index 567962d615368aac963c7c6111ac23a0a6e7a610..aaebe02628ad26d646d15e63eafb13e4f454ce80 100644
--- a/setup.py
+++ b/setup.py
@@ -150,6 +150,9 @@ setup(
extras_require={
"docs": [
"nbsphinx",
+ "mkdocstrings-python",
+ "mkdocs-glightbox",
+ "mkdocs-literate-nav",
],
"test": [
"coverage",