Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • calibration/pycalibration
1 result
Show changes
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1251 additions and 1446 deletions
docs/operation/calibration_configurations.md 0 → 100644
View file @ 4786d0dc
# Calibration Configurations
[Calibration configurations](https://git.xfel.eu/detectors/calibration_configurations) is a separate project that is used by the offline calibration webservice to input the needed arguments for the triggered calibrations (e.g. correct or dark) per proposal.
These configurations are collected in YAML files. Each proposal can has it's own configuration, otherwise a default YAML file consists of the configurations for all calibrations, instruments, and detectors in case a proposal didn't have a dedicated YAML file.
# Configuration project structure
The default configuration is located in the *default.yaml* file on the top
hierarchy level.
The proposal specific configurations are optional (if none is found) the
default is used, and are organized by facility cycles:
```
--<CYCLE A>
|--<PROPOSAL 1>.yaml
|--<PROPOSAL 2>.yaml
--<CYCLE B>
|--<PROPOSAL 3>.yaml
|--<PROPOSAL 4>.yaml
```
Where cycles and proposals are identified by their numerical number, without
any prefixes or suffixes.
## Configuration file contents
Each proposal YAML file (and the default) is structured in the following way
``` YAML
<CALIBRATION>:
<INSTRUMENT>:
<DETECTOR-IDENTIFIER>:
<cal-parameter-name>: <cal-parameter-value>
<cal-parameter-name>: <cal-parameter-value>
<DETECTOR-IDENTIFIER>:
<cal-parameter-name>: <cal-parameter-value>
<cal-parameter-name>: <cal-parameter-value>
<data-mapping>:
<DETECTOR-IDENTIFIER>:
detector-type: <DET-TYPE>
karabo-da:
- <Aggregator-1>
- <Aggregator-2>
<xfel-calibrate-setting F>: XXX
<parameter-name>: <parameter-value>
```
Multiple instruments and detector types per YAML are possible. The `data-mapping` key must have all detectors added in any of the available calibrations (`<CALIBRATION>`). It consists of the parameter names and values that are used to access data and that are expected to be the same for all calibrations.
Each calibration (e.g. dark or correct) can have the cal-parameter-names and cal-parameter-values that correlate to those expected in their calibration notebooks, with `_` replaced with `-`.
<!-- The *inset* parameter is special, in that it is used to determine if files
of that detector are present, by evaluating if the following expression in the
raw input path returns a non-empty list:
``` bash
RAW-*<inset>*.h5
```
The inset paramter is not passed to xfel-calibrate. -->
Below is an example for the AGIPD detector at SPB:
``` YAML
correct:
SPB:
SPB_DET_AGIPD1M-1:
adjust-mg-baseline: true
force-hg-if-below: true
force-mg-if-below: true
hg-hard-threshold: 1000
low-medium-gap: true
mg-hard-threshold: 1000
rel-gain: true
dark:
SPB_DET_AGIPD1M-1:
thresholds-offset-hard:
- 0
- 0
thresholds-offset-hard-hg:
- 3500
- 6000
thresholds-offset-hard-lg:
- 6000
- 9000
thresholds-offset-hard-mg:
- 6000
- 9000
data-mapping:
SPB_DET_AGIPD1M-1:
ctrl-source-template: '{}/MDL/FPGA_COMP'
detector-type: agipd
karabo-da:
- AGIPD00
- AGIPD01
- AGIPD02
- AGIPD03
- AGIPD04
- AGIPD05
- AGIPD06
- AGIPD07
- AGIPD08
- AGIPD09
- AGIPD10
- AGIPD11
- AGIPD12
- AGIPD13
- AGIPD14
- AGIPD15
karabo-id-control: SPB_IRU_AGIPD1M1
receiver-template: '{}CH0'
```
Note how Boolean flags are indicated by *<parameter>: true*.
--------------------------------------------------------------
## Updating configuration through the webservice (update_config)
`update_config.py` is a python script that can be used to update the calibration configurations directly in the production environment without having access to the production node. Through command line interface the user can apply the desired modifications to the proposal YAML files at https://git.xfel.eu/detectors/calibration_configurations
This script is expected to be available in a location which is easily accessed by any user to modify the configurations of specific proposal. Not all detectors or parameters can be modified through the script.
### Changing the configuration:
- SSH to maxwell node
- run `module load anaconda3`
At this moment you are ready to start interacting with the `update_config` script at: `/gpfs/exfel/sw/calsoft/update_config.py`
The available detectors and parameters to modify for a proposal can be checked using `--help`. `python /gpfs/exfel/sw/calsoft/update_config.py --help`
<div style="page-break-after: always;"></div>
Below is the expected output. As it can be seen from the karabo-id argument, the only available detectors at the moment are: SPB_DET_AGIPD1M-1, MID_DET_AGIPD1M-, SQS_REMI_DLD6
```bash
usage: update_config.py [-h]
[--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}]
[--proposal PROPOSAL] [--cycle CYCLE]
[--correct | --dark] [--apply]
[--webservice-address WEBSERVICE_ADDRESS]
[--instrument {CALLAB}]
Request update of configuration
optional arguments:
-h, --help show this help message and
exit
--apply Apply and push the
requested configuration
update to the git.
--webservice-address WEBSERVICE_ADDRESS The port of the webservice
to update calibration
configurations repository.
--instrument {CALLAB} This is only used for
testing purposes.
required arguments:
--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}
--proposal PROPOSAL The proposal number,
without leading p, but
with leading zeros.
--cycle CYCLE The facility cycle.
--correct, -c
--dark, -d
```
<div style="page-break-after: always;"></div>
To check the available parameters that can be modified one can run: `python /gpfs/exfel/sw/calsoft/update_config.py --karabo-id SPB_DET_AGIPD1M-1 --help`
Below is a part of the output of the CL. As can be seen under the optional arguments are the exposed parameters by `update_config` for SPB_DET_AGIPD1M-1.
```bash
optional arguments:
-h, --help show this help message and
exit
--apply Apply and push the
requested configuration
update to the git.
--webservice-address WEBSERVICE_ADDRESS The port of the webservice
to update calibration
configurations repository.
--instrument {CALLAB} This is only used for
testing purposes.
--force-hg-if-below FORCE_HG_IF_BELOW TYPE: INT
--rel-gain REL_GAIN TYPE: BOOL
--no-rel-gain NO_REL_GAIN TYPE: BOOL
--xray-gain XRAY_GAIN TYPE: BOOL
--no-xray-gain NO_XRAY_GAIN TYPE: BOOL
--blc-noise BLC_NOISE TYPE: BOOL
--no-blc-noise NO_BLC_NOISE TYPE: BOOL
--blc-set-min BLC_SET_MIN TYPE: BOOL
--no-blc-set-min NO_BLC_SET_MIN TYPE: BOOL
--dont-zero-nans DONT_ZERO_NANS TYPE: BOOL
--no-dont-zero-nans NO_DONT_ZERO_NANS TYPE: BOOL
--dont-zero-orange DONT_ZERO_ORANGE TYPE: BOOL
--no-dont-zero-orange NO_DONT_ZERO_ORANGE TYPE: BOOL
--max-pulses MAX_PULSES [MAX_PULSES ...] Range list of maximum
pulse indices (--max-
pulses start end step). 3
max input elements. TYPE:
LIST
--use-litframe-finder USE_LITFRAME_FINDER TYPE: STR
--litframe-device-id LITFRAME_DEVICE_ID TYPE: STR
--energy-threshold ENERGY_THRESHOLD TYPE: INT
--karabo-da KARABO-DA [KARABO-DA ...] Choices: [AGIPD00 ...
AGIPD15]. TYPE: LIST
required arguments:
--karabo-id {SPB_DET_AGIPD1M-1,MID_DET_AGIPD1M-1,SQS_REMI_DLD6}
--proposal PROPOSAL The proposal number,
without leading p, but
with leading zeros.
--cycle CYCLE The facility cycle.
--correct, -c
--dark, -d
```
Every exposed parameter has its type available beside the name.
Note: The boolean parameters can not be set to false. For example to set `xray-gain` to false, one should set `no-xray-gain` to true.
<div style="page-break-after: always;"></div>
An example running the CL:
`python /gpfs/exfel/sw/calsoft/update_config.py --cycle 202031 --proposal 900146 --karabo-id SPB_DET_AGIPD1M-1 --rel-gain true --no-xray-gain true --max-pulses 1 20 1`
The output can be something like this:
```bash
--------------------------------------------------------
THIS IS A DRY RUN ONLY, NO CHANGES ARE MADE
---------------------------------------------------------
# Sending the following update:
correct:
SPB:
SPB_DET_AGIPD1M-1:
max-pulses:
- '1'
- '20'
- '1'
rel-gain: true
xray-gain: false
---------------------------------------------------------
# Configuration now in place is:
correct:
SPB:
SPB_DET_AGIPD1M-1:
adjust-mg-baseline: false
blc-noise: true
blc-set-min: false
blc-stripes: false
cm-dark-fraction: 0.5
cm-dark-range:
- -50
- 30
cm-n-itr: 4
common-mode: false
force-hg-if-below: true
force-mg-if-below: true
hg-hard-threshold: 1000
low-medium-gap: false
max-pulses:
- '1'
- '20'
- '1'
mg-hard-threshold: 1000
rel-gain: true
xray-gain: false
```
As the output shows, this is a dry run only. That means that the changes are not applied. This is just a display of what the changes would look like.
To run the changes and apply it to the calibration configurations, you should execute the same command with `--apply` argument. e.g.:
`python /gpfs/exfel/sw/calsoft/update_config.py --cycle 202031 --proposal 900146 --karabo-id SPB_DET_AGIPD1M-1 --rel-gain true --no-xray-gain true --max-pulses 1 20 1 --apply`
This should update the production with your changes. To validate the added changes into production, you can simply check the gitlab [calibration_configurations repository](https://git.xfel.eu/detectors/calibration_configurations) and look for the latest commit to master. This should have your applied changes.
docs/operation/calibration_database.md 0 → 100644
View file @ 4786d0dc
# Calibration database
This is the main database where all calibration constants are stored.
The calibration database is based on CALCAT as the main web interface
and the GPFS file system where all calibration constant files are
stored.
It is responsible for storing and managing all calibration detector constants. It maintains the history of calibration constants and manage different versions for the calibration database.
The Calibration database is expected to be queried using:
- Calibration constant name
- Conditions
- Detector module name
## Calibration constant
## CALCAT
![image](../static/calcat/home_calcat.png)
CALCAT can be accessed through <https://in.xfel.eu/calibration>. It
consists of multiple definitions and metadata related to the stored
calibration constants. the definitions for the available detector types
(AGIPD, JUNGFRAU, ePix100, ...), the available instruments, the
available detectors in all instrument, the mapping of each detector to
it's connected physical detector units, and all calibration constants.
All of these content ensure retrieving the correct calibration constant
any point in time.
Through CALCAT page one can check all stored calibration constants,
where the files are stored, and their metadata. Additionally, there are
different ways to edit various parameters like the detector to PDU
mapping \<TODO: add a definition link\>, adding new calibrations, and
new parameter conditions.
In order to proceed with the relevant parts of CALCAT, it is critical to
first establish clear definitions for the terms used in both CALCAT and
pyCalibration within the context of the Calibration database.
- Calibrations: These are the types of calibration available that can
be applied to RAW data. For example Offset, Noise, and BadPixelsDark
are some calibrations available in CALCAT.
- Calibration Constants (CC): Calibration Constants are a list of
calibration constant versions with the same parameter conditions.
For example 3 Offsets injected in with 3 different creation time for
AGIPD and the same parameter conditions are in the same CC list.
![image](../static/calcat/home_cc_calcat.png)
- Calibration Constant Version (CCV): CCVs are injected through time
into the database. Each CCV must have a list of parameter
conditions, an attached timestamp of when it was created, the
physical detector unit, the file path, and the dataset name to
access its data.
![image](../static/calcat/home_ccv_calcat.png)
- Parameters conditions: The CC parameter conditions are decided by
detector experts because the CC\'s data would differ based on it.
Some examples are memory cells, integration time, and bias voltage.
![image](../static/calcat/home_conditions_calcat.png)
- PDU: Physical detector unit is the physical name given by the manufacturer or detector experts to the detector module's hardware. This representation is very important to calibration, as it is only trace of the physical movement for the detector modules.
![image](../static/calcat/home_pdu_calcat.png)
- Detector: A detector for CALCAT is based on a detector identifier
name, which was based on the Karabo ID give for detector karabo
devices. A detector can consist of one module like
HED_IA1_JF500K1, or multiple modules like MID_DET_AGIPD1M-1,
FXE_DET_LPD1M-1, and SPB_IRDA_JF4M.
![image](../static/calcat/home_detectors_calcat.png)
- Reports: pyCalibration's PDF reports are the main data
visualization source for any calibration processing, either
correcting raw data or generating calibration constants. For CALCAT
the only relevant reports are the reports for generating new
calibration constants. The paths for these reports are one injected
metadata along with the calibration constant.
![image](../static/calcat/home_reports_calcat.png)
## Detector Mapping
Detector mapping is the operation of associating physical detector units
(modules) with its data sources (raw data, data aggregators and calibration
constants files)
<figure markdown>
![AGIPD PDUs](../static/detector_mapping/AGIPD_modules_pic.png){ align=left width="500" }
<figcaption>AGIPD PDUs [AGIPD Nr. 315 = AGIPD_SIV1_AGIPDV11_M315]</figcaption>
</figure>
<figure markdown>
![JUNGFRAU PDU](../static/detector_mapping/JUNGFRAU_PDU_pic.png){ align=left width="500" }
<figcaption>JUNGFRAU PDU [Jungfrau Nr. 035 = Jungfrau_M035]</figcaption>
</figure>
PDUs can be transferred between detectors and instruments or replaced. To correct raw data, the calibration pipeline produces calibration constants, which are specific to the module they were generated from. As a result, it's crucial to map the hardware to the corresponding software modules.
As the detector is
### Modifying detector mapping
The detector experts are the most up-to date with the PDU changes. They are able to update and track the PDU names to software module using CALCAT.
Updating the mapping immediately after making any hardware modifications is crucial before acquiring RAW data for calibration. This is because during offline calibration, the timestamp of the acquired RAW data serves as a reference point to retrieve the correct mapping from the database.
### History of Detector mapping
![Screenshot from iCalibrationDB](../static/detector_mapping/icalibrationdb_agipd_det_mapping.png){ align=right width=200 height=350}
In the beginning, detector mapping was designed to be hard coded in [iCalibrationDB](https://git.xfel.eu/detectors/cal_db_interactive). It is an interface package to [CalibrationDBRemote](https://git.xfel.eu/karaboDevices/calibrationDBRemote) and CALCAT.
This screenshot shows the last updated hard coded physical detector units for AGIPD detector at SPB.
The of course led with error-prone to not be updated in time after there were changes reported when PDUs where replaced or moved.
## Database snapshot
A database snapshot is a point-in-time copy of a database that provides a read-only, consistent view of the data as it existed at the time the snapshot was created. It's a static picture of the database and includes all the data that was in the database at the time of the snapshot's creation. Database snapshots are useful for performing data analysis or creating reports without interfering with ongoing database transactions. They can also be used for backup and recovery purposes or to provide a consistent view of the data for testing and development purposes.
CALCAT snapshot feature is used for obtaining the correct detector mapping. This is done by using the RAW data timestamp (creation_time) as the snapshot timestamp.
This is why it is crucial to update the detector mapping immediately after any hardware changes.
\ No newline at end of file
docs/operation/myMDC.md 0 → 100644
View file @ 4786d0dc
# Metadata Catalouge
European XFEL Data Portal is provided by myMdC, allowing users to access the metadata of scientific data taken at the European XFEL facility. Datasets are linked to experiment proposals, annotated by data collection types, samples and are assigned DOIs to be referenced in publications. myMdC helps data management activities and allows users to find open and own proposals data based on the metadata.
myMDC stands for Metadata Catalog. It arranges the facility's data into proposals. Each proposal consists of a number of acquired runs. Offline calibration only works with runs set to Good `Run Quality`, which means it was migrated from the online cluster to `gpfs`.
Through myMDC users can request to correct migrated runs. Additionally, users can trigger dark calibration constants generation detector data.
## Proposals
Proposals are created for commissioning or user experiments. There can be a proposal per instrument. Every proposal has a number of team members who have access to interact with the proposal's runs.
proposal's RAW data location can be found at: `/gpfs/exfel/exp/<instrument>/<cycle>/<proposal>/raw`
## Runs
Runs have multiple properties to modify through myMDC.
The most relevant properties are:
- Run Quality: Currently only migrated runs are allowed to be offline calibrated.
- Run Types: A run type can be relevant in accepting a correction request or not. e.g. Correcting dark runs can be skipped on purpose.
## Triggering offline correction
![Trigger correction](../static/myMDC/correction.png)
Through the `Runs` page, one can trigger offline correction with the right proposal access. This screenshot shows a sample from the `Runs` page for the test proposal 900203.
The main important columns in the `Runs` page are:
- Run number: The run number
- Run type: This is an assigned type for the run.
- Start date: The data when the run was acquired.
- Run status: {TODO}
- Data Assessment: If data was migrated or not.
- Calibration: A symbol showing the calibration availability.
??? Warning "Dark runs are skipped."
Dark runs are not calibrated. In case a manual request was done. The calibration status will be in error state.
![!Run correction status](../static/myMDC/run_9037_general_status.png){align=right width=240}
To check more details about a specific run like its ID, size, number of files, first and last train Ids, clicking on the run number will convert you the run page.
This page is very useful in checking the correction status, as both `Calibrated Process data status` and `Calibration Pipeline auto Reply` keys are updated regularly by the calibration service until a correction is finished.
For example if a run correction request failed. `Calibration Pipeline auto Reply` can have very useful information on the issue and ideally how to solve it.
---------------------------------------------------
## Generating calibration constants
![calibration constant generation](../static/myMDC/calibration_constants.png)
Currently, only dark calibration constants generation can be triggered through myMDC. the below screenshot shows the `Calibration Constants` page for proposal 900203. As noted before the current available option is to request dark calibration. To go through:
- Detector: First you need to select the desired detector from the available detectors of the proposal.
- Detector Units: By default, all the detector's modules are selected. But you can deselect the undesired modules. e.g. if a module is known not to have data for this run.
- Operation Mode: This can be left with the default value. It is meant to be used if there is a detector that should generate constants in different operation modes. But this is not used at the moment.
- Run Number(s): At the end you need to include the number of dark runs to create the calibration constants. This can be one or more runs depending on the detector.
- Description [optional]: You can optionally add a description for this particular dark constant request.
### Calibration Constants requests
![dark requests status](../static/myMDC/dark_different_requests.png)
!!! Note
The screenshots are from a special test proposal, therefore there are multiple detectors from different instruments in the same proposal.
As it can be seen in the screenshot, this section has the latest dark requests with their status, last update, PDF report link, CALCAT report link, and two blue buttons to clone the request for another submit and to check the dark request details, as can be seen in the below screenshot.
### Dark runs' request details
This page is very useful to check more details regarding the dark request. For example the report path, the output path, who triggered request and when, and the calibration pipeline feedback.
![Dark request status success](../static/myMDC/dark_request_status_success.png){: style="height:280px"}
![Dark request status error](../static/myMDC/dark_request_status_error.png){: style="height:280px"}
The calibration pipeline feedback is crucial in case the dark request is in error state. As it can have helpful message over what was the cause of the failure.
??? Note "Dark runs' request details doesn't auto-refresh."
This dark runs' request details page doesn't update automatically. Therefore, you need to close and reopen to see new updates.
\ No newline at end of file
docs/operation/webservice.md 0 → 100644
View file @ 4786d0dc
# Calibration Webservice
As the heart of the calibration data pipeline. The webservice receives requests from [myMDC](myMDC.md) to correct data or generate dark calibration constants.
The webservice is a calibration service that is deployed on a Maxwell node as a part of pyCalibration's new releases.
This service hanlde requests from myMDC via ZMQ interface to create the needed xfel-calibrate CL for the selected detector (i.e dark request), or for the available detectors in the RAW data (i.e correction request). The CL arguments are given from different configurations in the production environment, e.g. [calibration configuration](calibration_configurations.md)
Beside forming and executing the CL for the corresponding detector and calibration, the webservice monitor the states for the calibration Slurm nodes to report periodically it's state to myMDC and to conclude the last state for the whole calibration request with a response that can help the user to reach the successfully calibration data or to have an indication on why the Slurm job was not completed at the end.
## Job database
The webservice uses SQLite database to store and keep track of the requests, executions, and calibration Slurm jobs.
![job database](../static/webservice_job_db.png)
As can be seen, there are three tables. Executions, Slurm jobs, Requests.
## Handling dark request
Users can generate calibration constants using [myMDC](myMDC.md#calibration-constants-requests). At the moment only dark calibration constants can be generated through myMDC and the webservice. Via ZMQ, the webservice start handling a dark request and update the [Job DB](#job-database) with the new request ID. Similar to the correction, the [configurations](calibration_configurations.md) are used for adding the needed configurations for xfel-calibrate CLI. The launch will not go through as long as the dark runs are migrated and it will either wait for the transfer to finish or timeout. The webservice is expected to reply to myMDC with the status for the dark request and send a successful message or an error message with the reason for the error.
## Handling correction request
Users can trigger offline calibration through [myMDC](myMDC.md#triggering-offline-correction). The webservice would handle this remote request via ZMQ and start with assigning this request into the [Job DB](#job-database). Next step would be reading configurations for the correction xfel-calibrate CLI and launch the correction after confirming that the RAW data is migrated, otherwise the correction waits until the transfer is complete. By default the corrections are disabled and skipped for dark run types. The webservice replies to [myMDC](myMDC.md) with a success message that the correction is launched or with an error message if the correction was not launched.
## job monitor
The Job DB is regularly monitored by a dedicated service
docs/overview.md 0 → 100644
View file @ 4786d0dc
# Offline calibration overview
TODO: HERE I PLAN TO GIVE DIAGRAMS OVER THE OFFLINE CALIBRATION SERVICES AND PIECES
!!! quote
European XFEL aims to provide facility users with a fully corrected and calibrated dataset as the primary data product. - SRN 27.4, 35 (2014)
![image](./static/xfel_calibrate_diagrams/overview_all_services.png)
Offline calibration consists of multiple algorithms and processes to achieve this role. These steps can differ across detectors. Offline calibration process big amount of data, and it is essential to have the calibration pipeline compatible with RAW data format across time.
pyCalibration is the main pipeline package for offline calibration at European XFEL. It utilizes MAXWELL HPC to run Jupyter notebooks concurrently using SLURM nodes to process `RAW` data. Before correcting detector `RAW` data, and after the detector experts characterize the detector
The pieces we have are:
- [Calibration Database](operation/calibration_database.md) The calibration database (CALCAT)
- [Metadata Catalog](operation/myMDC.md) myMDC for requesting run corrects and dark constants' generation.
- [Calibration Notebooks](operation/available_notebooks.md) All available detector calibration notebooks
- [xfel-calibrate](development/configuration.md) The machinery for running calibration notebooks using SLURM
- [Calibration webservice](operation/webservice) The main entry for offline calibration pipeline in production
docs/reference/xfel-calibrate_cli_process_no_caldbremote.png 0 → 100644
View file @ 4786d0dc
docs/reference/xfel-calibrate_cli_process_no_caldbremote.png

20.4 KiB

docs/references/changelog.md 0 → 100644
View file @ 4786d0dc
This diff is collapsed.
docs/references/faq.md 0 → 100644
View file @ 4786d0dc
# Frequently Asked Questions
docs/requirements.txt
View file @ 4786d0dc
iCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
mkdocs==1.4.2
mkdocs-material==9.0.15
mkdocstrings[python]==0.19
mkdocs-glightbox
mkdocs-section-index==0.3.5
mkdocs-literate-nav==0.5.0
mkdocs-redirects==1.2.0
mkdocs-gen-files==0.4.0
griffe==0.27.1
jupyter
docs/source/_static/css/test_decorators.css deleted 100644 → 0
View file @ 2b0ad6ba
.header-passed {
display: inline-block;
width: 40px;
line-height: 40px;
border: 1px solid #196038;
text-align: center;
background-color: #91e0b4;
font-size: 200%;
font-weight: bold;
float: right;
}
.header-failed {
display: inline-block;
width: 40px;
line-height: 40px;
border: 1px solid #871b1b;
text-align: center;
background-color: #e09191;
font-size: 200%;
font-weight: bold;
float: right;
}
.header-skipped {
display: inline-block;
width: 40px;
line-height: 40px;
border: 1px solid #a59808;
text-align: center;
background-color: #f7ee8c;
font-size: 200%;
font-weight: bold;
float: right;
}
.passed {
display: inline-block;
width: 80px;
border: 1px solid #196038;
text-align: center;
background-color: #91e0b4;
font-weight: bold;
}
.failed {
display: inline-block;
width: 80px;
border: 1px solid #871b1b;
text-align: center;
background-color: #e09191;
font-weight: bold;
}
.skipped {
display: inline-block;
width: 80px;
border: 1px solid #a59808;
text-align: center;
background-color: #f7ee8c;
font-weight: bold;
}
docs/source/cal_tools_algorithms.rst deleted 100644 → 0
View file @ 2b0ad6ba
cal_tools
=========
.. module:: cal_tools.agipdlib
.. class:: AgipdCorrections
.. attribute:: read_file
.. attribute:: write_file
.. attribute:: cm_correction
.. attribute:: mask_zero_std
.. attribute:: offset_correction
.. attribute:: baseline_correction
.. attribute:: gain_correction
.. attribute:: get_valid_image_idx
\ No newline at end of file
docs/source/calcat_interface.rst deleted 100644 → 0
View file @ 2b0ad6ba
CALCAT Interface
================
.. module:: cal_tools.calcat_interface
.. class:: CalCatError
.. class:: CalibrationData
.. attribute:: metadata
.. attribute:: ndarray
.. attribute:: ndarray_map
.. class:: SplitConditionCalibrationData
.. class:: LPD_CalibrationData
.. class:: DSSC_CalibrationData
.. class:: JUNGFRAU_CalibrationData
.. class:: PNCCD_CalibrationData
.. class:: EPIX100_CalibrationData
.. class:: GOTTHARD2_CalibrationData
docs/source/changelog.rst deleted 100644 → 0
View file @ 2b0ad6ba
Release Notes
=============
3.7.5
-----
16-11-2022
- [JUNGFRAU] Fix index location for ROI instrument output https://git.xfel.eu/detectors/pycalibration/-/merge_requests/765
3.7.4
-----
15-11-2022
- [Jungfrau] Only emit a warning if all sequences of a job are empty
- [Jungfrau] Fix storing multiple ROIs for a single module
- [Jungfrau] Replicate a raw dataset in processed data for legacy analysis code
3.7.3
-----
10-11-2022
- [DataFile] Add support for older INDEX and METADATA versions.
- [LPD] Fix histogram plotting if selected cell's data is identical for trains.
- [Jungfrau] Avoid unclear error messages when the run has no trains.
- [AGIPD] Change notebook-default setting of max-task-per-worker to 1.
- [REMI] Allow pulse separation in REMI/DLD reconstruction to ignore FEL pulses.
- [webservice] Fix timestamps in the overview page.
- [Webservice] Check run type more directly in myMdC response.
3.7.2
-----
28-10-2022
- [Jungfrau] Add support for strixel sensors
- [Jungfrau] Shorter summary notebook for darks by default
- [Jungfrau] Always align corrected data properly
- [Jungfrau] Harden against 0 adc data in dark characterization
- [REMI] Add support for quad nodes and various pulse separation improvements
- [AGIPD] Allow manual photon energy input for photonization and add consistency plots
- Grant priority partitions for set-up time
3.7.1
-----
14-10-2022
- [EPIX100] Add Common mode corrections to dark characterization
- [EPIX100] Use a single value for temperature
- [JUNGFRAU] Fix bad pixel masking
- [JUNGFRAU] Limit number of plotted trains to 500 during corrections
- [AGIPD] Don't fail when there are no trains in sequence file
- [REMI] Use DataFile API and support asymmetric pump-probe patterns
- Skip corrections for runs which match specific types
3.7.0
-----
07-09-2022
- [JUNGFRAU] Mask double sized pixels for Jungfrau https://git.xfel.eu/detectors/pycalibration/-/merge_requests/689
- [LPD] Storing retrieved calibration constants in calibration_metadata.yml.
- Storing the running logs in the report folder path instead of a `temp` folder wherever the processing ran from.
- Selecting partitions based on beamtime dates. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/725
- Creation time is not read from myMDC anymore.
3.6.4
-----
20-08-2022
- [AGIPD] Significant improvements to AGIPG LitFrameFinder implementation, including offline support.
- [AGIPD] Fix misalignment of data and gain/mask when common mode is used with manual pulse slicing.
3.6.3
-----
23-08-2022
- [AGIPD] litframe finder configs in update_config https://git.xfel.eu/detectors/pycalibration/-/merge_requests/723
3.6.2
-----
15-08-2022
- [EPIX100][Jungfrau][pnCCD] Retrieve constants precorrection notebooks.
- [AGIPD] skip sanitization https://git.xfel.eu/detectors/pycalibration/-/merge_requests/654
- [AGIPD] New PC notebook https://git.xfel.eu/detectors/pycalibration/-/merge_requests/604
- [JUNGFRAU] Allow selecting ROIs to save 1D projections https://git.xfel.eu/detectors/pycalibration/-/merge_requests/699
3.6.1
-----
26-07-2022
- [GOTTHARD2] Correction and Dark notebooks. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/658, https://git.xfel.eu/detectors/pycalibration/-/merge_requests/684, https://git.xfel.eu/detectors/pycalibration/-/merge_requests/707
- pin `xarray` to 2022.3.0 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/706
3.6.0
-----
20-07-2022
- [LPD][Dark] Save LPD bad pixel dark data as uint32, not float64. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/698
- [LPD][Correct] Add option to disable Oauth when using CalCat proxy. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/681
- [AGIPD][LPD][DSSC][DARK] Avoid printing errors for expected missing constant files. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/664
- [TESTs] Refactor and split `AGIPDCtrl` methods (old and new RAW data versions.) and add unit tests https://git.xfel.eu/detectors/pycalibration/-/merge_requests/651, https://git.xfel.eu/detectors/pycalibration/-/merge_requests/646
- [Webservice] Monitor SLURM jobs in a separate process. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/668
- [Webservice] Restructure the SLURM database to give more meaningful success/failure information. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/683
- [XFEL-CALIBRATE] Recreate environments on demand for xfel-calibrate-repeat. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/660
- Upgrade the Jupyter packages to the latest version. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/697/diffs
- Update Extra-data to 1.12: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/693
- [EXtra-data] Disable `_use_voview` to be able to use `.files` or avoid using `.files` https://git.xfel.eu/detectors/pycalibration/-/merge_requests/682, https://git.xfel.eu/detectors/pycalibration/-/merge_requests/688
3.5.5
-----
15-06-2022
- [AGIPD][CORRECT] Expose max tasks per pool worker.
3.5.4
-----
13-06-2022
- [AGIPD] Convert bias_voltage parameter condition to integer in cal_tools.
- [LPD] Fix correcting a single pulse.
- [LPD] VCXI require 4 modules.
3.5.3
-----
19-05-2022
- [LPD][CORRECT] Optionally create virtual CXI files
- [LPD][CORRECT] Expose max-nodes parameter
- [AGIPD] Replace gain_choose_int by fused types
- Fix missing install of restful_config.yaml
- Fix use of xfel-calibrate --skip-report
3.5.2
-----
16.05.2022
- [LPD][CORRECT] New correction notebook for LPD
- New `files` module to write European XFEL HDF5 corrected data files.
3.5.1
-----
05-04-2022
- Calibration Constant version's new `Variant` file attribute. To indicate method of handling the constant post retrieval. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/619
- Epix100 dark Badpixels Map. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/637
- `skip-plots` flag to finish correction before plotting. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/635
- First trainId's timestamp as RAW data creation_time, if there is myMDC connection. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/630
- AGIPD correction can correct one cellId without plotting errors. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/642
- Fixed mode relative gain constants in Jungfrau can be retrieved. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/639
- Only instrument source is selected to check number of trains to dark process. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/636
- AGIPD trains for dark processing is selected for each module individually. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/653
- Produce report after trying to correct AGIPD run with no images. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/641
- AGIPD's bias voltage for AGIPD1M is read from slow data. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/647
- Removed psutil dependency. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/653
- Update Pasha to 0.1.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/638
3.5.0
-----
01-03-2022
- Updating Correction and dark notebooks for JUNGFRAU: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/518
- Updating Correction and dark notebooks for AGIPD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
- Updating Correction and dark notebooks for PnCCD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/559
- Updating Correction and dark notebooks for ePix100: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/500
* EXtra-data is integrated to read files in pycalibration for AGIPD, JUNGFRAU, ePix100, and PnCCD. Dark and Correction notebooks.
* Pasha is now used for processing data for JUNGFRAU, ePix100 and PnCCD.
* pyDetLib correction functions were removed (except for common-mode correction).
* `db-module` is useless now for JUNGFRAU, ePix100 and PnCCD. Some parameters were updated in dark and correction notebooks for the mentioned detectors.
- `gain_mode` and burst mode are now available for JUNGFRAU. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/591
- JUNGFRAU has now a new badpixel value, `WRONG_GAIN_VALUE`. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/591
- Pass through available for testing in-progress ORCA service. https://git.xfel.eu/detectors/pycalibration/-/merge_requests?scope=all&state=merged&search=orca
- Non-calibrated RAW h5files are no longer copied.
- High priority partitions (`upex-high`and `upex-middle`) are used for runs from ACTIVE and READY proposals, only. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/628
- Supporting to disable LPD Correction through the webservice. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/629
- Compatibility for old DAQ files for REMI is added. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/607
- server-overview refactors. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/593 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/589
- AGIPD correction notebook support AgipdLitFrameFinder device. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/603
- Parsing code arguments in xfel-calibrate is refactored. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/575
- skip-plots option for AGIPD. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/581
- Native implementation for transposition of constants AGIPD. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/580
- Trains for AGIPD can be selected for correction. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/574
- Skip report flag in xfel-calibrate. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/579
- Fix ReadTheDocs. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/448
- Fix error reporting for re-injecting the same CCV. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/627
- Fix AGIPD for legacy runs without `gain_mode`. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/617 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/626
- Pinning markupsafe version 2.0.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/631
- Pinning psutil 5.9.0 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
- Updating Extra-data to 1.9.1 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
- Updating h5py to 3.5.0 https://git.xfel.eu/detectors/pycalibration/-/merge_requests/602
3.4.3
-----
20-10-2021
- Update pyDetLib tag.
- Add explicit dependencies on matplotlib, scipy.
- Remove outdated matplotlib rcParams setting.
- Update EXtra-geom to 1.6.
- Remove cluster_profile parameter from notebooks which don't use it.
- Fix checking availability for the concurrency parameter.
- Fix launching work directly (not via Slurm).
- Fix `sphinx-rep` temp folder recreation, if sphinx-rep already existed.
- Fix missing string conversion for slurm-scheduling argument.
- Fix title reports for multiple detectors per run folder.
- Append to .out files for preemptable finalize job.
- [AGIPD] [Correct] Reuse previously found constants.
- [AGIPD] Fix missing memory cell index in SlopesPC constant sanitization.
- [AGIPD] Only use bad pixels from darks in agipdutils.baseline_correct_via_stripes.
- [AGIPD] [Dark] Use function to get list of karabo_da from run for making Slurm jobs.
- [EPIX100][CORRECT] Set absolute_gain to false if relative gain was not retrieved.
- [JUNGFRAU] Fix running for multiple modules and flip logic for do_relative_gain.
- [JUNGFRAU] Style changes for Dark and Correct notebooks.
- [REMI] Add notebook to reconstruct detector hits from raw data.
- [webservice] Check run migration status using MyMDC.
- Resolve "Skip ZMQ tests if zmq connection for calibration DB not available".
- Reproducibility, step 1.
3.4.2
-----
17-09-2021
- Remove driver=core from all notebook
- [webservice] Make use of Dynaconf for managing secrets.
- [webservice] Make use of dedicated slurm partitions.
- [webservice] Handle missing migration information (missing user.status fattr).
- [webservice] Implement, raise, and catch, migration errors to send mdc messages.
- [webservice] Simplify handling of user notebook paths.
- [webservice] Update princess to 0.4 (use Unix sockets).
- [webservice] Update MyMDC with begin and end times.
- [webservice] create output folder before copying slow data.
- [AGIPD] [CORRECT] read acq_rate from slow data.
- [AGIPD][CORRECT] Set default memory cells to 352.
- [AGIPD] [CORRECT] Set maximum pulses to correct based on file content.
- [AGIPD] [FF] Correctly label legends in figures.
- [AGIPD] [FF] Add HIBEF AGIPD500K and fix some issue with retrieval of conditions.
- [Jungfrau] Add Gain setting to Jungfrau notebooks.
- [Jungfrau] Fix max gain plot in LPD correct notebook
- [JUNGFRAU] [DARK] Clearer error message for Jungfrau Dark notebooks no suitable files are found
- [LPD] [CORRECT] Fix max gain plot.
- [EPIX100] [CORRECT] Solve conflict between gain correction and clustering
3.4.1
-----
16-07-2021
- Update h5py to 3.3
- Stop execution on notebook errors
- [AGIPD] Add integration time as operating condition to all notebooks
- [webservice] Add blocklist pattern when copying untouched files in webservice.
- [webservice] Expose dark configurations in update_config.py
- Fix MetadataClient.get_proposal_runs arguments call.
- Fix Use snapshot for injecting constants for old PDU mappings
- Fix the old time-summary (creation time for retrieved constants)
- Update documentation notes on venv installation
- Ignore all .so files in gitignore
3.4.0
-----
28-06-2021
- Update to Python 3.8.
- Bump numpy to 1.20.3 and remove fabio.
- remove PyQT dependency.
- Disable dark requests from serve overview.
- Update report upload parameter key.
- Override locale to always use UTF-8.
- Assorted cleanup of xfel-calibrate.
- Fix pre-commit.
- Use argparse only if name is main, call main with args dict.
- [webservice] Use full hostname for webservice overview.
- [webservice] Show clearer messages when running webservice in sim mode.
- [webservice] Fix filename lineno and typos in webservice logs.
- [webservice] Fix creating an extra run folder in run output folder.
- [AGIPD] Parallelize gain/mask compression for writing corrected AGIPD files.
- [AGIPD][DARK] Fix processing empty sequence files.
- [AGIPD][PC][FF] Update notebooks with new CALCAT mapping.
- [AGIPD][JUNGFRAU] Use all available sequences for processing darks for AGIPD and Jungfrau.
- [AGIPD][LPD][DSSC] Fix retrieve old constants for comparison for modular detectors.
- [LPD] Fix data paths in LPD notebook.
- [REMI] Fix user notebook path for REMI correct notebook provisionally.
- [EPIX][CORRECT] Add Common mode correction.
- Fix plotting-related warnings.
- Test update config.
- Test get_from_db and send_to_db.
docs/source/conf.py deleted 100644 → 0
View file @ 2b0ad6ba
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
# European XFEL Offline Calibration documentation build configuration file, created by
# sphinx-quickstart on Sun Jun 10 17:32:30 2018.
#
# This file is execfile()d with the current directory set to its
# containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import glob
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
import os
import shutil
import sys
import textwrap
from datetime import datetime
from subprocess import Popen, check_output
from textwrap import dedent, indent
from uuid import uuid4
import nbformat
import tabulate
from dateutil.parser import parse
from lxml import etree
from nbconvert import RSTExporter
# generate the list of available notebooks
from xfel_calibrate import notebooks
# -- General configuration ------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.intersphinx',
'sphinx.ext.todo',
'sphinx.ext.mathjax',
'sphinx.ext.ifconfig',
'sphinx.ext.viewcode',
]
sys.path.append(os.path.abspath("../pycalibration/"))
p = Popen(["./makeAllDocs.sh"])
p.communicate()
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
# source_suffix = ['.rst', '.md']
source_suffix = '.rst'
# The encoding of source files.
#
# source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = 'European XFEL Offline Calibration'
copyright = '2018, The European XFEL Detector Group'
author = 'The European XFEL Detector Group'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
# built documents.
#
# The short X.Y version.
version = '1.0'
# The full version, including alpha/beta/rc tags.
release = '1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#
# today = ''
#
# Else, today_fmt is used as the format for a strftime call.
#
# today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This patterns also effect to html_static_path and html_extra_path
exclude_patterns = []
# The reST default role (used for this markup: `text`) to use for all
# documents.
#
# default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#
# add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
#
# add_module_names = True
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
#
# show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
# modindex_common_prefix = []
# If true, keep warnings as "system message" paragraphs in the built documents.
# keep_warnings = False
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True
# -- Options for HTML output ----------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'sphinx_rtd_theme'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
# html_theme_path = []
# The name for this set of Sphinx documents.
# "<project> v<release> documentation" by default.
#
# html_title = 'European XFEL Offline Calibration v1.0'
# A shorter title for the navigation bar. Default is the same as html_title.
#
# html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#
# html_logo = None
# The name of an image file (relative to this directory) to use as a favicon of
# the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#
# html_favicon = None
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Add any extra paths that contain custom files (such as robots.txt or
# .htaccess) here, relative to this directory. These files are copied
# directly to the root of the documentation.
#
# html_extra_path = []
# If not None, a 'Last updated on:' timestamp is inserted at every page
# bottom, using the given strftime format.
# The empty string is equivalent to '%b %d, %Y'.
#
# html_last_updated_fmt = None
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#
# html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#
# html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#
# html_additional_pages = {}
# If false, no module index is generated.
#
# html_domain_indices = True
# If false, no index is generated.
#
# html_use_index = True
# If true, the index is split into individual pages for each letter.
#
# html_split_index = False
# If true, links to the reST sources are added to the pages.
#
# html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#
# html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#
# html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#
# html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
# html_file_suffix = None
# Language to be used for generating the HTML full-text search index.
# Sphinx supports the following languages:
# 'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja'
# 'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr', 'zh'
#
# html_search_language = 'en'
# A dictionary with options for the search language support, empty by default.
# 'ja' uses this config value.
# 'zh' user can custom change `jieba` dictionary path.
#
# html_search_options = {'type': 'default'}
# The name of a javascript file (relative to the configuration directory) that
# implements a search results scorer. If empty, the default will be used.
#
# html_search_scorer = 'scorer.js'
# Output file base name for HTML help builder.
htmlhelp_basename = 'EuropeanXFELOfflineCalibrationdoc'
# -- Options for LaTeX output ---------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
'extraclassoptions': 'openany, oneside',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'EuropeanXFELOfflineCalibration.tex', 'European XFEL Offline Calibration Documentation',
'The European XFEL Detector Group', 'manual', True),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#
# latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#
# latex_use_parts = False
# If true, show page references after internal links.
#
# latex_show_pagerefs = False
# If true, show URL addresses after external links.
#
# latex_show_urls = False
# Documents to append as an appendix to all manuals.
#
# latex_appendices = []
# It false, will not define \strong, \code, itleref, \crossref ... but only
# \sphinxstrong, ..., \sphinxtitleref, ... To help avoid clash with user added
# packages.
#
# latex_keep_old_macro_names = True
# If false, no module index is generated.
#
# latex_domain_indices = True
# -- Options for manual page output ---------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
(master_doc, 'europeanxfelofflinecalibration', 'European XFEL Offline Calibration Documentation',
[author], 1)
]
# If true, show URL addresses after external links.
#
# man_show_urls = False
# -- Options for Texinfo output -------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'EuropeanXFELOfflineCalibration', 'European XFEL Offline Calibration Documentation',
author, 'EuropeanXFELOfflineCalibration', 'One line description of project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#
# texinfo_appendices = []
# If false, no module index is generated.
#
# texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#
# texinfo_show_urls = 'footnote'
# If true, do not generate a @detailmenu in the "Top" node's menu.
#
# texinfo_no_detailmenu = False
# Example configuration for intersphinx: refer to the Python standard library.
intersphinx_mapping = {'https://docs.python.org/': None}
# first install pandoc if necessary, this is mainly meant for the RTD builds
try:
from nbconvert.utils.pandoc import check_pandoc_version
check_pandoc_version()
print("Pandoc was found!")
except:
print("Installing Pandoc!")
from subprocess import check_call
pandoc_url = "https://github.com/jgm/pandoc/releases/download/2.2.1/pandoc-2.2.1-1-amd64.deb"
pandoc_pack = "pandoc-2.2.1-1-amd64.deb"
check_call(["wget", pandoc_url])
check_call(["dpkg", "-i", pandoc_pack])
rst_exporter = RSTExporter()
with open("available_notebooks.rst", "w") as f:
f.write(dedent("""
.. _available_notebooks:
Available Notebooks
===================
The following notebooks are currently integrated into the European XFEL
Offline Calibration tool chain.
"""))
for detector in sorted(notebooks.notebooks.keys()):
# Avoid having test notebooks in detector notebooks documentations.
if "TEST" in detector.upper():
continue
values = notebooks.notebooks[detector]
f.write("{}\n".format(detector))
f.write("{}\n".format("-"*len(detector)))
f.write("\n")
for caltype in sorted(values.keys()):
data = values[caltype]
nbpath = os.path.abspath("{}/../../../{}".format(__file__, data["notebook"]))
with open(nbpath, "r") as nf:
nb = nbformat.read(nf, as_version=4)
def first_markdown_cell(nb):
for cell in nb.cells:
if cell.cell_type == 'markdown':
return cell
mdcell = first_markdown_cell(nb)
nb.cells = [mdcell] # we only want this single cell
body, _ = rst_exporter.from_notebook_node(nb)
adjusted = []
# adjust titles
for line in body.split("\n"):
if line.startswith("=="):
line = line.replace("=", "+")
if line.startswith("--"):
line = line.replace("-", "~")
adjusted.append(line)
f.write("\n".join(adjusted))
f.write("\n")
f.write("To invoke this notebook and display help use:\n\n")
f.write(".. code-block:: bash\n\n")
f.write(" xfel-calibrate {} {} --help\n\n".format(detector, caltype))
f.write("The full parameter list of this notebook (with defaults is): \n\n")
f.write(".. code-block:: bash\n\n")
nb_help = ["xfel-calibrate", detector, caltype, "--help"]
output = check_output(nb_help).decode('utf8')
f.write(indent(output.replace("DETECTOR", detector).replace("TYPE", caltype), " "*4))
f.write("\n\n")
# add test results
test_artefact_dir = os.path.realpath("../../tests/legacy/artefacts")
def xml_to_rst_report(xml, git_tag, reports=[]):
e = etree.fromstring(xml.encode())
rst = [""]
rst += ["Test execution for {test_name} on {ex_date}"]
test_name, ex_date = e.get("name").split("-")
ex_date = parse(ex_date)
rst[-1] = rst[-1].format(test_name=test_name, ex_date=ex_date)
rst += ["="*len(rst[-1])]
rst += [""]
num_tests = e.get("tests")
num_err = int(e.get("errors"))
num_fail = int(e.get("failures"))
num_skip = int(e.get("skipped"))
# create a summary header
if num_err + num_fail == 0:
rst += [":header-passed:`✓`"]
else:
rst += [":header-failed:`❌`"]
if num_skip > 0:
rst[-1] += ":header-skipped:`⚠`"
rst += [""]
# give a summary
rst += [":Git tag: {git_tag}".format(git_tag=git_tag)]
rst += [":Tests: {num_tests}".format(num_tests=num_tests)]
rst += [":Errors: {num_err}".format(num_err=num_err)]
rst += [":Failures: {num_fail}".format(num_fail=num_fail)]
rst += [":Skipped: {num_skip}".format(num_skip=num_skip)]
rst += [":Duration: {duration}s".format(duration=e.get("time"))]
for rname, rpath in reports:
rst += [":Report: `{} <{}>`_".format(rname, rpath)]
rst += [""]
# now the details
rst += ["Detailed Results"]
rst += ["-"*len(rst[-1])]
rst += [""]
detailed_failures = []
rows = []
for child in e:
if child.tag != "testcase":
continue
name = child.get("name")
extime = child.get("time")
status = ":passed:`passed`"
msg = ""
etype = ""
if len(child): # children are pressent so test failed or skipped
detail = child[0]
etype = detail.get("type")
msg = detail.get("message")
if etype == "skip":
status = ":skipped:`skipped`"
else:
status= ":failed:`failed`"
detailed_failures.append((name, detail.text))
msg = "\n".join(textwrap.wrap(msg, 20))
row = [status, name, etype, msg, extime ]
rows.append(row)
header = ["Result", "Test", "Error", "Message", "Duration (s)"]
tblrst = tabulate.tabulate(rows, headers=header, tablefmt="rst")
rst += tblrst.split("\n")
rst += [""]
for test, report in detailed_failures:
rst += ["Failure report for: {}".format(test)]
rst += ["~"*len(rst[-1])]
rst += [""]
rst += [".. code-block:: python"]
rst += textwrap.indent(report, " "*4).split("\n")
rst += [""]
do_console = False
for child in e:
if child.tag == "system-out" and len(child.text.strip()):
do_console = True
break
if do_console:
# console output
rst += ["Console Output"]
rst += ["-"*len(rst[-1])]
rst += [""]
for child in e:
if child.tag != "system-out":
continue
rst += [".. code-block:: console"]
rst += textwrap.indent(child.text, " "*4).split("\n")
return "\n".join(rst)
def sorted_dir(folder):
def getmtime(name):
path = os.path.join(folder, name)
return os.path.getmtime(path)
sort = sorted(os.listdir(folder), key=getmtime, reverse=True)
return [(s, datetime.fromtimestamp(getmtime(s))) for s in sort]
header = """
Test Results
++++++++++++
Results are organized by git commit, and sorted descending by date.
Contents:
.. toctree::
:maxdepth: 2
"""
if not os.path.exists("./test_rsts"):
os.makedirs("./test_rsts")
with open("test_results.rst", "w") as f:
f.write(header)
for commit, modtime in sorted_dir(test_artefact_dir):
with open("./test_rsts/{}.rst".format(commit), "w") as fr:
rst = [".. include:: roles.rst"]
rst += [""]
rst += ["{} - {}".format(commit[:8], modtime)]
rst += ["+"*len(rst[-1])]
rst += [""]
fr.write("\n".join(rst))
# copy reports
pdfs = glob.glob("{}/{}/*/*.pdf".format(test_artefact_dir, commit))
if not os.path.exists("./_static/reports/{}".format(commit)):
os.makedirs("./_static/reports/{}".format(commit))
reports = {}
for pdf in pdfs:
ppath = "{}/{}.pdf".format(commit, uuid4())
shutil.copyfile(pdf, "./_static/reports/{}".format(ppath))
rloc = pdf.split("/")[-2]
rlist = reports.get(rloc, [])
rname = os.path.basename(pdf).split(".")[0]
rlist.append((rname, "../_static/reports/{}".format(ppath)))
reports[rloc] = rlist
xmls = glob.glob("{}/{}/*/TEST*.xml".format(test_artefact_dir, commit))
for xml in xmls:
rloc = xml.split("/")[-2]
with open(xml, "r") as xf:
xs = xf.read()
rst = xml_to_rst_report(xs, commit, reports=reports.get(rloc, []))
fr.write(rst)
f.write(" test_rsts/{}\n".format(commit))
def setup(app):
app.add_stylesheet('css/test_decorators.css')
docs/source/configuration.rst deleted 100644 → 0
View file @ 2b0ad6ba
.. _configuration:
Configuration
=============
The European XFEL Offline Calibration is configured through `settings.py`
and `notebooks.py` files. Both can be found in `xfel_calibrate` source directory. The `notebook.py` file exposes and configures the
the notebooks by the `xfel-calibrate` command line interface.
Settings
--------
The `settings.py` is a python configuration file, which configures the tool's environment.::
# path into which temporary files from each run are placed
temp_path = "{}/temp/".format(os.getcwd())
# Path to use for calling Python. If the environment is correctly set, simply the command
python_path = "python"
# Path to store reports in
report_path = "{}/calibration_reports/".format(os.getcwd())
# Also try to output the report to an out_folder defined by the notebook
try_report_to_output = True
# the command to run this concurrently. It is prepended to the actual call
launcher_command = "sbatch -p exfel -t 24:00:00 --mem 500G --mail-type END --requeue --output {temp_path}/slurm-%j.out"
A comment is given for the meaning of each configuration parameter.
Notebooks
---------
The `xfel-calibrate` tool exposes configured notebooks to the
command line by automatically parsing the parameters given in the notebooks first cell.
The configuration is given in the form of a python dictionary::
notebooks = {
"AGIPD": {
"DARK": {
"notebook": "AGIPD/Characterize_AGIPD_Gain_Darks_NBC.ipynb",
"concurrency": {"parameter": "modules",
"default concurrency": 16,
"cluster cores": 16},
},
"PC": {
"notebook": "AGIPD/Chracterize_AGIPD_Gain_PC_NBC.ipynb",
"concurrency": "parameter": "modules",
"default concurrency": 16,
"cluster cores": 16},
},
"CORRECT": {
"notebook": "notebooks/AGIPD/AGIPD_Correct_and_Verify.ipynb",
"concurrency": {"parameter": "sequences",
"use function": "balance_sequences",
"default concurrency": [-1],
"cluster cores": 32},
...
}
}
The first key is the detector, e.g. AGIPD. The second key is the calibration type name, e.g. DARK or PC.
A dictionary is expected for each calibration type with a notebook path and concurrency configuration.
For the concurrency three values are expected. Key `parameter` with a value name of type list, which is defined in the first notebook cell.
The key `default concurrency` to define the range of values for `parameter` in each concurrent notebook, if it is not defined by the user.
e.g. `"default concurrency": 16` leads to running 16 concurrent jobs, each processing one module with values of [0,1,2,...,15].
Finally, a hint for the number of `cluster cores` that is used if the notebook is using ipcluster parallelization, only.
This value should be derived e.g. by profiling memory usage per core, run times, etc.
.. note::
It is good practice to name command line enabled notebooks with an `_NBC` suffix as shown in
the above example.
The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function
by setting the `use function` parameter. This function must be defined in the first cell in the notebook
its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters
to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`.
.. note::
The function only needs to be defined, but not executed within the notebook context itself.
docs/source/index.rst deleted 100644 → 0
View file @ 2b0ad6ba
.. European XFEL Offline Calibration documentation master file, created by
sphinx-quickstart on Sun Jun 10 17:32:30 2018.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.
European XFEL Offline Calibration
=================================
The European XFEL Offline Calibration (pyCalibration) is a python package that consists of
different services, responsible for applying most of the offline calibration
and characterization for the detectors.
Running a calibration
---------------------
It utilizes tools such as nbconvert_ and nbparameterise_
to expose Jupyter_ notebooks to a command line interface.
In the process reports are generated from these notebooks.
The general interface is::
% xfel-calibrate DETECTOR TYPE
where `DETECTOR` and `TYPE` specify the task to be performed.
Additionally, it leverages the DESY/XFEL Maxwell cluster to run these jobs in parallel
via SLURM_.
Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are looking
for details on how to use as detector group staff.
If you would like to integrate additional notebooks please see the :ref:`development_workflow`.
.. _nbparameterise: https://github.com/takluyver/nbparameterise
.. _nbconvert: https://github.com/jupyter/nbconvert
.. _jupyter: http://jupyter.org/
.. _SLURM: https://slurm.schedmd.com
Documentation contents:
.. toctree::
:maxdepth: 2
installation
configuration
workflow
available_notebooks
advanced
tutorial
_notebooks/index
testing
.. toctree::
:caption: Reference
:maxdepth: 2
xfel_calibrate_conf
cal_tools_algorithms
calcat_interface
.. toctree::
:caption: Development
:maxdepth: 2
changelog
architecture
Indices and tables
==================
* :ref:`genindex`
* :ref:`search`
\ No newline at end of file
docs/source/installation.rst deleted 100644 → 0
View file @ 2b0ad6ba
.. _installation:
************
Installation
************
It's recommended to install the offline calibration (pycalibration) package on
maxwell, using the anaconda/3 environment.
The following instructions clone from the EuXFEL GitLab instance using SSH
remote URLs, this assumes that you have set up SSH keys for use with GitLab
already. If you have not then read the appendix section on `SSH Key Setup for
GitLab`_ for instructions on how to do this .
Installation using python virtual environment - recommended
===========================================================
`pycalibration` uses the same version of Python as Karabo, which in June 2021
updated to use Python 3.8. Currently the default python installation on Maxwell
is still Python 3.6.8, so Python 3.8 needs to be loaded from a different
location.
One option is to use the Maxwell Spack installation, running `module load
maxwell` will activate the test Spack instance from DESY, then you can use
`module load python-3.8.6-gcc-10.2.0-622qtxd` to Python 3.8. Note that this Spack
instance is currently a trial phase and may not be stable.
Another option is to use `pyenv`, we provide a pyenv installation at
`/gpfs/exfel/sw/calsoft/.pyenv` which we use to manage different versions of
python. This can be activated with ``source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate``
A quick setup would be:
1. ``source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate``
2. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
3. ``pyenv shell 3.8.11`` - load required version of python
4. ``python3 -m venv .venv`` - create the virtual environment
5. ``source .venv/bin/activate`` - activate the virtual environment
6. ``python3 -m pip install --upgrade pip`` - upgrade version of pip
7. ``python3 -m pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
Copy/paste script:
.. code:: bash
source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
pyenv shell 3.8.11
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install . # `-e` flag for editable install, e.g. `pip install -e .`
Creating an ipython kernel for virtual environments
===================================================
To create an ipython kernel with pycalibration available you should (if using a
venv) activate the virtual environment first, and then run:
.. code:: bash
python3 -m pip install ipykernel # If not using a venv add `--user` flag
python3 -m ipykernel install --user --name pycalibration --display-name "pycalibration" # If not using a venv pick different name
This can be useful for Jupyter notebook tools as https://max-jhub.desy.de/hub/login
SSH Key Setup for GitLab
========================
It is highly recommended to set up SSH keys for access to GitLab as this
simplifies the setup process for all of our internal software present on GitLab.
To set up the keys:
1. Connect to Maxwell
2. Generate a new keypair with ``ssh-keygen -o -a 100 -t ed25519``, you can
either leave this in the default location (``~/.ssh/id_ed25519``) or place it
into a separate directory to make management of keys easier if you already
have multiple ones. If you are using a password for your keys please check
this page to learn how to manage them: https://docs.github.com/en/github/authenticating-to-github/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent#adding-your-ssh-key-to-the-ssh-agent
3. Add the public key (``id_ed25519.pub``) to your account on GitLab: https://git.xfel.eu/gitlab/profile/keys
4. Add the following to your ``~/.ssh/config`` file
.. code::
# Special flags for gitlab over SSH
Host git.xfel.eu
User git
Port 10022
ForwardX11 no
IdentityFile ~/.ssh/id_ed25519
Once this is done you can clone repositories you have access to from GitLab
without having to enter your password each time. As ``pycalibration``
requirements are installed from SSH remote URLs having SSH keys set up is a
requirement for installing pycalibration.
docs/source/makeAllDocs.sh deleted 100755 → 0
View file @ 2b0ad6ba
#!/bin/bash
# first convert notebooks to rst files
# the list of notebooks to convert:
notebooks=(calversion)
mkdir _notebooks
cd _notebooks
echo "Tutorial Example"> index.rst
echo "================">> index.rst
echo "" >> index.rst
echo "Tutorial Notebook:" >> index.rst
echo "" >> index.rst
echo ".. toctree::" >> index.rst
echo " :maxdepth: 5" >> index.rst
echo " :titlesonly:" >> index.rst
echo "" >> index.rst
for nb in ${notebooks[*]}
do
jupyter nbconvert --to rst ../../../notebooks/Tutorial/${nb}.ipynb --output-dir=./
##fix code blocks
sed -i.bak 's/code::/code-block::/g' ${nb}.rst
sed -i.bak 's/ipython3/python/g' ${nb}.rst
sed -i.bak 's/raw:: latex/math::/g' ${nb}.rst
sed -i.bak 's/``/`/g' ${nb}.rst
mv ${nb}.rst ${nb}.rst.end
echo ".. _${nb}:" > ${nb}.rst
cat ${nb}.rst.end >> ${nb}.rst
rm ${nb}.rst.end
echo " ${nb}" >> index.rst
done
rm *.bak
#cd .. rm api/* sphinx-apidoc -o ./api/ -E ../../iCalibrationDB/
docs/source/test_rsts/roles.rst deleted 100644 → 0
View file @ 2b0ad6ba
.. role:: header-passed
.. role:: header-skipped
.. role:: header-failed
.. role:: passed
.. role:: skipped
.. role:: failed
docs/source/testing.rst deleted 100644 → 0
View file @ 2b0ad6ba
Reproducibility
***************
The `test` directory contains tests which can ensure that updates to
notebooks and libraries do not result in unintended changes to
notebook output. This assures consistency of correction results
for subsequent versions.
.. note::
Tests can be quite resource intensive, and thus should be run
on a dedicated cluster node, allocated using `salloc`.
Running Tests
+++++++++++++
Before you run tests, commit your changes, so that the test
run can be assigned to that commit::
git add ...
git commit -m "Added test section to docs"
To run all tests, navigate to the test directory and execute::
python -m unittest discover
This will usually entail executing a notebook under test via SLURM
first, then checking its output against the last commited artefacts
of that test type.
If individual tests are run, e.g. for debugging, additional options
exist to skip tests, or notebook execution::
python test_XXX.py --help
where `test_XXX.py` is the test name, will give you a list of options
available for that test.
If all tests pass, you can commit and push your updates. If you have
failures, either check your changes, or if changes are intended,
generate new artefacts.
.. note::
Running tests will generate entries for test reports in the
artefacts directory under the most recent commit.
Reviewers should check that such updates are present in the
list of changed files.
Generating new Artefacts
++++++++++++++++++++++++
If an update intents to change output, the tests can be used to
generate new artefacts against which subsequent tests will then run.
First, commit your changes which you want to produce new artefacts
for::
git add ...
git commit -m "AGIPD corrections handle baseline shift"
Contrary to running tests alone, new artefacts need to be generated
for each affected test individually::
python test_XXX.py --generate
replacing `test_XXX.py` with the test you'd like to run. This
will execute the notebook, create artefact entries in the artefact
dir, and after that will check for consistency by executing the test against
these artefacts. This last part is important: the test should not
fail on its own input. If it does, something is very likely wrong!
After artefacts are created and tests using these have passed,
commit the new artefacts and create a merge request for your branch::
git add tests/artefacts/
git commit -m "Added new artefacts for changes related to baseline shifts"
Please also add comments in the MR description on why artefacts have
changed.
.. note::
Reviewers should always evaluate if the changes in test artefacts are
appropriate, intended and acceptable.
Test Reports
++++++++++++
Test reports are automatically generated when building documentation
from all xml report files found in sub-directories of the artefact
directory.
.. note::
Please make sure not to commit any additional files into the
`test_rsts subfolder` of this documentation. Also, do not commit
`test_results.rst`. It is autogenerated.
Test Data
+++++++++
In order to perform described test a detector data as well as calibration constants are required. Detector data for use in testing as well as calibration constants can be found in::
/gpfs/exfel/exp/XMPL/201750/p700001/raw/
Tests should be configured to output into a common location::
/gpfs/exfel/exp/XMPL/201750/p700001/scratch/
Repositories of calibration constants used in testing can be found at::
/gpfs/exfel/exp/XMPL/201750/p700001/usr
.. include:: test_results.rst

Legal Notice | Contacts