add images and references

.gitignore

@@ -8,8 +8,6 @@
 *.npy
 *.out
 *.pkl
-*.png
-*.png
 *.secrets.yaml
 *.so
 *.tar

docs/references/changelog.md

+# Release Notes
+
+## 3.7.5
+
+16-11-2022
+
+-   `JUNGFRAU` Fix index location for ROI instrument output
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/765>
+
+## 3.7.4
+
+
+15-11-2022
+
+-   `JUNGFRAU` Only emit a warning if all sequences of a job are empty
+-   `JUNGFRAU` Fix storing multiple ROIs for a single module
+-   `JUNGFRAU` Replicate a raw dataset in processed data for legacy analysis code
+
+## 3.7.3
+
+10-11-2022
+
+-   `DataFile` Add support for older INDEX and METADATA versions.
+-   `LPD` Fix histogram plotting if selected cell's data is identical
+    for trains.
+-   `JUNGFRAU` Avoid unclear error messages when the run has no
+    trains.
+-   `AGIPD` Change notebook-default setting of max-task-per-worker
+    to 1.
+-   `REMI` Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses.
+-   `webservice` Fix timestamps in the overview page.
+-   `webservice` Check run type more directly in myMdC response.
+
+## 3.7.2
+
+28-10-2022
+
+-   `JUNGFRAU` Add support for strixel sensors
+-   `JUNGFRAU` Shorter summary notebook for darks by default
+-   `JUNGFRAU` Always align corrected data properly
+-   `JUNGFRAU` Harden against 0 adc data in dark characterization
+-   `REMI` Add support for quad nodes and various pulse separation
+    improvements
+-   `AGIPD` Allow manual photon energy input for photonization and add
+    consistency plots
+-   Grant priority partitions for set-up time
+
+## 3.7.1
+
+14-10-2022
+
+-   `EPIX100` Add Common mode corrections to dark characterization
+-   `EPIX100` Use a single value for temperature
+-   `JUNGFRAU` Fix bad pixel masking
+-   `JUNGFRAU` Limit number of plotted trains to 500 during
+    corrections
+-   `AGIPD` Don't fail when there are no trains in sequence file
+-   `REMI` Use DataFile API and support asymmetric pump-probe patterns
+-   Skip corrections for runs which match specific types
+
+## 3.7.0
+
+07-09-2022
+
+-   `JUNGFRAU` Mask double sized pixels
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/689>
+-   `LPD` Storing retrieved calibration constants in
+    calibration_metadata.yml.
+-   Storing the running logs in the report folder path instead of a
+    [temp] folder wherever the processing ran from.
+-   Selecting partitions based on beam time dates.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/725>
+-   Creation time is not read from myMDC anymore.
+
+## 3.6.4
+
+20-08-2022
+
+-   `AGIPD` Significant improvements to AGIPG LitFrameFinder
+    implementation, including offline support.
+-   `AGIPD` Fix misalignment of data and gain/mask when common mode is
+    used with manual pulse slicing.
+
+## 3.6.3
+
+23-08-2022
+
+-   `AGIPD` litframe finder configs in update_config
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/723>
+
+## 3.6.2
+
+15-08-2022
+
+-   `EPIX100` - `JUNGFRAU` - `pnCCD` Retrieve constants precorrection
+    notebooks.
+-   `AGIPD` skip sanitization
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/654>
+-   `AGIPD` New PC notebook
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/604>
+-   `JUNGFRAU` Allow selecting ROIs to save 1D projections
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/699>
+
+## 3.6.1
+
+26-07-2022
+
+-   `GOTTHARD2` Correction and Dark notebooks.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/658>,
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/684>,
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/707>
+-   pin [xarray] to 2022.3.0
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/706>
+
+## 3.6.0
+
+20-07-2022
+
+-   `LPD` - `DARK` Save bad pixel dark data as uint32, not float64.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/698>
+-   `LPD` - `CORRECT` Add option to disable Oauth when using CalCat
+    proxy.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/681>
+-   `AGIPD` - `LPD` - `DSSC` - `DARK` Avoid printing errors for expected
+    missing constant files.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/664>
+-   `TESTS` Refactor and split [AGIPDCtrl] methods (old
+    and new RAW data versions.) and add unit tests
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/651>,
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/646>
+-   `webservice` Monitor SLURM jobs in a separate process.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/668>
+-   `webservice` Restructure the SLURM database to give more
+    meaningful success/failure information.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/683>
+-   `XFEL-CALIBRATE` Recreate environments on demand for
+    xfel-calibrate-repeat.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/660>
+-   Upgrade the Jupyter packages to the latest version.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/697/diffs>
+-   Update Extra-data to 1.12:
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/693>
+    -   [EXtra-data] Disable [_use_voview] to be able to
+        use [.files] or avoid using [.files]
+        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/682>,
+        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/688>
+
+## 3.5.5
+
+15-06-2022
+
+-   `AGIPD` - `CORRECT` Expose max tasks per pool worker.
+
+## 3.5.4
+
+13-06-2022
+
+-   `AGIPD` Convert bias_voltage parameter condition to integer in
+    cal_tools.
+-   `LPD` Fix correcting a single pulse.
+-   `LPD` VCXI require 4 modules.
+
+## 3.5.3
+
+19-05-2022
+
+-   `LPD` - `CORRECT` Optionally create virtual CXI files
+-   `LPD` - `CORRECT` Expose max-nodes parameter
+-   `AGIPD` Replace gain_choose_int by fused types
+-   Fix missing install of restful_config.yaml
+-   Fix use of xfel-calibrate --skip-report
+
+## 3.5.2
+
+16.05.2022
+
+-   `LPD` - `CORRECT` New correction notebook for `LPD`
+-   New [files] module to write European XFEL HDF5 corrected
+    data files.
+
+## 3.5.1
+
+05-04-2022
+
+-   Calibration Constant version's new [Variant] file
+    attribute. To indicate method of handling the constant post
+    retrieval.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/619>
+-   `Epix100` dark Badpixels Map.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/637>
+-   `skip-plots` flag to finish correction before plotting.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/635>
+-   First trainId's timestamp as RAW data creation_time, if there is
+    myMDC connection.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/630>
+-   `AGIPD` correction can correct one cellId without plotting errors.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/642>
+-   Fixed mode relative gain constants in `Jungfrau` can be retrieved.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/639>
+-   Only instrument source is selected to check number of trains to dark
+    process.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/636>
+-   `AGIPD` trains for dark processing is selected for each module
+    individually.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
+-   Produce report after trying to correct `AGIPD` run with no images.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/641>
+-   `AGIPD`'s bias voltage for AGIPD1M is read from slow data.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/647>
+-   Removed psutil dependency.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
+-   Update Pasha to 0.1.1
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/638>
+
+## 3.5.0
+
+01-03-2022
+
+-   Updating Correction and dark notebooks for `JUNGFRAU`:
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/518>
+-   Updating Correction and dark notebooks for `AGIPD`:
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
+-   Updating Correction and dark notebooks for `PnCCD`:
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/559>
+-   Updating Correction and dark notebooks for `ePix100`:
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/500>
+    -   `EXtra-data` is integrated to read files in pycalibration for
+        `AGIPD`, `JUNGFRAU`, `ePix100`, and `PnCCD`. Dark and Correction
+        notebooks.
+    -   Pasha is now used for processing data for `JUNGFRAU`, `ePix100` and
+        `PnCCD`.
+    -   pyDetLib correction functions were removed (except for
+        common-mode correction).
+    -   [db-module] is useless now for `JUNGFRAU`, `ePix100`, and
+        `PnCCD`. Some parameters were updated in dark and correction
+        notebooks for the mentioned detectors.
+-   `gain_mode` and burst mode are now available for
+    `JUNGFRAU`.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
+-   `JUNGFRAU` has now a new badpixel value,
+    `WRONG_GAIN_VALUE`.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
+-   Pass through available for testing in-progress ORCA service.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests?scope=all&state=merged&search=orca>
+-   Non-calibrated RAW h5files are no longer copied.
+-   High priority partitions ([upex-high and
+    upex-middle]) are used for runs from ACTIVE and READY
+    proposals, only.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/628>
+-   Supporting to disable LPD Correction through the webservice.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/629>
+-   Compatibility for old DAQ files for REMI is added.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/607>
+-   server-overview refactors.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/593>
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/589>
+-   AGIPD correction notebook support AgipdLitFrameFinder device.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/603>
+-   Parsing code arguments in xfel-calibrate is refactored.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/575>
+-   skip-plots option for AGIPD.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/581>
+-   Native implementation for transposition of constants AGIPD.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/580>
+-   Trains for AGIPD can be selected for correction.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/574>
+-   Skip report flag in xfel-calibrate.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/579>
+-   Fix ReadTheDocs.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/448>
+-   Fix error reporting for re-injecting the same CCV.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/627>
+-   Fix AGIPD for legacy runs without `gain_mode`.
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/617>
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/626>
+-   Pinning markupsafe version 2.0.1
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/631>
+-   Pinning psutil 5.9.0
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
+-   Updating Extra-data to 1.9.1
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
+-   Updating h5py to 3.5.0
+    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/602>
+
+## 3.4.3
+
+20-10-2021
+
+-   Update pyDetLib tag.
+-   Add explicit dependencies on matplotlib, scipy.
+-   Remove outdated matplotlib rcParams setting.
+-   Update EXtra-geom to 1.6.
+-   Remove cluster_profile parameter from notebooks which don't use
+    it.
+-   Fix checking availability for the concurrency parameter.
+-   Fix launching work directly (not via Slurm).
+-   Fix `sphinx-rep` temp folder recreation, if sphinx-rep
+    already existed.
+-   Fix missing string conversion for slurm-scheduling argument.
+-   Fix title reports for multiple detectors per run folder.
+-   Append to .out files for preemptable finalize job.
+-   `AGIPD` `CORRECT` Reuse previously found constants.
+-   `AGIPD` Fix missing memory cell index in SlopesPC constant
+    sanitization.
+-   `AGIPD` Only use bad pixels from darks in
+    agipdutils.baseline_correct_via_stripes.
+-   `AGIPD` `DARK` Use function to get list of karabo_da from run
+    for making Slurm jobs.
+-   `EPIX100` - `CORRECT` Set absolute_gain to false if relative gain
+    was not retrieved.
+-   `JUNGFRAU` Fix running for multiple modules and flip logic for
+    do_relative_gain.
+-   `JUNGFRAU` Style changes for Dark and Correct notebooks.
+-   `REMI` Add notebook to reconstruct detector hits from raw data.
+-   `webservice` Check run migration status using MyMDC.
+-   Resolve \"Skip ZMQ tests if zmq connection for calibration DB not
+    available\".
+-   Reproducibility, step 1.
+
+## 3.4.2
+
+17-09-2021
+
+-   Remove driver=core from all notebook
+-   `webservice` Make use of Dynaconf for managing secrets.
+-   `webservice` Make use of dedicated slurm partitions.
+-   `webservice` Handle missing migration information (missing
+    user.status fattr).
+-   `webservice` Implement, raise, and catch, migration errors to send
+    mdc messages.
+-   `webservice` Simplify handling of user notebook paths.
+-   `webservice` Update princess to 0.4 (use Unix sockets).
+-   `webservice` Update MyMDC with begin and end times.
+-   `webservice` create output folder before copying slow data.
+-   `AGIPD` `CORRECT` read acq_rate from slow data.
+-   `AGIPD` - `CORRECT` Set default memory cells to 352.
+-   `AGIPD` `CORRECT` Set maximum pulses to correct based on file
+    content.
+-   `AGIPD` - `FF` Correctly label legends in figures.
+-   `AGIPD` - `FF` Add HIBEF AGIPD500K and fix some issue with
+    retrieval of conditions.
+-   `JUNGFRAU` Add Gain setting to Jungfrau notebooks.
+-   `JUNGFRAU` Fix max gain plot in LPD correct notebook
+-   `JUNGFRAU` `DARK` Clearer error message for Jungfrau Dark
+    notebooks no suitable files are found
+-   `LPD` `CORRECT` Fix max gain plot.
+-   `EPIX100` `CORRECT` Solve conflict between gain correction and
+    clustering
+
+## 3.4.1
+
+
+16-07-2021
+
+-   Update h5py to 3.3
+-   Stop execution on notebook errors
+-   `AGIPD` Add integration time as operating condition to all
+    notebooks
+-   `webservice` Add blocklist pattern when copying untouched files in
+    webservice.
+-   `webservice` Expose dark configurations in update_config.py
+-   Fix MetadataClient.get_proposal_runs arguments call.
+-   Fix Use snapshot for injecting constants for old PDU mappings
+-   Fix the old time-summary (creation time for retrieved constants)
+-   Update documentation notes on venv installation
+-   Ignore all .so files in gitignore
+
+## 3.4.0
+
+28-06-2021
+
+-   Update to Python 3.8.
+-   Bump numpy to 1.20.3 and remove fabio.
+-   remove PyQT dependency.
+-   Disable dark requests from serve overview.
+-   Update report upload parameter key.
+-   Override locale to always use UTF-8.
+-   Assorted cleanup of xfel-calibrate.
+-   Fix pre-commit.
+-   Use argparse only if name is main, call main with args dict.
+-   `webservice` Use full hostname for webservice overview.
+-   `webservice` Show clearer messages when running webservice in sim
+    mode.
+-   `webservice` Fix filename lineno and typos in webservice logs.
+-   `webservice` Fix creating an extra run folder in run output
+    folder.
+-   `AGIPD` Parallelize gain/mask compression for writing corrected
+    AGIPD files.
+-   `AGIPD` - `DARK` Fix processing empty sequence files.
+-   `AGIPD` - `PC` - `FF` Update notebooks with new CALCAT mapping.
+-   `AGIPD` - `JUNGFRAU` Use all available sequences for processing
+    darks for AGIPD and Jungfrau.
+-   `AGIPD` - `LPD` - `DSSC` Fix retrieve old constants for comparison
+    for modular detectors.
+-   `LPD` Fix data paths in LPD notebook.
+-   `REMI` Fix user notebook path for REMI correct notebook
+    provisionally.
+-   `EPIX100` - `CORRECT` Add Common mode correction.
+-   Fix plotting-related warnings.
+-   Test update config.
+-   Test get_from_db and send_to_db.

docs/references/faq.md

+# Frequently Asked Questions

docs/references/glossary.md

+# Glossary
+
+- PDU: Physical Detector Units - The detector module hardware name given by detector expert or manufacturer.
+
+- Detector UUID:
+