changelog.md

# Release Notes

## 3.15.0

Moving to python 3.11: updating dependencies and update calibration notebooks and modules accordingly

# 3.14.4

[AGIPD][Correct] New feature for lit-pixel counter
[AGIPD][Correct] Refactor write_file on AgipdCorrections to create indexes using the DataFile
[AGIPD][PC] Fixing plotting in summary notebook
[AGIPD][LPD] Add DataFile.create_legacy_source
[LPD][Correct] Write corrected data as new source and link to old source name
[Jungfrau][Dark] use CalibrationData to retrieve prior CCVs
[Jungfrau][Correct] Test ROI for JF1M
[Jungfrau][Correct] use new correct data source and link to old data source
[Jungfrau] roi-definitions to update_config.py script
[PNCCD][Correct] New corrected data source and link to legacy source
[Epix100][Correct] New corrected data source and a link to old data source
[Gotthard2]: Avoid getting wrong data sources or including control devices' sources.
[xfel-calibrate]: expose mincpus argument as --slurm-mincpus
[Fix] Pin the pillow version
[Test] Configure and Include new test runs
[Test] Swap order of dicts for DeepDiff
[Webservice] Remove extra slash from reports-folder config
[Webservice] Use MUNGE credential with update_conf requests
[CalCatAPI] port markdown tables of found constants
[CalCatAPI] Convert digits and boolean to float and the rest to string
Make constant tables in both markdown & latex format

## 3.14.3

Update report creation to use texlive/2022
fix: Update documentations after EL9 update

## 3.14.2

[AGIPD][Correct][CS] Fix badpixelsCS before we have multiple CCVss to have same BadPixels shapes.
[AGIPD] Modernize use of extra_data to get AGIPD control data
[AGIPD][Correct] use np.nanmin and np.nanmax to avoid nans when modules are missing
[AGIPD][Correct] bug when plotting gain data of only 0 after it's info was discarded by DAQ
[Shimadzu][Correct] fix for less expected images
Remove unnecessary 'module load' command from jobs to be repeated
Remove reference to anaconda/3 module
flake8 repository has moved from gitlab to github

## 3.14.1

[AGIPD][CORRECT][DARK] Break wrong assumption for availability for 1st module.
[AGIPD][CS] new current source calibration constant.
[AGIPD][CS] Correction using Current source slopes.
[Shimadzu][CORRECT] Fix name of index group in output files.
[JUNGFRAU][FF] Feat: new notebook for producing gain constants.
Feat/ci new runners python311.
Create an unreadable temp directory for testing PermissionError.

## 3.14.0

- [shimadzu] Add support ofr PCA-based dynamic flat-field corrections of Shimadzu HPVX2 data.
- [AGIPD] Fix PC processing to allow for a single module.
- [AGIPD] Remove default value for caldb_root.
- Add staging version of new CalCat API to retrieve calibration data (to be superseded by extra.calibration).
- Add API to create calibration constant data files and inject directly into CalCat.
- Configure number of required characterization runs from CalCat based on operation mode.

## 3.13.0

- [AGIPD][PC] Fixing ValueError.
- [Gotthard2][Dark] Sort dark runs.
- [ePix100][Correct] Correcting one train for epix100 and storing a list of one pulseId.
- [Doc] Change log update with latest releases.
- [Gotthard2][Correct] Add `idx` to `da_to_pdu` to use in picking correct `data_source`.
- [LPD][Dark] Sort Dark runs by Gain.
- [Webservice] Use AW status on myMdC for warnings that prevent launching correction.
- [Webservice] Move list of three gain/run detectors into webservice config yaml.
- [Webservice] add environment bin/ to PATH if not already there.
- Convert request time to local timezone in report.
- Don't keep DEBUG level logging from requests_oauthlib.
- Avoid using `run_metadata()` by default, to not fail for EXDF-v0.5 files.

## 3.12.6

- [Jungfrau][Correct] Avoid NAN disturbance for correction plots scale
- [Timepix] Updating Timepix3 Calibration Pipeline for compatibility with new Karabo Implementation + some general improvements
- [Timepix] Allow pipelines to be part of nodes in DataFile API
- [Timepix] Throw pasha at it
- [Timepix] Store labels assigned to pixels

## 3.12.5

- SPARTA AGIPD Single Module integration

## 3.12.4

- [Gotthard2][Correct] Store mask data properly for gotthard2 25um
- Make the run type regex for skipping JF darks less aggressive

## 3.12.3

- [webservice] Update Kafka config to use EuXFEL broker

## 3.12.2

- [GH2] Temporary workaround for processing HIREX detector's dark runs using the correct data sources
- [GH2][Correct] Show number of available trains and raise ValueError for runs with no trains
- [GH2][Correct] New paramater to reverse second GH2 25um module
- [Jungfrau][Correct] Fix not returning after finding a dark fixed gain constants
- [Jungfrau][Dark][Correct] Improve Jungfrau plots
- [Webservice] Add new detector HIREX to 3 dark runs detectors

## 3.12.1

- [AGIPD][PC] Summary notebook for PC processing
- [GH2][Correct][Dark] Break assumptions on receiver names
- [Webservice] Create correction reports in MyMdC when jobs have finished
- [Webservice] Use leading zeros in usr/Reports folders
- Make serve_overview resistant against missing karabo-da in data-mapping
- Add links to DESY grafana for running jobs in serve_overview

## 3.12.0

- [AGIPD][Correct] Bug for using drop with a list in AGIPD CORR NB
- [AGIPD][Correct] Try to simplify & speed up file reading code
- [GH2][Correct][DARK] Feat/add support for gh2 25um
- [Jungfrau][Correct] Update rows A1256 LUT after rotation
- [Jungfrau][Dark] Account for dark runs not taken by the MDL device
- [JUNGFRAU][Dark] Fix bad pixel constant from darks taken in burst mode
- [REMI] Add and enhance plots, more documentation, clean-up and some deprecation
- [TEST] Update AGIPD tests with xray-gain, add DSSC, ePix100 tests and update reference folder path
- [TEST] clearer comparison of HDF5 files
- DataFile: Don't write creationDate & updateDate unless specified

## 3.11.5

- Update CalParrot==0.3 and EXtra-data==1.15.1 dependencies
- [DSSC][Dark][Correct] No longer restrict memory cells to a multiple of 100s and add lower deviation for memory cells parameter conditions.
- [Webservice] Catch errors on failure to launch dark processing
- Add script to update dark run status in myMdC
- [Epix100][Correct] Calcat error when no gain is retrieved
- [REMI] Disable trailing trigger by default

## 3.11.4
- [Jungfrau][Correct] Force fixed gain for JF data in burst mode

- [Jungfrau][Correct] Force replacement for gain value in Jungfrau correction
- [DSSC] Allow 900 memory cells for DSSC darks to workaround appearance of cell 810

- [Jungfrau][Dark] Reflect WRONG_GAIN_VALUE over a pixel in all gain for badpixels map
- [AGIPD][Dark] Add timings
- [Jungfrau][Correct] New A1256 JF Strixel

- [webservice] Add JUNGF and PEP 8 on the line


## 3.11.3
- [AGIPD][LPD][DARK] Show table for bad pixels bitmaps
- [AGIPD][CORRECT] Process all AGIPD trains if the PPU device is missing or if no trigger
- [AGIPD][DARK] Sort dark runs
- [AGIPD][DARK] Improvements for reading conditions by creating a new data class for multiple runs
- [AGIPD][FF] Fixing FF summary performance plots

- [Jungfrau] [Correct] Add thresholding for ROI projections
- [Jungfrau][Correct][Dark] Fix manual edit for operating conditions
- [Jungfrau][DARK] Validate and reorder dark runs before processing
- [EPIX][FF] ePixFF characterization

- [REMI] Add support for virtual trailing trigger

- Fix manually submitting the confirmation

- move some logs to DEBUG and extend the report sleep

- Expose --blc-stripes to update_config.py and fix old parameter names

## 3.11.2
- Operational release for SPB to support configurable rounding thresholds.

## 3.11.1
- [AGIPD][CORRECT] Use calcat_interface and remove precorrection notebook
- [EPIX100] Feat: Compliance with update to receiver device
- [REMI] Various fixes and improvements for quad DLDs
- [REMI] Fix missing re-allocation of trigger array with neither FEL nor PPL
- [Tests] Fix: Accept uppercase calibration type
- [Test] Find difference by default
- Clearer error when xfel-calibrate would run no jobs
- Detect cycle automatically in update_config script
- Fix link to CalCat

## 3.11.0

- [AGIPD][Correct] Handle selecting multiple trains per PPU trigger
- [AGIPD][Dark] Fix: Skip corrupted frame from dark processing
- [LPD1M] Automatically decide whether to inject & use memory cell order
- [LPD1M][Dark] Use EXtra-data to create darks from >1 sequence file
- [LPD1M][Correct] Use parameter names instead of IDs to find constants
- [LPD1M][Correct] Using CALCAT interface
- [LPD1M][Correct] Fix: Constant type conversion
- [LPD1M][Correct] Use the fragment file and remove the precorrection notebook
- [LPD-Mini] Rework cell order condition to match LPD-1M again
- [JUNGFRAU][pnCCD][ePix100] Feat: new method to display CCV metadata in reports
- [JUNGFRAU][CORRECT] Add fragment file and remove precorrection notebook
- [EPIX][DARK] Mark dead pixels as Bad Pixels
- [EPIX][CORR] Optimize histograms and plots
- [GH2][Correct] Move false warning and disable gain correction as printed.
- [GH2][Correct] Remove the precorrection notebook and add fragment
- [TIMEPIX] Add select parameters to update_config
- [TIMEPIX] Fix types of notebook arguments
- [xfel-calibrate] Fix: Break the line properly into latex when the next line starts with `_`
- [Webservice] Don't mark jobs as finished just because they disappear from `squeue` output
- [Webservice] Use status AW in myMdC if correction failed for some detectors in a run
- Add a pytest to run a dict of CALLAB test runs before releases
- Look up CCVs using parameter_name in place of parameter_id
- Replace `max-exfl016` and `max-exfl017` into `max-exfl-cal001` and `max-exfl-cal002`, respectively.
- Make metadata directory name match report filename
- Add reorder_axes function

## 3.10.3

- [LPD][Correct] Harden against empty sequencee sets with train-on-demand
- [JF][correct] Add missing gain mode parameter
- [Timepix3] Add centroiding notebook

## 3.10.2

- [PNCCD][CORRECT] Fix: Skip error for missing gain
- [PNCCD][CORRECT] Fix: Hack to wrong ctrl bias voltage values p002857

- [LPD][Correct] Fix axis order for LPD-1M RelativeGain constant

- [LPDMini][Dark] Add only number of available data trains into data_samples
- [LPDMini][Dark] Fix first notebook cell to execute CL through the webservice
- [LPDMini] Feat: Inject gain constants notebook

- [Jungfrau] Workaround for `SPB_CFEL_JF1M` as the modules start with `09` not `01`

- Fix update_config to work with non-AGIPD and add REMI

## 3.10.1

- [JUNGFRAU][CORRECT] Using calcat interface
- [JUNGFRAU][CORRECT][DARK] Extend accepted detectors based on substrings of karabo_id

- [LPD Mini][CORRECT][DARK] Initial work on LPD Mini notebooks
- [PNCCD][CORRECT] Avoid raising a CalCat error while retrieving metadata for missing gain constant from DB

## 3.10.0

- [ePix100][Correct] Avoid including histogram calculator with empty array
- [ePix100][Correct] Remove pre notebook

- [pnCCD][Correct] Using calcat interface
- [pnCCD][Correct] Record fragment file and remove pre correction notebook
- [ePix100][pnCCD][Correct] Display creation time for retrieved constants

- [AGIPD][Correct] error out only if all sources are empty
- [AGIPD][Correct] exit notebook if no correction files are found for the selected sequence

- [DSSC][DARK] Group all slow data to the same aggregator

- [JUNGFRAU][Correct] Use DataCollection.from_paths for reading JF CORR files for plots
- Support for saving metadata fragments & merging into calibration_metadata.yml

- [REMI] Save pulse amplitudes during discrimination
- [REMI] Make plots robust against no edges or no hits in data

- [Webservice] Add script to check run in webservice DB
- [Webservice] Fix database lock timeouts

## 3.9.2

- [AGIPD] Adding back overwrite parameter
- [AGIPD] Another hotfix for experimenting DAQ filtering at SPB

## 3.9.1

- Hotfix version for 3.9.0
- [AGIPD] LitFrameFinder bug fix for experimenting DAQ filtering at SPB

## 3.9.0

- [GH2][CORRECT]CALCAT interface.
- [ePix100][CORRECT]CALCAT interface.
- [pnCCD][CORRECT]Use `DataFile` to store aligned corrected data.
- [GH2][CORRECT]Use `DataFile` to store aligned corrected data.
- [ePix100][CORRECT]Use `DataFile` to store aligned corrected data.


- [AGIPD][SlopesFF][CORRECT]Add the deviation for all possible memory cells and enable correction using
these FF constants with more memory cells.
  - https://git.xfel.eu/calibration/pycalibration/-/merge_requests/806
  - https://git.xfel.eu/calibration/pycalibration/-/merge_requests/613
- [AGIPD][CORRECT]Fix checking ccv_variant condition for AGIPD.
- [DSSC][Dark]Remove unused instrument parameter from DSSC dark notebook
- Update `nbparameterise` to 0.6.
- Update `EXtra-redu` to version 0.0.7

## 3.8.1

- Hotfix version for 3.8.0 with
- update extra_redu version

## 3.8.0
-----

- [AGIPD][CORRECT] Clean up before AGIPD calcalt_interface changes
- [AGIPD][FF] Styling modification for both FF notebooks:1st nb cell, and removing unneeded imports
- [LPD] uses memory cell order as a condition for constants
- [LPD] [Correct] Don't pass default snapshot_at=None to calibration_client
- [LPD][JF][DataFile] Add parallel dataset compression
- [JF] Replace strixel cython code by NumPy implementation
- [EPIX100][CORRECT] Include gain and charge sharing to corrected data.
- New CALCAT interface
- Store CalCat requests/responses for reproducibility
- Update to calibration_client 11.2.0

## 3.7.6

- [AGIPD][CORRECT] Fix common mode correction: array reshaping and hardcoded 256 trains per file.
- [AGIPD][LFF] Improve lit frame selection
- [AGIPD][CORRECT] Fix common mode correction
  - Array reshaping.
  - Hardcoded 256 trains per file.

- [AGIPD][LFF] Improve lit frame selection:

  - fix the misalignment of selection by trains.
  - introduce super pattern selection.
  - guess missed patterns for patterns that repeat with constant step.
  - shows patterns that repeat with a constant step as one line in the report.


## 3.7.5

16-11-2022

-   `JUNGFRAU` Fix index location for ROI instrument output
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/765>

## 3.7.4


15-11-2022

-   `JUNGFRAU` Only emit a warning if all sequences of a job are empty
-   `JUNGFRAU` Fix storing multiple ROIs for a single module
-   `JUNGFRAU` Replicate a raw dataset in processed data for legacy analysis code

## 3.7.3

10-11-2022

-   `DataFile` Add support for older INDEX and METADATA versions.
-   `LPD` Fix histogram plotting if selected cell's data is identical
    for trains.
-   `JUNGFRAU` Avoid unclear error messages when the run has no
    trains.
-   `AGIPD` Change notebook-default setting of max-task-per-worker
    to 1.
-   `REMI` Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses.
-   `webservice` Fix timestamps in the overview page.
-   `webservice` Check run type more directly in myMdC response.

## 3.7.2

28-10-2022

-   `JUNGFRAU` Add support for strixel sensors
-   `JUNGFRAU` Shorter summary notebook for darks by default
-   `JUNGFRAU` Always align corrected data properly
-   `JUNGFRAU` Harden against 0 adc data in dark characterization
-   `REMI` Add support for quad nodes and various pulse separation
    improvements
-   `AGIPD` Allow manual photon energy input for photonization and add
    consistency plots
-   Grant priority partitions for set-up time

## 3.7.1

14-10-2022

-   `EPIX100` Add Common mode corrections to dark characterization
-   `EPIX100` Use a single value for temperature
-   `JUNGFRAU` Fix bad pixel masking
-   `JUNGFRAU` Limit number of plotted trains to 500 during
    corrections
-   `AGIPD` Don't fail when there are no trains in sequence file
-   `REMI` Use DataFile API and support asymmetric pump-probe patterns
-   Skip corrections for runs which match specific types

## 3.7.0

07-09-2022

-   `JUNGFRAU` Mask double sized pixels
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/689>
-   `LPD` Storing retrieved calibration constants in
    calibration_metadata.yml.
-   Storing the running logs in the report folder path instead of a
    [temp] folder wherever the processing ran from.
-   Selecting partitions based on beam time dates.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/725>
-   Creation time is not read from myMDC anymore.

## 3.6.4

20-08-2022

-   `AGIPD` Significant improvements to AGIPG LitFrameFinder
    implementation, including offline support.
-   `AGIPD` Fix misalignment of data and gain/mask when common mode is
    used with manual pulse slicing.

## 3.6.3

23-08-2022

-   `AGIPD` litframe finder configs in update_config
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/723>

## 3.6.2

15-08-2022

-   `EPIX100` - `JUNGFRAU` - `pnCCD` Retrieve constants precorrection
    notebooks.
-   `AGIPD` skip sanitization
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/654>
-   `AGIPD` New PC notebook
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/604>
-   `JUNGFRAU` Allow selecting ROIs to save 1D projections
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/699>

## 3.6.1

26-07-2022

-   `GH2` Correction and Dark notebooks.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/658>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/684>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/707>
-   pin [xarray] to 2022.3.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/706>

## 3.6.0

20-07-2022

-   `LPD` - `DARK` Save bad pixel dark data as uint32, not float64.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/698>
-   `LPD` - `CORRECT` Add option to disable Oauth when using CalCat
    proxy.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/681>
-   `AGIPD` - `LPD` - `DSSC` - `DARK` Avoid printing errors for expected
    missing constant files.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/664>
-   `TESTS` Refactor and split [AGIPDCtrl] methods (old
    and new RAW data versions.) and add unit tests
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/651>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/646>
-   `webservice` Monitor SLURM jobs in a separate process.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/668>
-   `webservice` Restructure the SLURM database to give more
    meaningful success/failure information.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/683>
-   `XFEL-CALIBRATE` Recreate environments on demand for
    xfel-calibrate-repeat.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/660>
-   Upgrade the Jupyter packages to the latest version.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/697/diffs>
-   Update Extra-data to 1.12:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/693>
    -   [EXtra-data] Disable [_use_voview] to be able to
        use [.files] or avoid using [.files]
        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/682>,
        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/688>

## 3.5.5

15-06-2022

-   `AGIPD` - `CORRECT` Expose max tasks per pool worker.

## 3.5.4

13-06-2022

-   `AGIPD` Convert bias_voltage parameter condition to integer in
    cal_tools.
-   `LPD` Fix correcting a single pulse.
-   `LPD` VCXI require 4 modules.

## 3.5.3

19-05-2022

-   `LPD` - `CORRECT` Optionally create virtual CXI files
-   `LPD` - `CORRECT` Expose max-nodes parameter
-   `AGIPD` Replace gain_choose_int by fused types
-   Fix missing install of restful_config.yaml
-   Fix use of xfel-calibrate --skip-report

## 3.5.2

16.05.2022

-   `LPD` - `CORRECT` New correction notebook for `LPD`
-   New [files] module to write European XFEL HDF5 corrected
    data files.

## 3.5.1

05-04-2022

-   Calibration Constant version's new [Variant] file
    attribute. To indicate method of handling the constant post
    retrieval.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/619>
-   `Epix100` dark Badpixels Map.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/637>
-   `skip-plots` flag to finish correction before plotting.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/635>
-   First trainId's timestamp as RAW data creation_time, if there is
    myMDC connection.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/630>
-   `AGIPD` correction can correct one cellId without plotting errors.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/642>
-   Fixed mode relative gain constants in `Jungfrau` can be retrieved.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/639>
-   Only instrument source is selected to check number of trains to dark
    process.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/636>
-   `AGIPD` trains for dark processing is selected for each module
    individually.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
-   Produce report after trying to correct `AGIPD` run with no images.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/641>
-   `AGIPD`'s bias voltage for AGIPD1M is read from slow data.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/647>
-   Removed psutil dependency.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
-   Update Pasha to 0.1.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/638>

## 3.5.0

01-03-2022

-   Updating Correction and dark notebooks for `JUNGFRAU`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/518>
-   Updating Correction and dark notebooks for `AGIPD`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating Correction and dark notebooks for `PnCCD`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/559>
-   Updating Correction and dark notebooks for `ePix100`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/500>
    -   `EXtra-data` is integrated to read files in pycalibration for
        `AGIPD`, `JUNGFRAU`, `ePix100`, and `PnCCD`. Dark and Correction
        notebooks.
    -   Pasha is now used for processing data for `JUNGFRAU`, `ePix100` and
        `PnCCD`.
    -   pyDetLib correction functions were removed (except for
        common-mode correction).
    -   [db-module] is useless now for `JUNGFRAU`, `ePix100`, and
        `PnCCD`. Some parameters were updated in dark and correction
        notebooks for the mentioned detectors.
-   `gain_mode` and burst mode are now available for
    `JUNGFRAU`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
-   `JUNGFRAU` has now a new badpixel value,
    `WRONG_GAIN_VALUE`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
-   Pass through available for testing in-progress ORCA service.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests?scope=all&state=merged&search=orca>
-   Non-calibrated RAW h5files are no longer copied.
-   High priority partitions ([upex-high and
    upex-middle]) are used for runs from ACTIVE and READY
    proposals, only.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/628>
-   Supporting to disable LPD Correction through the webservice.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/629>
-   Compatibility for old DAQ files for REMI is added.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/607>
-   server-overview refactors.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/593>
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/589>
-   AGIPD correction notebook support AgipdLitFrameFinder device.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/603>
-   Parsing code arguments in xfel-calibrate is refactored.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/575>
-   skip-plots option for AGIPD.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/581>
-   Native implementation for transposition of constants AGIPD.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/580>
-   Trains for AGIPD can be selected for correction.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/574>
-   Skip report flag in xfel-calibrate.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/579>
-   Fix ReadTheDocs.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/448>
-   Fix error reporting for re-injecting the same CCV.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/627>
-   Fix AGIPD for legacy runs without `gain_mode`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/617>
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/626>
-   Pinning markupsafe version 2.0.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/631>
-   Pinning psutil 5.9.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating Extra-data to 1.9.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating h5py to 3.5.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/602>

## 3.4.3

20-10-2021

-   Update pyDetLib tag.
-   Add explicit dependencies on matplotlib, scipy.
-   Remove outdated matplotlib rcParams setting.
-   Update EXtra-geom to 1.6.
-   Remove cluster_profile parameter from notebooks which don't use
    it.
-   Fix checking availability for the concurrency parameter.
-   Fix launching work directly (not via Slurm).
-   Fix `sphinx-rep` temp folder recreation, if sphinx-rep
    already existed.
-   Fix missing string conversion for slurm-scheduling argument.
-   Fix title reports for multiple detectors per run folder.
-   Append to .out files for preemptable finalize job.
-   `AGIPD` `CORRECT` Reuse previously found constants.
-   `AGIPD` Fix missing memory cell index in SlopesPC constant
    sanitization.
-   `AGIPD` Only use bad pixels from darks in
    agipdutils.baseline_correct_via_stripes.
-   `AGIPD` `DARK` Use function to get list of karabo_da from run
    for making Slurm jobs.
-   `EPIX100` - `CORRECT` Set absolute_gain to false if relative gain
    was not retrieved.
-   `JUNGFRAU` Fix running for multiple modules and flip logic for
    do_relative_gain.
-   `JUNGFRAU` Style changes for Dark and Correct notebooks.
-   `REMI` Add notebook to reconstruct detector hits from raw data.
-   `webservice` Check run migration status using MyMDC.
-   Resolve \"Skip ZMQ tests if zmq connection for calibration DB not
    available\".
-   Reproducibility, step 1.

## 3.4.2

17-09-2021

-   Remove driver=core from all notebook
-   `webservice` Make use of Dynaconf for managing secrets.
-   `webservice` Make use of dedicated slurm partitions.
-   `webservice` Handle missing migration information (missing
    user.status fattr).
-   `webservice` Implement, raise, and catch, migration errors to send
    mdc messages.
-   `webservice` Simplify handling of user notebook paths.
-   `webservice` Update princess to 0.4 (use Unix sockets).
-   `webservice` Update MyMDC with begin and end times.
-   `webservice` create output folder before copying slow data.
-   `AGIPD` `CORRECT` read acq_rate from slow data.
-   `AGIPD` - `CORRECT` Set default memory cells to 352.
-   `AGIPD` `CORRECT` Set maximum pulses to correct based on file
    content.
-   `AGIPD` - `FF` Correctly label legends in figures.
-   `AGIPD` - `FF` Add HIBEF AGIPD500K and fix some issue with
    retrieval of conditions.
-   `JUNGFRAU` Add Gain setting to Jungfrau notebooks.
-   `JUNGFRAU` Fix max gain plot in LPD correct notebook
-   `JUNGFRAU` `DARK` Clearer error message for Jungfrau Dark
    notebooks no suitable files are found
-   `LPD` `CORRECT` Fix max gain plot.
-   `EPIX100` `CORRECT` Solve conflict between gain correction and
    clustering

## 3.4.1


16-07-2021

-   Update h5py to 3.3
-   Stop execution on notebook errors
-   `AGIPD` Add integration time as operating condition to all
    notebooks
-   `webservice` Add blocklist pattern when copying untouched files in
    webservice.
-   `webservice` Expose dark configurations in update_config.py
-   Fix MetadataClient.get_proposal_runs arguments call.
-   Fix Use snapshot for injecting constants for old PDU mappings
-   Fix the old time-summary (creation time for retrieved constants)
-   Update documentation notes on venv installation
-   Ignore all .so files in gitignore

## 3.4.0

28-06-2021

-   Update to Python 3.8.
-   Bump numpy to 1.20.3 and remove fabio.
-   remove PyQT dependency.
-   Disable dark requests from serve overview.
-   Update report upload parameter key.
-   Override locale to always use UTF-8.
-   Assorted cleanup of xfel-calibrate.
-   Fix pre-commit.
-   Use argparse only if name is main, call main with args dict.
-   `webservice` Use full hostname for webservice overview.
-   `webservice` Show clearer messages when running webservice in sim
    mode.
-   `webservice` Fix filename lineno and typos in webservice logs.
-   `webservice` Fix creating an extra run folder in run output
    folder.
-   `AGIPD` Parallelize gain/mask compression for writing corrected
    AGIPD files.
-   `AGIPD` - `DARK` Fix processing empty sequence files.
-   `AGIPD` - `PC` - `FF` Update notebooks with new CALCAT mapping.
-   `AGIPD` - `JUNGFRAU` Use all available sequences for processing
    darks for AGIPD and Jungfrau.
-   `AGIPD` - `LPD` - `DSSC` Fix retrieve old constants for comparison
    for modular detectors.
-   `LPD` Fix data paths in LPD notebook.
-   `REMI` Fix user notebook path for REMI correct notebook
    provisionally.
-   `EPIX100` - `CORRECT` Add Common mode correction.
-   Fix plotting-related warnings.
-   Test update config.
-   Test get_from_db and send_to_db.