# Release Notes

## 3.7.5

16-11-2022

-   `JUNGFRAU` Fix index location for ROI instrument output
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/765>

## 3.7.4


15-11-2022

-   `JUNGFRAU` Only emit a warning if all sequences of a job are empty
-   `JUNGFRAU` Fix storing multiple ROIs for a single module
-   `JUNGFRAU` Replicate a raw dataset in processed data for legacy analysis code

## 3.7.3

10-11-2022

-   `DataFile` Add support for older INDEX and METADATA versions.
-   `LPD` Fix histogram plotting if selected cell's data is identical
    for trains.
-   `JUNGFRAU` Avoid unclear error messages when the run has no
    trains.
-   `AGIPD` Change notebook-default setting of max-task-per-worker
    to 1.
-   `REMI` Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses.
-   `webservice` Fix timestamps in the overview page.
-   `webservice` Check run type more directly in myMdC response.

## 3.7.2

28-10-2022

-   `JUNGFRAU` Add support for strixel sensors
-   `JUNGFRAU` Shorter summary notebook for darks by default
-   `JUNGFRAU` Always align corrected data properly
-   `JUNGFRAU` Harden against 0 adc data in dark characterization
-   `REMI` Add support for quad nodes and various pulse separation
    improvements
-   `AGIPD` Allow manual photon energy input for photonization and add
    consistency plots
-   Grant priority partitions for set-up time

## 3.7.1

14-10-2022

-   `EPIX100` Add Common mode corrections to dark characterization
-   `EPIX100` Use a single value for temperature
-   `JUNGFRAU` Fix bad pixel masking
-   `JUNGFRAU` Limit number of plotted trains to 500 during
    corrections
-   `AGIPD` Don't fail when there are no trains in sequence file
-   `REMI` Use DataFile API and support asymmetric pump-probe patterns
-   Skip corrections for runs which match specific types

## 3.7.0

07-09-2022

-   `JUNGFRAU` Mask double sized pixels
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/689>
-   `LPD` Storing retrieved calibration constants in
    calibration_metadata.yml.
-   Storing the running logs in the report folder path instead of a
    [temp] folder wherever the processing ran from.
-   Selecting partitions based on beam time dates.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/725>
-   Creation time is not read from myMDC anymore.

## 3.6.4

20-08-2022

-   `AGIPD` Significant improvements to AGIPG LitFrameFinder
    implementation, including offline support.
-   `AGIPD` Fix misalignment of data and gain/mask when common mode is
    used with manual pulse slicing.

## 3.6.3

23-08-2022

-   `AGIPD` litframe finder configs in update_config
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/723>

## 3.6.2

15-08-2022

-   `EPIX100` - `JUNGFRAU` - `pnCCD` Retrieve constants precorrection
    notebooks.
-   `AGIPD` skip sanitization
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/654>
-   `AGIPD` New PC notebook
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/604>
-   `JUNGFRAU` Allow selecting ROIs to save 1D projections
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/699>

## 3.6.1

26-07-2022

-   `GOTTHARD2` Correction and Dark notebooks.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/658>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/684>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/707>
-   pin [xarray] to 2022.3.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/706>

## 3.6.0

20-07-2022

-   `LPD` - `DARK` Save bad pixel dark data as uint32, not float64.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/698>
-   `LPD` - `CORRECT` Add option to disable Oauth when using CalCat
    proxy.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/681>
-   `AGIPD` - `LPD` - `DSSC` - `DARK` Avoid printing errors for expected
    missing constant files.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/664>
-   `TESTS` Refactor and split [AGIPDCtrl] methods (old
    and new RAW data versions.) and add unit tests
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/651>,
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/646>
-   `webservice` Monitor SLURM jobs in a separate process.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/668>
-   `webservice` Restructure the SLURM database to give more
    meaningful success/failure information.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/683>
-   `XFEL-CALIBRATE` Recreate environments on demand for
    xfel-calibrate-repeat.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/660>
-   Upgrade the Jupyter packages to the latest version.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/697/diffs>
-   Update Extra-data to 1.12:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/693>
    -   [EXtra-data] Disable [_use_voview] to be able to
        use [.files] or avoid using [.files]
        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/682>,
        <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/688>

## 3.5.5

15-06-2022

-   `AGIPD` - `CORRECT` Expose max tasks per pool worker.

## 3.5.4

13-06-2022

-   `AGIPD` Convert bias_voltage parameter condition to integer in
    cal_tools.
-   `LPD` Fix correcting a single pulse.
-   `LPD` VCXI require 4 modules.

## 3.5.3

19-05-2022

-   `LPD` - `CORRECT` Optionally create virtual CXI files
-   `LPD` - `CORRECT` Expose max-nodes parameter
-   `AGIPD` Replace gain_choose_int by fused types
-   Fix missing install of restful_config.yaml
-   Fix use of xfel-calibrate --skip-report

## 3.5.2

16.05.2022

-   `LPD` - `CORRECT` New correction notebook for `LPD`
-   New [files] module to write European XFEL HDF5 corrected
    data files.

## 3.5.1

05-04-2022

-   Calibration Constant version's new [Variant] file
    attribute. To indicate method of handling the constant post
    retrieval.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/619>
-   `Epix100` dark Badpixels Map.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/637>
-   `skip-plots` flag to finish correction before plotting.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/635>
-   First trainId's timestamp as RAW data creation_time, if there is
    myMDC connection.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/630>
-   `AGIPD` correction can correct one cellId without plotting errors.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/642>
-   Fixed mode relative gain constants in `Jungfrau` can be retrieved.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/639>
-   Only instrument source is selected to check number of trains to dark
    process.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/636>
-   `AGIPD` trains for dark processing is selected for each module
    individually.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
-   Produce report after trying to correct `AGIPD` run with no images.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/641>
-   `AGIPD`'s bias voltage for AGIPD1M is read from slow data.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/647>
-   Removed psutil dependency.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653>
-   Update Pasha to 0.1.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/638>

## 3.5.0

01-03-2022

-   Updating Correction and dark notebooks for `JUNGFRAU`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/518>
-   Updating Correction and dark notebooks for `AGIPD`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating Correction and dark notebooks for `PnCCD`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/559>
-   Updating Correction and dark notebooks for `ePix100`:
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/500>
    -   `EXtra-data` is integrated to read files in pycalibration for
        `AGIPD`, `JUNGFRAU`, `ePix100`, and `PnCCD`. Dark and Correction
        notebooks.
    -   Pasha is now used for processing data for `JUNGFRAU`, `ePix100` and
        `PnCCD`.
    -   pyDetLib correction functions were removed (except for
        common-mode correction).
    -   [db-module] is useless now for `JUNGFRAU`, `ePix100`, and
        `PnCCD`. Some parameters were updated in dark and correction
        notebooks for the mentioned detectors.
-   `gain_mode` and burst mode are now available for
    `JUNGFRAU`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
-   `JUNGFRAU` has now a new badpixel value,
    `WRONG_GAIN_VALUE`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591>
-   Pass through available for testing in-progress ORCA service.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests?scope=all&state=merged&search=orca>
-   Non-calibrated RAW h5files are no longer copied.
-   High priority partitions ([upex-high and
    upex-middle]) are used for runs from ACTIVE and READY
    proposals, only.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/628>
-   Supporting to disable LPD Correction through the webservice.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/629>
-   Compatibility for old DAQ files for REMI is added.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/607>
-   server-overview refactors.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/593>
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/589>
-   AGIPD correction notebook support AgipdLitFrameFinder device.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/603>
-   Parsing code arguments in xfel-calibrate is refactored.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/575>
-   skip-plots option for AGIPD.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/581>
-   Native implementation for transposition of constants AGIPD.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/580>
-   Trains for AGIPD can be selected for correction.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/574>
-   Skip report flag in xfel-calibrate.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/579>
-   Fix ReadTheDocs.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/448>
-   Fix error reporting for re-injecting the same CCV.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/627>
-   Fix AGIPD for legacy runs without `gain_mode`.
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/617>
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/626>
-   Pinning markupsafe version 2.0.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/631>
-   Pinning psutil 5.9.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating Extra-data to 1.9.1
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535>
-   Updating h5py to 3.5.0
    <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/602>

## 3.4.3

20-10-2021

-   Update pyDetLib tag.
-   Add explicit dependencies on matplotlib, scipy.
-   Remove outdated matplotlib rcParams setting.
-   Update EXtra-geom to 1.6.
-   Remove cluster_profile parameter from notebooks which don't use
    it.
-   Fix checking availability for the concurrency parameter.
-   Fix launching work directly (not via Slurm).
-   Fix `sphinx-rep` temp folder recreation, if sphinx-rep
    already existed.
-   Fix missing string conversion for slurm-scheduling argument.
-   Fix title reports for multiple detectors per run folder.
-   Append to .out files for preemptable finalize job.
-   `AGIPD` `CORRECT` Reuse previously found constants.
-   `AGIPD` Fix missing memory cell index in SlopesPC constant
    sanitization.
-   `AGIPD` Only use bad pixels from darks in
    agipdutils.baseline_correct_via_stripes.
-   `AGIPD` `DARK` Use function to get list of karabo_da from run
    for making Slurm jobs.
-   `EPIX100` - `CORRECT` Set absolute_gain to false if relative gain
    was not retrieved.
-   `JUNGFRAU` Fix running for multiple modules and flip logic for
    do_relative_gain.
-   `JUNGFRAU` Style changes for Dark and Correct notebooks.
-   `REMI` Add notebook to reconstruct detector hits from raw data.
-   `webservice` Check run migration status using MyMDC.
-   Resolve \"Skip ZMQ tests if zmq connection for calibration DB not
    available\".
-   Reproducibility, step 1.

## 3.4.2

17-09-2021

-   Remove driver=core from all notebook
-   `webservice` Make use of Dynaconf for managing secrets.
-   `webservice` Make use of dedicated slurm partitions.
-   `webservice` Handle missing migration information (missing
    user.status fattr).
-   `webservice` Implement, raise, and catch, migration errors to send
    mdc messages.
-   `webservice` Simplify handling of user notebook paths.
-   `webservice` Update princess to 0.4 (use Unix sockets).
-   `webservice` Update MyMDC with begin and end times.
-   `webservice` create output folder before copying slow data.
-   `AGIPD` `CORRECT` read acq_rate from slow data.
-   `AGIPD` - `CORRECT` Set default memory cells to 352.
-   `AGIPD` `CORRECT` Set maximum pulses to correct based on file
    content.
-   `AGIPD` - `FF` Correctly label legends in figures.
-   `AGIPD` - `FF` Add HIBEF AGIPD500K and fix some issue with
    retrieval of conditions.
-   `JUNGFRAU` Add Gain setting to Jungfrau notebooks.
-   `JUNGFRAU` Fix max gain plot in LPD correct notebook
-   `JUNGFRAU` `DARK` Clearer error message for Jungfrau Dark
    notebooks no suitable files are found
-   `LPD` `CORRECT` Fix max gain plot.
-   `EPIX100` `CORRECT` Solve conflict between gain correction and
    clustering

## 3.4.1


16-07-2021

-   Update h5py to 3.3
-   Stop execution on notebook errors
-   `AGIPD` Add integration time as operating condition to all
    notebooks
-   `webservice` Add blocklist pattern when copying untouched files in
    webservice.
-   `webservice` Expose dark configurations in update_config.py
-   Fix MetadataClient.get_proposal_runs arguments call.
-   Fix Use snapshot for injecting constants for old PDU mappings
-   Fix the old time-summary (creation time for retrieved constants)
-   Update documentation notes on venv installation
-   Ignore all .so files in gitignore

## 3.4.0

28-06-2021

-   Update to Python 3.8.
-   Bump numpy to 1.20.3 and remove fabio.
-   remove PyQT dependency.
-   Disable dark requests from serve overview.
-   Update report upload parameter key.
-   Override locale to always use UTF-8.
-   Assorted cleanup of xfel-calibrate.
-   Fix pre-commit.
-   Use argparse only if name is main, call main with args dict.
-   `webservice` Use full hostname for webservice overview.
-   `webservice` Show clearer messages when running webservice in sim
    mode.
-   `webservice` Fix filename lineno and typos in webservice logs.
-   `webservice` Fix creating an extra run folder in run output
    folder.
-   `AGIPD` Parallelize gain/mask compression for writing corrected
    AGIPD files.
-   `AGIPD` - `DARK` Fix processing empty sequence files.
-   `AGIPD` - `PC` - `FF` Update notebooks with new CALCAT mapping.
-   `AGIPD` - `JUNGFRAU` Use all available sequences for processing
    darks for AGIPD and Jungfrau.
-   `AGIPD` - `LPD` - `DSSC` Fix retrieve old constants for comparison
    for modular detectors.
-   `LPD` Fix data paths in LPD notebook.
-   `REMI` Fix user notebook path for REMI correct notebook
    provisionally.
-   `EPIX100` - `CORRECT` Add Common mode correction.
-   Fix plotting-related warnings.
-   Test update config.
-   Test get_from_db and send_to_db.