changelog.md

Release Notes

3.7.5

16-11-2022

• JUNGFRAU Fix index location for ROI instrument output !765

3.7.4

15-11-2022

• JUNGFRAU Only emit a warning if all sequences of a job are empty
• JUNGFRAU Fix storing multiple ROIs for a single module
• JUNGFRAU Replicate a raw dataset in processed data for legacy analysis code

3.7.3

10-11-2022

• DataFile Add support for older INDEX and METADATA versions.
• LPD Fix histogram plotting if selected cell's data is identical for trains.
• JUNGFRAU Avoid unclear error messages when the run has no trains.
• AGIPD Change notebook-default setting of max-task-per-worker to 1.
• REMI Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses.
• webservice Fix timestamps in the overview page.
• webservice Check run type more directly in myMdC response.

3.7.2

28-10-2022

• JUNGFRAU Add support for strixel sensors
• JUNGFRAU Shorter summary notebook for darks by default
• JUNGFRAU Always align corrected data properly
• JUNGFRAU Harden against 0 adc data in dark characterization
• REMI Add support for quad nodes and various pulse separation improvements
• AGIPD Allow manual photon energy input for photonization and add consistency plots
• Grant priority partitions for set-up time

3.7.1

14-10-2022

• EPIX100 Add Common mode corrections to dark characterization
• EPIX100 Use a single value for temperature
• JUNGFRAU Fix bad pixel masking
• JUNGFRAU Limit number of plotted trains to 500 during corrections
• AGIPD Don't fail when there are no trains in sequence file
• REMI Use DataFile API and support asymmetric pump-probe patterns
• Skip corrections for runs which match specific types

3.7.0

07-09-2022

• JUNGFRAU Mask double sized pixels !689
• LPD Storing retrieved calibration constants in calibration_metadata.yml.
• Storing the running logs in the report folder path instead of a [temp] folder wherever the processing ran from.
• Selecting partitions based on beam time dates. !725
• Creation time is not read from myMDC anymore.

3.6.4

20-08-2022

• AGIPD Significant improvements to AGIPG LitFrameFinder implementation, including offline support.
• AGIPD Fix misalignment of data and gain/mask when common mode is used with manual pulse slicing.

3.6.3

23-08-2022

• AGIPD litframe finder configs in update_config !723

3.6.2

15-08-2022

• EPIX100 - JUNGFRAU - pnCCD Retrieve constants precorrection notebooks.
• AGIPD skip sanitization !654
• AGIPD New PC notebook !604
• JUNGFRAU Allow selecting ROIs to save 1D projections !699

3.6.1

26-07-2022

• GOTTHARD2 Correction and Dark notebooks. !658, !684, !707
• pin [xarray] to 2022.3.0 !706

3.6.0

20-07-2022

• LPD - DARK Save bad pixel dark data as uint32, not float64. !698
• LPD - CORRECT Add option to disable Oauth when using CalCat proxy. !681
• AGIPD - LPD - DSSC - DARK Avoid printing errors for expected missing constant files. !664
• TESTS Refactor and split [AGIPDCtrl] methods (old and new RAW data versions.) and add unit tests !651, !646
• webservice Monitor SLURM jobs in a separate process. !668
• webservice Restructure the SLURM database to give more meaningful success/failure information. !683
• XFEL-CALIBRATE Recreate environments on demand for xfel-calibrate-repeat. !660
• Upgrade the Jupyter packages to the latest version. !697 (diffs)
• Update Extra-data to 1.12: !693
  • [EXtra-data] Disable [_use_voview] to be able to use [.files] or avoid using [.files] !682, !688

3.5.5

15-06-2022

• AGIPD - CORRECT Expose max tasks per pool worker.

3.5.4

13-06-2022

• AGIPD Convert bias_voltage parameter condition to integer in cal_tools.
• LPD Fix correcting a single pulse.
• LPD VCXI require 4 modules.

3.5.3

19-05-2022

• LPD - CORRECT Optionally create virtual CXI files
• LPD - CORRECT Expose max-nodes parameter
• AGIPD Replace gain_choose_int by fused types
• Fix missing install of restful_config.yaml
• Fix use of xfel-calibrate --skip-report

3.5.2

16.05.2022

• LPD - CORRECT New correction notebook for LPD
• New [files] module to write European XFEL HDF5 corrected data files.

3.5.1

05-04-2022

• Calibration Constant version's new [Variant] file attribute. To indicate method of handling the constant post retrieval. !619
• Epix100 dark Badpixels Map. !637
• skip-plots flag to finish correction before plotting. !635
• First trainId's timestamp as RAW data creation_time, if there is myMDC connection. !630
• AGIPD correction can correct one cellId without plotting errors. !642
• Fixed mode relative gain constants in Jungfrau can be retrieved. !639
• Only instrument source is selected to check number of trains to dark process. !636
• AGIPD trains for dark processing is selected for each module individually. !653
• Produce report after trying to correct AGIPD run with no images. !641
• AGIPD's bias voltage for AGIPD1M is read from slow data. !647
• Removed psutil dependency. !653
• Update Pasha to 0.1.1 !638

3.5.0

01-03-2022

• Updating Correction and dark notebooks for JUNGFRAU: !518
• Updating Correction and dark notebooks for AGIPD: !535
• Updating Correction and dark notebooks for PnCCD: !559
• Updating Correction and dark notebooks for ePix100: !500
  • EXtra-data is integrated to read files in pycalibration for AGIPD, JUNGFRAU, ePix100, and PnCCD. Dark and Correction notebooks.
  • Pasha is now used for processing data for JUNGFRAU, ePix100 and PnCCD.
  • pyDetLib correction functions were removed (except for common-mode correction).
  • [db-module] is useless now for JUNGFRAU, ePix100, and PnCCD. Some parameters were updated in dark and correction notebooks for the mentioned detectors.
• gain_mode and burst mode are now available for JUNGFRAU. !591
• JUNGFRAU has now a new badpixel value, WRONG_GAIN_VALUE. !591
• Pass through available for testing in-progress ORCA service. https://git.xfel.eu/calibration/pycalibration/-/merge_requests?scope=all&state=merged&search=orca
• Non-calibrated RAW h5files are no longer copied.
• High priority partitions ([upex-high and upex-middle]) are used for runs from ACTIVE and READY proposals, only. !628
• Supporting to disable LPD Correction through the webservice. !629
• Compatibility for old DAQ files for REMI is added. !607
• server-overview refactors. !593 !589
• AGIPD correction notebook support AgipdLitFrameFinder device. !603
• Parsing code arguments in xfel-calibrate is refactored. !575
• skip-plots option for AGIPD. !581
• Native implementation for transposition of constants AGIPD. !580
• Trains for AGIPD can be selected for correction. !574
• Skip report flag in xfel-calibrate. !579
• Fix ReadTheDocs. !448
• Fix error reporting for re-injecting the same CCV. !627
• Fix AGIPD for legacy runs without gain_mode. !617 !626
• Pinning markupsafe version 2.0.1 !631
• Pinning psutil 5.9.0 !535
• Updating Extra-data to 1.9.1 !535
• Updating h5py to 3.5.0 !602

3.4.3

20-10-2021

• Update pyDetLib tag.
• Add explicit dependencies on matplotlib, scipy.
• Remove outdated matplotlib rcParams setting.
• Update EXtra-geom to 1.6.
• Remove cluster_profile parameter from notebooks which don't use it.
• Fix checking availability for the concurrency parameter.
• Fix launching work directly (not via Slurm).
• Fix sphinx-rep temp folder recreation, if sphinx-rep already existed.
• Fix missing string conversion for slurm-scheduling argument.
• Fix title reports for multiple detectors per run folder.
• Append to .out files for preemptable finalize job.
• AGIPD CORRECT Reuse previously found constants.
• AGIPD Fix missing memory cell index in SlopesPC constant sanitization.
• AGIPD Only use bad pixels from darks in agipdutils.baseline_correct_via_stripes.
• AGIPD DARK Use function to get list of karabo_da from run for making Slurm jobs.
• EPIX100 - CORRECT Set absolute_gain to false if relative gain was not retrieved.
• JUNGFRAU Fix running for multiple modules and flip logic for do_relative_gain.
• JUNGFRAU Style changes for Dark and Correct notebooks.
• REMI Add notebook to reconstruct detector hits from raw data.
• webservice Check run migration status using MyMDC.
• Resolve "Skip ZMQ tests if zmq connection for calibration DB not available".
• Reproducibility, step 1.

3.4.2

17-09-2021

• Remove driver=core from all notebook
• webservice Make use of Dynaconf for managing secrets.
• webservice Make use of dedicated slurm partitions.
• webservice Handle missing migration information (missing user.status fattr).
• webservice Implement, raise, and catch, migration errors to send mdc messages.
• webservice Simplify handling of user notebook paths.
• webservice Update princess to 0.4 (use Unix sockets).
• webservice Update MyMDC with begin and end times.
• webservice create output folder before copying slow data.
• AGIPD CORRECT read acq_rate from slow data.
• AGIPD - CORRECT Set default memory cells to 352.
• AGIPD CORRECT Set maximum pulses to correct based on file content.
• AGIPD - FF Correctly label legends in figures.
• AGIPD - FF Add HIBEF AGIPD500K and fix some issue with retrieval of conditions.
• JUNGFRAU Add Gain setting to Jungfrau notebooks.
• JUNGFRAU Fix max gain plot in LPD correct notebook
• JUNGFRAU DARK Clearer error message for Jungfrau Dark notebooks no suitable files are found
• LPD CORRECT Fix max gain plot.
• EPIX100 CORRECT Solve conflict between gain correction and clustering

3.4.1

16-07-2021

• Update h5py to 3.3
• Stop execution on notebook errors
• AGIPD Add integration time as operating condition to all notebooks
• webservice Add blocklist pattern when copying untouched files in webservice.
• webservice Expose dark configurations in update_config.py
• Fix MetadataClient.get_proposal_runs arguments call.
• Fix Use snapshot for injecting constants for old PDU mappings
• Fix the old time-summary (creation time for retrieved constants)
• Update documentation notes on venv installation
• Ignore all .so files in gitignore

3.4.0

28-06-2021

• Update to Python 3.8.
• Bump numpy to 1.20.3 and remove fabio.
• remove PyQT dependency.
• Disable dark requests from serve overview.
• Update report upload parameter key.
• Override locale to always use UTF-8.
• Assorted cleanup of xfel-calibrate.
• Fix pre-commit.
• Use argparse only if name is main, call main with args dict.
• webservice Use full hostname for webservice overview.
• webservice Show clearer messages when running webservice in sim mode.
• webservice Fix filename lineno and typos in webservice logs.
• webservice Fix creating an extra run folder in run output folder.
• AGIPD Parallelize gain/mask compression for writing corrected AGIPD files.
• AGIPD - DARK Fix processing empty sequence files.
• AGIPD - PC - FF Update notebooks with new CALCAT mapping.
• AGIPD - JUNGFRAU Use all available sequences for processing darks for AGIPD and Jungfrau.
• AGIPD - LPD - DSSC Fix retrieve old constants for comparison for modular detectors.
• LPD Fix data paths in LPD notebook.
• REMI Fix user notebook path for REMI correct notebook provisionally.
• EPIX100 - CORRECT Add Common mode correction.
• Fix plotting-related warnings.
• Test update config.
• Test get_from_db and send_to_db.