diff --git a/.gitignore b/.gitignore index 7f2b6feb4f30aa8d9281b22cf2aab62b154f937e..0d12043e3fad17066cb9e79315155659be840df9 100644 --- a/.gitignore +++ b/.gitignore @@ -8,8 +8,6 @@ *.npy *.out *.pkl -*.png -*.png *.secrets.yaml *.so *.tar diff --git a/docs/references/changelog.md b/docs/references/changelog.md new file mode 100644 index 0000000000000000000000000000000000000000..70363151f880e857f13ef7e2820fa60d4db61fd4 --- /dev/null +++ b/docs/references/changelog.md @@ -0,0 +1,401 @@ +# Release Notes + +## 3.7.5 + +16-11-2022 + +- `JUNGFRAU` Fix index location for ROI instrument output + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/765> + +## 3.7.4 + + +15-11-2022 + +- `JUNGFRAU` Only emit a warning if all sequences of a job are empty +- `JUNGFRAU` Fix storing multiple ROIs for a single module +- `JUNGFRAU` Replicate a raw dataset in processed data for legacy analysis code + +## 3.7.3 + +10-11-2022 + +- `DataFile` Add support for older INDEX and METADATA versions. +- `LPD` Fix histogram plotting if selected cell's data is identical + for trains. +- `JUNGFRAU` Avoid unclear error messages when the run has no + trains. +- `AGIPD` Change notebook-default setting of max-task-per-worker + to 1. +- `REMI` Allow pulse separation in REMI_DLD reconstruction to ignore FEL pulses. +- `webservice` Fix timestamps in the overview page. +- `webservice` Check run type more directly in myMdC response. + +## 3.7.2 + +28-10-2022 + +- `JUNGFRAU` Add support for strixel sensors +- `JUNGFRAU` Shorter summary notebook for darks by default +- `JUNGFRAU` Always align corrected data properly +- `JUNGFRAU` Harden against 0 adc data in dark characterization +- `REMI` Add support for quad nodes and various pulse separation + improvements +- `AGIPD` Allow manual photon energy input for photonization and add + consistency plots +- Grant priority partitions for set-up time + +## 3.7.1 + +14-10-2022 + +- `EPIX100` Add Common mode corrections to dark characterization +- `EPIX100` Use a single value for temperature +- `JUNGFRAU` Fix bad pixel masking +- `JUNGFRAU` Limit number of plotted trains to 500 during + corrections +- `AGIPD` Don't fail when there are no trains in sequence file +- `REMI` Use DataFile API and support asymmetric pump-probe patterns +- Skip corrections for runs which match specific types + +## 3.7.0 + +07-09-2022 + +- `JUNGFRAU` Mask double sized pixels + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/689> +- `LPD` Storing retrieved calibration constants in + calibration_metadata.yml. +- Storing the running logs in the report folder path instead of a + [temp] folder wherever the processing ran from. +- Selecting partitions based on beam time dates. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/725> +- Creation time is not read from myMDC anymore. + +## 3.6.4 + +20-08-2022 + +- `AGIPD` Significant improvements to AGIPG LitFrameFinder + implementation, including offline support. +- `AGIPD` Fix misalignment of data and gain/mask when common mode is + used with manual pulse slicing. + +## 3.6.3 + +23-08-2022 + +- `AGIPD` litframe finder configs in update_config + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/723> + +## 3.6.2 + +15-08-2022 + +- `EPIX100` - `JUNGFRAU` - `pnCCD` Retrieve constants precorrection + notebooks. +- `AGIPD` skip sanitization + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/654> +- `AGIPD` New PC notebook + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/604> +- `JUNGFRAU` Allow selecting ROIs to save 1D projections + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/699> + +## 3.6.1 + +26-07-2022 + +- `GOTTHARD2` Correction and Dark notebooks. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/658>, + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/684>, + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/707> +- pin [xarray] to 2022.3.0 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/706> + +## 3.6.0 + +20-07-2022 + +- `LPD` - `DARK` Save bad pixel dark data as uint32, not float64. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/698> +- `LPD` - `CORRECT` Add option to disable Oauth when using CalCat + proxy. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/681> +- `AGIPD` - `LPD` - `DSSC` - `DARK` Avoid printing errors for expected + missing constant files. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/664> +- `TESTS` Refactor and split [AGIPDCtrl] methods (old + and new RAW data versions.) and add unit tests + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/651>, + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/646> +- `webservice` Monitor SLURM jobs in a separate process. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/668> +- `webservice` Restructure the SLURM database to give more + meaningful success/failure information. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/683> +- `XFEL-CALIBRATE` Recreate environments on demand for + xfel-calibrate-repeat. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/660> +- Upgrade the Jupyter packages to the latest version. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/697/diffs> +- Update Extra-data to 1.12: + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/693> + - [EXtra-data] Disable [_use_voview] to be able to + use [.files] or avoid using [.files] + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/682>, + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/688> + +## 3.5.5 + +15-06-2022 + +- `AGIPD` - `CORRECT` Expose max tasks per pool worker. + +## 3.5.4 + +13-06-2022 + +- `AGIPD` Convert bias_voltage parameter condition to integer in + cal_tools. +- `LPD` Fix correcting a single pulse. +- `LPD` VCXI require 4 modules. + +## 3.5.3 + +19-05-2022 + +- `LPD` - `CORRECT` Optionally create virtual CXI files +- `LPD` - `CORRECT` Expose max-nodes parameter +- `AGIPD` Replace gain_choose_int by fused types +- Fix missing install of restful_config.yaml +- Fix use of xfel-calibrate --skip-report + +## 3.5.2 + +16.05.2022 + +- `LPD` - `CORRECT` New correction notebook for `LPD` +- New [files] module to write European XFEL HDF5 corrected + data files. + +## 3.5.1 + +05-04-2022 + +- Calibration Constant version's new [Variant] file + attribute. To indicate method of handling the constant post + retrieval. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/619> +- `Epix100` dark Badpixels Map. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/637> +- `skip-plots` flag to finish correction before plotting. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/635> +- First trainId's timestamp as RAW data creation_time, if there is + myMDC connection. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/630> +- `AGIPD` correction can correct one cellId without plotting errors. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/642> +- Fixed mode relative gain constants in `Jungfrau` can be retrieved. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/639> +- Only instrument source is selected to check number of trains to dark + process. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/636> +- `AGIPD` trains for dark processing is selected for each module + individually. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653> +- Produce report after trying to correct `AGIPD` run with no images. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/641> +- `AGIPD`'s bias voltage for AGIPD1M is read from slow data. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/647> +- Removed psutil dependency. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/653> +- Update Pasha to 0.1.1 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/638> + +## 3.5.0 + +01-03-2022 + +- Updating Correction and dark notebooks for `JUNGFRAU`: + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/518> +- Updating Correction and dark notebooks for `AGIPD`: + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535> +- Updating Correction and dark notebooks for `PnCCD`: + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/559> +- Updating Correction and dark notebooks for `ePix100`: + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/500> + - `EXtra-data` is integrated to read files in pycalibration for + `AGIPD`, `JUNGFRAU`, `ePix100`, and `PnCCD`. Dark and Correction + notebooks. + - Pasha is now used for processing data for `JUNGFRAU`, `ePix100` and + `PnCCD`. + - pyDetLib correction functions were removed (except for + common-mode correction). + - [db-module] is useless now for `JUNGFRAU`, `ePix100`, and + `PnCCD`. Some parameters were updated in dark and correction + notebooks for the mentioned detectors. +- `gain_mode` and burst mode are now available for + `JUNGFRAU`. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591> +- `JUNGFRAU` has now a new badpixel value, + `WRONG_GAIN_VALUE`. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/591> +- Pass through available for testing in-progress ORCA service. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests?scope=all&state=merged&search=orca> +- Non-calibrated RAW h5files are no longer copied. +- High priority partitions ([upex-high and + upex-middle]) are used for runs from ACTIVE and READY + proposals, only. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/628> +- Supporting to disable LPD Correction through the webservice. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/629> +- Compatibility for old DAQ files for REMI is added. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/607> +- server-overview refactors. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/593> + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/589> +- AGIPD correction notebook support AgipdLitFrameFinder device. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/603> +- Parsing code arguments in xfel-calibrate is refactored. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/575> +- skip-plots option for AGIPD. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/581> +- Native implementation for transposition of constants AGIPD. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/580> +- Trains for AGIPD can be selected for correction. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/574> +- Skip report flag in xfel-calibrate. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/579> +- Fix ReadTheDocs. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/448> +- Fix error reporting for re-injecting the same CCV. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/627> +- Fix AGIPD for legacy runs without `gain_mode`. + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/617> + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/626> +- Pinning markupsafe version 2.0.1 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/631> +- Pinning psutil 5.9.0 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535> +- Updating Extra-data to 1.9.1 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/535> +- Updating h5py to 3.5.0 + <https://git.xfel.eu/calibration/pycalibration/-/merge_requests/602> + +## 3.4.3 + +20-10-2021 + +- Update pyDetLib tag. +- Add explicit dependencies on matplotlib, scipy. +- Remove outdated matplotlib rcParams setting. +- Update EXtra-geom to 1.6. +- Remove cluster_profile parameter from notebooks which don't use + it. +- Fix checking availability for the concurrency parameter. +- Fix launching work directly (not via Slurm). +- Fix `sphinx-rep` temp folder recreation, if sphinx-rep + already existed. +- Fix missing string conversion for slurm-scheduling argument. +- Fix title reports for multiple detectors per run folder. +- Append to .out files for preemptable finalize job. +- `AGIPD` `CORRECT` Reuse previously found constants. +- `AGIPD` Fix missing memory cell index in SlopesPC constant + sanitization. +- `AGIPD` Only use bad pixels from darks in + agipdutils.baseline_correct_via_stripes. +- `AGIPD` `DARK` Use function to get list of karabo_da from run + for making Slurm jobs. +- `EPIX100` - `CORRECT` Set absolute_gain to false if relative gain + was not retrieved. +- `JUNGFRAU` Fix running for multiple modules and flip logic for + do_relative_gain. +- `JUNGFRAU` Style changes for Dark and Correct notebooks. +- `REMI` Add notebook to reconstruct detector hits from raw data. +- `webservice` Check run migration status using MyMDC. +- Resolve \"Skip ZMQ tests if zmq connection for calibration DB not + available\". +- Reproducibility, step 1. + +## 3.4.2 + +17-09-2021 + +- Remove driver=core from all notebook +- `webservice` Make use of Dynaconf for managing secrets. +- `webservice` Make use of dedicated slurm partitions. +- `webservice` Handle missing migration information (missing + user.status fattr). +- `webservice` Implement, raise, and catch, migration errors to send + mdc messages. +- `webservice` Simplify handling of user notebook paths. +- `webservice` Update princess to 0.4 (use Unix sockets). +- `webservice` Update MyMDC with begin and end times. +- `webservice` create output folder before copying slow data. +- `AGIPD` `CORRECT` read acq_rate from slow data. +- `AGIPD` - `CORRECT` Set default memory cells to 352. +- `AGIPD` `CORRECT` Set maximum pulses to correct based on file + content. +- `AGIPD` - `FF` Correctly label legends in figures. +- `AGIPD` - `FF` Add HIBEF AGIPD500K and fix some issue with + retrieval of conditions. +- `JUNGFRAU` Add Gain setting to Jungfrau notebooks. +- `JUNGFRAU` Fix max gain plot in LPD correct notebook +- `JUNGFRAU` `DARK` Clearer error message for Jungfrau Dark + notebooks no suitable files are found +- `LPD` `CORRECT` Fix max gain plot. +- `EPIX100` `CORRECT` Solve conflict between gain correction and + clustering + +## 3.4.1 + + +16-07-2021 + +- Update h5py to 3.3 +- Stop execution on notebook errors +- `AGIPD` Add integration time as operating condition to all + notebooks +- `webservice` Add blocklist pattern when copying untouched files in + webservice. +- `webservice` Expose dark configurations in update_config.py +- Fix MetadataClient.get_proposal_runs arguments call. +- Fix Use snapshot for injecting constants for old PDU mappings +- Fix the old time-summary (creation time for retrieved constants) +- Update documentation notes on venv installation +- Ignore all .so files in gitignore + +## 3.4.0 + +28-06-2021 + +- Update to Python 3.8. +- Bump numpy to 1.20.3 and remove fabio. +- remove PyQT dependency. +- Disable dark requests from serve overview. +- Update report upload parameter key. +- Override locale to always use UTF-8. +- Assorted cleanup of xfel-calibrate. +- Fix pre-commit. +- Use argparse only if name is main, call main with args dict. +- `webservice` Use full hostname for webservice overview. +- `webservice` Show clearer messages when running webservice in sim + mode. +- `webservice` Fix filename lineno and typos in webservice logs. +- `webservice` Fix creating an extra run folder in run output + folder. +- `AGIPD` Parallelize gain/mask compression for writing corrected + AGIPD files. +- `AGIPD` - `DARK` Fix processing empty sequence files. +- `AGIPD` - `PC` - `FF` Update notebooks with new CALCAT mapping. +- `AGIPD` - `JUNGFRAU` Use all available sequences for processing + darks for AGIPD and Jungfrau. +- `AGIPD` - `LPD` - `DSSC` Fix retrieve old constants for comparison + for modular detectors. +- `LPD` Fix data paths in LPD notebook. +- `REMI` Fix user notebook path for REMI correct notebook + provisionally. +- `EPIX100` - `CORRECT` Add Common mode correction. +- Fix plotting-related warnings. +- Test update config. +- Test get_from_db and send_to_db. diff --git a/docs/references/faq.md b/docs/references/faq.md new file mode 100644 index 0000000000000000000000000000000000000000..318b08dc2fed81834a28ff6b6f7934c316d1298c --- /dev/null +++ b/docs/references/faq.md @@ -0,0 +1 @@ +# Frequently Asked Questions diff --git a/docs/references/glossary.md b/docs/references/glossary.md new file mode 100644 index 0000000000000000000000000000000000000000..a0dcfa198a07508478a2f35c5835e8acfb8f4104 --- /dev/null +++ b/docs/references/glossary.md @@ -0,0 +1,6 @@ +# Glossary + +- PDU: Physical Detector Units - The detector module hardware name given by detector expert or manufacturer. + +- Detector UUID: + diff --git a/docs/static/CC_multiple_CCVs_calcat.png b/docs/static/CC_multiple_CCVs_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..03029432378e9d0f3e66dc6ee9e677dbfcd57cde Binary files /dev/null and b/docs/static/CC_multiple_CCVs_calcat.png differ diff --git a/docs/static/detector_mapping/AGIPD_modules_pic.png b/docs/static/detector_mapping/AGIPD_modules_pic.png new file mode 100644 index 0000000000000000000000000000000000000000..521d5186cc3d54f199d5ffcfa3995715044b6802 Binary files /dev/null and b/docs/static/detector_mapping/AGIPD_modules_pic.png differ diff --git a/docs/static/detector_mapping/JUNGFRAU_PDU_pic.png b/docs/static/detector_mapping/JUNGFRAU_PDU_pic.png new file mode 100644 index 0000000000000000000000000000000000000000..71c3dea6e92d404b54d19f4bc75f67faf0099c8d Binary files /dev/null and b/docs/static/detector_mapping/JUNGFRAU_PDU_pic.png differ diff --git a/docs/static/detector_mapping/icalibrationdb_agipd_det_mapping.png b/docs/static/detector_mapping/icalibrationdb_agipd_det_mapping.png new file mode 100644 index 0000000000000000000000000000000000000000..5d66d89970ea3743e10970529ab66e62b3fdaff9 Binary files /dev/null and b/docs/static/detector_mapping/icalibrationdb_agipd_det_mapping.png differ diff --git a/docs/static/home_calcat.png b/docs/static/home_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..b90501b10a51adeda7809468c33f1f8919292fd3 Binary files /dev/null and b/docs/static/home_calcat.png differ diff --git a/docs/static/home_cc_calcat.png b/docs/static/home_cc_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..da4481822d93a2944ba96db923925319f72f9c45 Binary files /dev/null and b/docs/static/home_cc_calcat.png differ diff --git a/docs/static/home_ccv_calcat.png b/docs/static/home_ccv_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..1d4f28f79d56e69e4b8b86155a158d356d76184b Binary files /dev/null and b/docs/static/home_ccv_calcat.png differ diff --git a/docs/static/home_conditions_calcat.png b/docs/static/home_conditions_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..fd8486cf3af96b07aa5360e9b50aa2b6177b035e Binary files /dev/null and b/docs/static/home_conditions_calcat.png differ diff --git a/docs/static/home_detectors_calcat.png b/docs/static/home_detectors_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..a92f792e30a78ee9391ad1b27216fa16e2ea4dc5 Binary files /dev/null and b/docs/static/home_detectors_calcat.png differ diff --git a/docs/static/home_pdu_calcat.png b/docs/static/home_pdu_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..dcd28e589025bdc2f358b1440aabb2760f28e7f9 Binary files /dev/null and b/docs/static/home_pdu_calcat.png differ diff --git a/docs/static/home_reports_calcat.png b/docs/static/home_reports_calcat.png new file mode 100644 index 0000000000000000000000000000000000000000..8018001ac81cca5f80627f2f5e5207d085c9fdde Binary files /dev/null and b/docs/static/home_reports_calcat.png differ diff --git a/docs/static/how_to_write_xfel_calibrate_notebook_NBC_14_0.png b/docs/static/how_to_write_xfel_calibrate_notebook_NBC_14_0.png new file mode 100644 index 0000000000000000000000000000000000000000..16bc34871e0f06b1eb2f828d7e30c717a49159be Binary files /dev/null and b/docs/static/how_to_write_xfel_calibrate_notebook_NBC_14_0.png differ diff --git a/docs/static/myMDC/calibration_constants.png b/docs/static/myMDC/calibration_constants.png new file mode 100644 index 0000000000000000000000000000000000000000..2fc6af22d56584806546cbb0d2220c7429873a27 Binary files /dev/null and b/docs/static/myMDC/calibration_constants.png differ