From cb3e3f84f6c6b3bdce88a8ddde496666d642e668 Mon Sep 17 00:00:00 2001
From: ahmedk <karim.ahmed@xfel.eu>
Date: Tue, 21 Jun 2022 19:35:17 +0200
Subject: [PATCH] add release dates and refactors
---
docs/requirements.txt | 2 +-
docs/source/changelog.rst | 39 +++++++++++++++++++++++++----
docs/source/configuration.rst | 37 +++------------------------
docs/source/index.rst | 2 +-
docs/source/installation.rst | 27 --------------------
docs/source/tutorial.rst | 2 +-
docs/source/workflow.rst | 12 ++++-----
docs/source/xfel_calibrate_conf.rst | 3 ---
8 files changed, 46 insertions(+), 78 deletions(-)
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 84d75d6b0..b3c28f3e8 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1 +1 @@
-docs/requirements.txtiCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
+iCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst
index 24374db8f..f2f266a62 100644
--- a/docs/source/changelog.rst
+++ b/docs/source/changelog.rst
@@ -3,24 +3,44 @@ Release Notes
3.5.5
-----
-- [AGIPD][CORRECT] Set max tasks per worker to 1 to avoid Out Of Memory issue for AGIPD.
+
+15-06-2022
+
+- [AGIPD][CORRECT] Expose max tasks per pool worker.
3.5.4
-----
+
+13-06-2022
+
- [AGIPD] Convert bias_voltage parameter condition to integer in cal_tools.
- [LPD] Fix correcting a single pulse.
- [LPD] VCXI require 4 modules.
3.5.3
-----
+
+19-05-2022
+
- [LPD][CORRECT] Optionally create virtual CXI files
- [LPD][CORRECT] Expose max-nodes parameter
- [AGIPD] Replace gain_choose_int by fused types
- Fix missing install of restful_config.yaml
- Fix use of xfel-calibrate --skip-report
+3.5.2
+-----
+
+16.05.2022
+
+- [LPD][CORRECT] New correction notebook for LPD
+- New `files` module to write European XFEL HDF5 corrected data files.
+
3.5.1
-----
+
+05-04-2022
+
- Calibration Constant version's new `Variant` file attribute. To indicate method of handling the constant post retrieval. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/619
- Epix100 dark Badpixels Map. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/637
- `skip-plots` flag to finish correction before plotting. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/635
@@ -36,6 +56,9 @@ Release Notes
3.5.0
-----
+
+01-03-2022
+
- Updating Correction and dark notebooks for JUNGFRAU: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/518
- Updating Correction and dark notebooks for AGIPD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
- Updating Correction and dark notebooks for PnCCD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/559
@@ -70,6 +93,9 @@ Release Notes
3.4.3
-----
+
+20-10-2021
+
- Update pyDetLib tag.
- Add explicit dependencies on matplotlib, scipy.
- Remove outdated matplotlib rcParams setting.
@@ -93,12 +119,11 @@ Release Notes
- Resolve "Skip ZMQ tests if zmq connection for calibration DB not available".
- Reproducibility, step 1.
-
-
-
3.4.2
-----
+17-09-2021
+
- Remove driver=core from all notebook
- [webservice] Make use of Dynaconf for managing secrets.
- [webservice] Make use of dedicated slurm partitions.
@@ -123,6 +148,8 @@ Release Notes
3.4.1
-----
+16-07-2021
+
- Update h5py to 3.3
- Stop execution on notebook errors
- [AGIPD] Add integration time as operating condition to all notebooks
@@ -138,6 +165,8 @@ Release Notes
3.4.0
-----
+28-06-2021
+
- Update to Python 3.8.
- Bump numpy to 1.20.3 and remove fabio.
- remove PyQT dependency.
@@ -161,4 +190,4 @@ Release Notes
- [EPIX][CORRECT] Add Common mode correction.
- Fix plotting-related warnings.
- Test update config.
-- Test get_from_db and send_to_db.
\ No newline at end of file
+- Test get_from_db and send_to_db.
diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
index bd508db2a..750cdb547 100644
--- a/docs/source/configuration.rst
+++ b/docs/source/configuration.rst
@@ -75,40 +75,9 @@ This value should be derived e.g. by profiling memory usage per core, run times,
the above example.
The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function
-by setting the `use function` parameter. This function must be defined in a code cell in the notebook,
-its parameters should be named like other exposed parameters. It should return a list of of parameters
-to be inserted into the concurrently run notebooks. The example given e.g. defines the `balance_sequences`
-function::
-
- def balance_sequences(in_folder, run, sequences, sequences_per_node):
- import glob
- import re
- import numpy as np
- if sequences_per_node != 0:
- sequence_files = glob.glob("{}/r{:04d}/*-S*.h5".format(in_folder, run))
- seq_nums = set()
- for sf in sequence_files:
- seqnum = re.findall(r".*-S([0-9]*).h5", sf)[0]
- seq_nums.add(int(seqnum))
- seq_nums -= set(sequences)
- return [l.tolist() for l in np.array_split(list(seq_nums),
- len(seq_nums)//sequences_per_node+1)]
- else:
- return sequences
-
-TESTING: :func:`balance_sequences`
-
-.. note::
-
- Note how imports are inlined in the definition. This is necessary, as only the function code,
- not the entire notebook is executed.
-
-which requires as exposed parameters e.g. ::
-
- in_folder = "/gpfs/exfel/exp/SPB/201701/p002038/raw/" # the folder to read data from, required
- run = 239 # runs to process, required
- sequences = [-1] # sequences to correct, set to -1 for all, range allowed
- sequences_per_node = 2 # number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel
+by setting the `use function` parameter. This function must be defined in the first cell in the notebook
+its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters
+to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`.
.. note::
diff --git a/docs/source/index.rst b/docs/source/index.rst
index cc3e055ea..b57a68dee 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -26,7 +26,7 @@ where `DETECTOR` and `TYPE` specify the task to be performed.
Additionally, it leverages the DESY/XFEL Maxwell cluster to run these jobs in parallel
via SLURM_.
-Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are
+Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are looking
for details on how to use as detector group staff.
If you would like to integrate additional notebooks please see the :ref:`development_workflow`.
diff --git a/docs/source/installation.rst b/docs/source/installation.rst
index 594f54ada..97d83ad8c 100644
--- a/docs/source/installation.rst
+++ b/docs/source/installation.rst
@@ -54,33 +54,6 @@ Copy/paste script:
python3 -m pip install . # `-e` flag for editable install, e.g. `pip install -e .`
-Installation into user home directory
-=====================================
-
-This is not recommended as `pycalibration` has pinned dependencies for
-stability, if you install it directly into you users home environment then it
-will downgrade/upgrade your local packages, which may cause major issues and may
-**break your local environment**, it is highly recommended to use the venv
-installation method instead.
-
-1. ``source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate``
-2. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
-3. ``pyenv shell 3.8.11`` - load required version of python
-4. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
-5. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable
-
-Copy/paste script:
-
-.. code:: bash
-
- source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
- git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
- pyenv shell 3.8.11
- cd pycalibration
- pip install --user . # `-e` flag for editable install, e.g. `pip install -e .`
- export PATH=$HOME/.local/bin:$PATH
-
-
Creating an ipython kernel for virtual environments
===================================================
diff --git a/docs/source/tutorial.rst b/docs/source/tutorial.rst
index 73018ff11..450b00464 100644
--- a/docs/source/tutorial.rst
+++ b/docs/source/tutorial.rst
@@ -53,7 +53,7 @@ Running the notebook
You can see your job in the queue with::
- squeue -u $USERNAME
+ squeue --me
3. Look at the generated report in the chosen output folder.
4. More information on the job run on the cluster can be found in the temp folder.
diff --git a/docs/source/workflow.rst b/docs/source/workflow.rst
index a9369b4fb..bb0264f46 100644
--- a/docs/source/workflow.rst
+++ b/docs/source/workflow.rst
@@ -4,7 +4,7 @@ Development Workflow
====================
The following walkthrough will guide you through a possible workflow
-when developing new notebook for offline calibration.
+when developing new notebooks for offline calibration.
Fresh Start
-----------
@@ -171,18 +171,18 @@ In principle there a not restrictions other than that parameters that are expose
command line need to be defined in the first code cell of the notebook.
However, a few guidelines should be observed to make notebook useful for display as
-reports and usage by other.
+reports and usage by others.
External Libraries
~~~~~~~~~~~~~~~~~~
-You may use a wide variaty of libraries available in Python, but keep in mind that others
+You may use a wide variety of libraries available in Python, but keep in mind that others
wanting to run the tool will need to install these requirements as well. Thus,
-* do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
+* Do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
be created using matplotlib_ and numerical processing should be done in numpy_.
-* keep runtime and library requirements in mind. A library doing its own parallelism either
+* Keep runtime and library requirements in mind. A library doing its own parallelism either
needs to programmatically be able to set this up, or automatically do so. If you need to
start something from the command line first, things might be tricky as you will likely
need to run this via `POpen` commands with appropriate environment variable.
@@ -298,4 +298,4 @@ documentation.
.. _matplotlib: https://matplotlib.org/
.. _numpy: http://www.numpy.org/
.. _h5py: https://www.h5py.org/
-.. _iCalibrationDB: https://in.xfel.eu/readthedocs/docs/icalibrationdb/en/latest/
+.. _iCalibrationDB: https://git.xfel.eu/detectors/cal_db_interactive
diff --git a/docs/source/xfel_calibrate_conf.rst b/docs/source/xfel_calibrate_conf.rst
index 52d1bdf5c..4fed8d265 100644
--- a/docs/source/xfel_calibrate_conf.rst
+++ b/docs/source/xfel_calibrate_conf.rst
@@ -4,6 +4,3 @@ xfel_calibrate
.. module:: xfel_calibrate.calibrate
.. autofunction:: balance_sequences
-
-
-:py:mod: xfel_calibrate.notebooks
\ No newline at end of file
--
GitLab