add release dates and refactors

docs/requirements.txt

-docs/requirements.txtiCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
+iCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0

docs/source/changelog.rst

@@ -3,24 +3,44 @@ Release Notes
 
 3.5.5
 -----
-- [AGIPD][CORRECT] Set max tasks per worker to 1 to avoid Out Of Memory issue for AGIPD.
+
+15-06-2022
+
+- [AGIPD][CORRECT] Expose max tasks per pool worker.
 
 3.5.4
 -----
+
+13-06-2022
+
 - [AGIPD] Convert bias_voltage parameter condition to integer in cal_tools.
 - [LPD] Fix correcting a single pulse.
 - [LPD] VCXI require 4 modules.
 
 3.5.3
 -----
+
+19-05-2022
+
 - [LPD][CORRECT] Optionally create virtual CXI files
 - [LPD][CORRECT] Expose max-nodes parameter
 - [AGIPD] Replace gain_choose_int by fused types
 - Fix missing install of restful_config.yaml
 - Fix use of xfel-calibrate --skip-report
 
+3.5.2
+-----
+
+16.05.2022
+
+- [LPD][CORRECT] New correction notebook for LPD
+- New `files` module to write European XFEL HDF5 corrected data files.
+
 3.5.1
 -----
+
+05-04-2022
+
 - Calibration Constant version's new `Variant` file attribute. To indicate method of handling the constant post retrieval. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/619
 - Epix100 dark Badpixels Map. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/637
 - `skip-plots` flag to finish correction before plotting. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/635
@@ -36,6 +56,9 @@ Release Notes
 
 3.5.0
 -----
+
+01-03-2022
+
 - Updating Correction and dark notebooks for JUNGFRAU: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/518
 - Updating Correction and dark notebooks for AGIPD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
 - Updating Correction and dark notebooks for PnCCD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/559
@@ -70,6 +93,9 @@ Release Notes
 
 3.4.3
 -----
+
+20-10-2021
+
 - Update pyDetLib tag.
 - Add explicit dependencies on matplotlib, scipy.
 - Remove outdated matplotlib rcParams setting.
@@ -93,12 +119,11 @@ Release Notes
 - Resolve "Skip ZMQ tests if zmq connection for calibration DB not available".
 - Reproducibility, step 1.
 
-
-
-
 3.4.2
 -----
 
+17-09-2021
+
 - Remove driver=core from all notebook
 - [webservice] Make use of Dynaconf for managing secrets.
 - [webservice] Make use of dedicated slurm partitions.
@@ -123,6 +148,8 @@ Release Notes
 3.4.1
 -----
 
+16-07-2021
+
 - Update h5py to 3.3
 - Stop execution on notebook errors
 - [AGIPD] Add integration time as operating condition to all notebooks
@@ -138,6 +165,8 @@ Release Notes
 3.4.0
 -----
 
+28-06-2021
+
 - Update to Python 3.8.
 - Bump numpy to 1.20.3 and remove fabio.
 - remove PyQT dependency.
@@ -161,4 +190,4 @@ Release Notes
 - [EPIX][CORRECT] Add Common mode correction.
 - Fix plotting-related warnings.
 - Test update config.
-- Test get_from_db and send_to_db.
\ No newline at end of file
+- Test get_from_db and send_to_db.

docs/source/configuration.rst

@@ -75,40 +75,9 @@ This value should be derived e.g. by profiling memory usage per core, run times,
     the above example.
 
 The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function
-by setting the `use function` parameter. This function must be defined in a code cell in the notebook,
-its parameters should be named like other exposed parameters. It should return a list of of parameters
-to be inserted into the concurrently run notebooks. The example given e.g. defines the `balance_sequences`
-function::
-
-    def balance_sequences(in_folder, run, sequences, sequences_per_node):
-        import glob
-        import re
-        import numpy as np
-        if sequences_per_node != 0:
-            sequence_files = glob.glob("{}/r{:04d}/*-S*.h5".format(in_folder, run))
-            seq_nums = set()
-            for sf in sequence_files:
-                seqnum = re.findall(r".*-S([0-9]*).h5", sf)[0]
-                seq_nums.add(int(seqnum))
-            seq_nums -= set(sequences)
-            return [l.tolist() for l in np.array_split(list(seq_nums),
-                                                       len(seq_nums)//sequences_per_node+1)]
-        else:
-            return sequences
-
-TESTING: :func:`balance_sequences`
-
-.. note::
-
-    Note how imports are inlined in the definition. This is necessary, as only the function code,
-    not the entire notebook is executed.
-
-which requires as exposed parameters e.g. ::
-
-    in_folder = "/gpfs/exfel/exp/SPB/201701/p002038/raw/" # the folder to read data from, required
-    run = 239 # runs to process, required
-    sequences = [-1] # sequences to correct, set to -1 for all, range allowed
-    sequences_per_node = 2 # number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel
+by setting the `use function` parameter. This function must be defined in the first cell in the notebook
+its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters
+to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`.
 
 .. note::
 

docs/source/index.rst

@@ -26,7 +26,7 @@ where `DETECTOR` and `TYPE` specify the task to be performed.
 Additionally, it leverages the DESY/XFEL Maxwell cluster to run these jobs in parallel
 via SLURM_.
 
-Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are
+Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are looking
 for details on how to use as detector group staff.
 
 If you would like to integrate additional notebooks please see the :ref:`development_workflow`.

docs/source/installation.rst

@@ -54,33 +54,6 @@ Copy/paste script:
   python3 -m pip install .  # `-e` flag for editable install, e.g. `pip install -e .`
 
 
-Installation into user home directory
-=====================================
-
-This is not recommended as `pycalibration` has pinned dependencies for
-stability, if you install it directly into you users home environment then it
-will downgrade/upgrade your local packages, which may cause major issues and may
-**break your local environment**, it is highly recommended to use the venv
-installation method instead.
-
-1. ``source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate``
-2. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
-3. ``pyenv shell 3.8.11`` - load required version of python
-4. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
-5. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable
-
-Copy/paste script:
-
-.. code:: bash
-
-  source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
-  git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
-  pyenv shell 3.8.11
-  cd pycalibration
-  pip install --user .  # `-e` flag for editable install, e.g. `pip install -e .`
-  export PATH=$HOME/.local/bin:$PATH
-
-
 Creating an ipython kernel for virtual environments
 ===================================================
 

docs/source/tutorial.rst

@@ -53,7 +53,7 @@ Running the notebook
 
    You can see your job in the queue with::
 
-     squeue -u $USERNAME
+     squeue --me
 
 3. Look at the generated report in the chosen output folder.
 4. More information on the job run on the cluster can be found in the temp folder.

docs/source/workflow.rst

@@ -4,7 +4,7 @@ Development Workflow
 ====================
 
 The following walkthrough will guide you through a possible workflow
-when developing new notebook for offline calibration.
+when developing new notebooks for offline calibration.
 
 Fresh Start
 -----------
@@ -171,18 +171,18 @@ In principle there a not restrictions other than that parameters that are expose
 command line need to be defined in the first code cell of the notebook.
 
 However, a few guidelines should be observed to make notebook useful for display as
-reports and usage by other.
+reports and usage by others.
 
 External Libraries
 ~~~~~~~~~~~~~~~~~~
 
-You may use a wide variaty of libraries available in Python, but keep in mind that others
+You may use a wide variety of libraries available in Python, but keep in mind that others
 wanting to run the tool will need to install these requirements as well. Thus,
 
-* do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
+* Do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
   be created using matplotlib_ and numerical processing should be done in numpy_.
 
-* keep runtime and library requirements in mind. A library doing its own parallelism either
+* Keep runtime and library requirements in mind. A library doing its own parallelism either
   needs to programmatically be able to set this up, or automatically do so. If you need to
   start something from the command line first, things might be tricky as you will likely
   need to run this via `POpen` commands with appropriate environment variable.
@@ -298,4 +298,4 @@ documentation.
 .. _matplotlib: https://matplotlib.org/
 .. _numpy: http://www.numpy.org/
 .. _h5py: https://www.h5py.org/
-.. _iCalibrationDB: https://in.xfel.eu/readthedocs/docs/icalibrationdb/en/latest/
+.. _iCalibrationDB: https://git.xfel.eu/detectors/cal_db_interactive

docs/source/xfel_calibrate_conf.rst

@@ -4,6 +4,3 @@ xfel_calibrate
 .. module:: xfel_calibrate.calibrate
 
 .. autofunction:: balance_sequences
-
-
-:py:mod: xfel_calibrate.notebooks
\ No newline at end of file