diff --git a/docs/requirements.txt b/docs/requirements.txt
index 84d75d6b093e9b94728cf99124b297b488479cf9..b3c28f3e8f7a208d68e3583b9fad97982a28037f 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -1 +1 @@
-docs/requirements.txtiCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
+iCalibrationDB @ git+https://xcalgitlab:${GITHUB_TOKEN}@git.xfel.eu/gitlab/detectors/cal_db_interactive.git@2.2.0
diff --git a/docs/source/changelog.rst b/docs/source/changelog.rst
index 24374db8f93eb67feecefa1446c4fb2768d234de..f2f266a62756c6a5161049ac4a07046d17087d29 100644
--- a/docs/source/changelog.rst
+++ b/docs/source/changelog.rst
@@ -3,24 +3,44 @@ Release Notes
3.5.5
-----
-- [AGIPD][CORRECT] Set max tasks per worker to 1 to avoid Out Of Memory issue for AGIPD.
+
+15-06-2022
+
+- [AGIPD][CORRECT] Expose max tasks per pool worker.
3.5.4
-----
+
+13-06-2022
+
- [AGIPD] Convert bias_voltage parameter condition to integer in cal_tools.
- [LPD] Fix correcting a single pulse.
- [LPD] VCXI require 4 modules.
3.5.3
-----
+
+19-05-2022
+
- [LPD][CORRECT] Optionally create virtual CXI files
- [LPD][CORRECT] Expose max-nodes parameter
- [AGIPD] Replace gain_choose_int by fused types
- Fix missing install of restful_config.yaml
- Fix use of xfel-calibrate --skip-report
+3.5.2
+-----
+
+16.05.2022
+
+- [LPD][CORRECT] New correction notebook for LPD
+- New `files` module to write European XFEL HDF5 corrected data files.
+
3.5.1
-----
+
+05-04-2022
+
- Calibration Constant version's new `Variant` file attribute. To indicate method of handling the constant post retrieval. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/619
- Epix100 dark Badpixels Map. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/637
- `skip-plots` flag to finish correction before plotting. https://git.xfel.eu/detectors/pycalibration/-/merge_requests/635
@@ -36,6 +56,9 @@ Release Notes
3.5.0
-----
+
+01-03-2022
+
- Updating Correction and dark notebooks for JUNGFRAU: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/518
- Updating Correction and dark notebooks for AGIPD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/535
- Updating Correction and dark notebooks for PnCCD: https://git.xfel.eu/detectors/pycalibration/-/merge_requests/559
@@ -70,6 +93,9 @@ Release Notes
3.4.3
-----
+
+20-10-2021
+
- Update pyDetLib tag.
- Add explicit dependencies on matplotlib, scipy.
- Remove outdated matplotlib rcParams setting.
@@ -93,12 +119,11 @@ Release Notes
- Resolve "Skip ZMQ tests if zmq connection for calibration DB not available".
- Reproducibility, step 1.
-
-
-
3.4.2
-----
+17-09-2021
+
- Remove driver=core from all notebook
- [webservice] Make use of Dynaconf for managing secrets.
- [webservice] Make use of dedicated slurm partitions.
@@ -123,6 +148,8 @@ Release Notes
3.4.1
-----
+16-07-2021
+
- Update h5py to 3.3
- Stop execution on notebook errors
- [AGIPD] Add integration time as operating condition to all notebooks
@@ -138,6 +165,8 @@ Release Notes
3.4.0
-----
+28-06-2021
+
- Update to Python 3.8.
- Bump numpy to 1.20.3 and remove fabio.
- remove PyQT dependency.
@@ -161,4 +190,4 @@ Release Notes
- [EPIX][CORRECT] Add Common mode correction.
- Fix plotting-related warnings.
- Test update config.
-- Test get_from_db and send_to_db.
\ No newline at end of file
+- Test get_from_db and send_to_db.
diff --git a/docs/source/configuration.rst b/docs/source/configuration.rst
index bd508db2afe3408d5f5b20e32420ebd9768513f4..750cdb5475e0f45247ca863e3efe51abd40ae805 100644
--- a/docs/source/configuration.rst
+++ b/docs/source/configuration.rst
@@ -75,40 +75,9 @@ This value should be derived e.g. by profiling memory usage per core, run times,
the above example.
The AGIPD `CORRECT` notebook (last notebook in the example) makes use of a concurrency generating function
-by setting the `use function` parameter. This function must be defined in a code cell in the notebook,
-its parameters should be named like other exposed parameters. It should return a list of of parameters
-to be inserted into the concurrently run notebooks. The example given e.g. defines the `balance_sequences`
-function::
-
- def balance_sequences(in_folder, run, sequences, sequences_per_node):
- import glob
- import re
- import numpy as np
- if sequences_per_node != 0:
- sequence_files = glob.glob("{}/r{:04d}/*-S*.h5".format(in_folder, run))
- seq_nums = set()
- for sf in sequence_files:
- seqnum = re.findall(r".*-S([0-9]*).h5", sf)[0]
- seq_nums.add(int(seqnum))
- seq_nums -= set(sequences)
- return [l.tolist() for l in np.array_split(list(seq_nums),
- len(seq_nums)//sequences_per_node+1)]
- else:
- return sequences
-
-TESTING: :func:`balance_sequences`
-
-.. note::
-
- Note how imports are inlined in the definition. This is necessary, as only the function code,
- not the entire notebook is executed.
-
-which requires as exposed parameters e.g. ::
-
- in_folder = "/gpfs/exfel/exp/SPB/201701/p002038/raw/" # the folder to read data from, required
- run = 239 # runs to process, required
- sequences = [-1] # sequences to correct, set to -1 for all, range allowed
- sequences_per_node = 2 # number of sequence files per cluster node if run as slurm job, set to 0 to not run SLURM parallel
+by setting the `use function` parameter. This function must be defined in the first cell in the notebook
+its given arguments should be named as the first cell notebook parameters. It is expected to return a list of parameters
+to concurrently run notebooks. Above the used function is :func:`xfel_calibrate.calibrate.balance_sequences`.
.. note::
diff --git a/docs/source/index.rst b/docs/source/index.rst
index cc3e055ea6c82ffbd12c31a34b446115a380c458..b57a68dee27860df4384878f2116ba92a7c4740f 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -26,7 +26,7 @@ where `DETECTOR` and `TYPE` specify the task to be performed.
Additionally, it leverages the DESY/XFEL Maxwell cluster to run these jobs in parallel
via SLURM_.
-Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are
+Here is a list of :ref:`available_notebooks`. See the :ref:`advanced_topics` if you are looking
for details on how to use as detector group staff.
If you would like to integrate additional notebooks please see the :ref:`development_workflow`.
diff --git a/docs/source/installation.rst b/docs/source/installation.rst
index 594f54ada19351e1d9efd9cc39b16d75f5338e95..97d83ad8c40fa9c4b84a30125a671689173ab6e3 100644
--- a/docs/source/installation.rst
+++ b/docs/source/installation.rst
@@ -54,33 +54,6 @@ Copy/paste script:
python3 -m pip install . # `-e` flag for editable install, e.g. `pip install -e .`
-Installation into user home directory
-=====================================
-
-This is not recommended as `pycalibration` has pinned dependencies for
-stability, if you install it directly into you users home environment then it
-will downgrade/upgrade your local packages, which may cause major issues and may
-**break your local environment**, it is highly recommended to use the venv
-installation method instead.
-
-1. ``source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate``
-2. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
-3. ``pyenv shell 3.8.11`` - load required version of python
-4. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
-5. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable
-
-Copy/paste script:
-
-.. code:: bash
-
- source /gpfs/exfel/sw/calsoft/.pyenv/bin/activate
- git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
- pyenv shell 3.8.11
- cd pycalibration
- pip install --user . # `-e` flag for editable install, e.g. `pip install -e .`
- export PATH=$HOME/.local/bin:$PATH
-
-
Creating an ipython kernel for virtual environments
===================================================
diff --git a/docs/source/tutorial.rst b/docs/source/tutorial.rst
index 73018ff118c4c350ea3e6b17e6a398f5264da329..450b00464705cdd887f27264bcd3da05a533cadd 100644
--- a/docs/source/tutorial.rst
+++ b/docs/source/tutorial.rst
@@ -53,7 +53,7 @@ Running the notebook
You can see your job in the queue with::
- squeue -u $USERNAME
+ squeue --me
3. Look at the generated report in the chosen output folder.
4. More information on the job run on the cluster can be found in the temp folder.
diff --git a/docs/source/workflow.rst b/docs/source/workflow.rst
index a9369b4fb016aab521711f070809daa328e4116a..bb0264f46af8d5c5d2dbb1db8e25ae9d1ec46e4a 100644
--- a/docs/source/workflow.rst
+++ b/docs/source/workflow.rst
@@ -4,7 +4,7 @@ Development Workflow
====================
The following walkthrough will guide you through a possible workflow
-when developing new notebook for offline calibration.
+when developing new notebooks for offline calibration.
Fresh Start
-----------
@@ -171,18 +171,18 @@ In principle there a not restrictions other than that parameters that are expose
command line need to be defined in the first code cell of the notebook.
However, a few guidelines should be observed to make notebook useful for display as
-reports and usage by other.
+reports and usage by others.
External Libraries
~~~~~~~~~~~~~~~~~~
-You may use a wide variaty of libraries available in Python, but keep in mind that others
+You may use a wide variety of libraries available in Python, but keep in mind that others
wanting to run the tool will need to install these requirements as well. Thus,
-* do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
+* Do not use a specialized tool if an accepted alternative exists. Plots e.g. should usually
be created using matplotlib_ and numerical processing should be done in numpy_.
-* keep runtime and library requirements in mind. A library doing its own parallelism either
+* Keep runtime and library requirements in mind. A library doing its own parallelism either
needs to programmatically be able to set this up, or automatically do so. If you need to
start something from the command line first, things might be tricky as you will likely
need to run this via `POpen` commands with appropriate environment variable.
@@ -298,4 +298,4 @@ documentation.
.. _matplotlib: https://matplotlib.org/
.. _numpy: http://www.numpy.org/
.. _h5py: https://www.h5py.org/
-.. _iCalibrationDB: https://in.xfel.eu/readthedocs/docs/icalibrationdb/en/latest/
+.. _iCalibrationDB: https://git.xfel.eu/detectors/cal_db_interactive
diff --git a/docs/source/xfel_calibrate_conf.rst b/docs/source/xfel_calibrate_conf.rst
index 52d1bdf5cf0fe28f1a299e5e057d36927d1de97b..4fed8d2650a0ab6a4713ac853392890976d8bc39 100644
--- a/docs/source/xfel_calibrate_conf.rst
+++ b/docs/source/xfel_calibrate_conf.rst
@@ -4,6 +4,3 @@ xfel_calibrate
.. module:: xfel_calibrate.calibrate
.. autofunction:: balance_sequences
-
-
-:py:mod: xfel_calibrate.notebooks
\ No newline at end of file