Skip to content
Snippets Groups Projects

Feat/pre commit checks

Merged Feat/pre commit checks
feat/pre-commit-checks into master
Merged Robert Rosca requested to merge feat/pre-commit-checks into master
Viewing commit d6b83d8b
Prev Next
Show latest version
1 file
+ 78
− 97
Compare changes
  • Side-by-side
  • Inline
  • d6b83d8b
    Robert Rosca
    Update README.rst · d6b83d8b
    Robert Rosca authored 
    Change to be compliant with RSTcheck
Fix some heading orders and enumerated lists
Update the setup instructions
README.rst
+ 78
− 97
###################
Offline Calibration
===================
###################
The offline calibration is a package that consists of different services,
responsible for applying most of the offline calibration and characterization
for the detectors.
Offline calibration installation
================================
It's recommended to install the offline calibration (pycalibration) package
over maxwell, using anaconda/3 environment.
Offline Calibration Installation
********************************
Installation using Anaconda
---------------------------
It's recommended to install the offline calibration (pycalibration) package over
maxwell, using anaconda/3 environment.
First you need to load the anaconda/3 environment through::
1. module load anaconda/3
Installation using python virtual environment - recommended
===========================================================
If installing into other python enviroments, this step can be skipped.
1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
2. ``module load anaconda/3`` - load the anaconda/3 environment
3. ``python3 -m venv .venv`` - create the virtual environment
4. ``source .venv/bin/activate`` - activate the virtual environment
5. ``python3 -m pip install --upgrade pip`` - upgrade version of pip
6. ``python3 -m pip install -r requirements.txt`` - install dependencies
7. ``python3 -m pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
8. ``pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib"``
Then the package for the offline calibration can be obtained from the git repository::
Copy/paste script:
2. git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
.. code::
git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
module load anaconda/3
python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -r requirements.txt
python3 -m pip install . # `-e` flag for editable install
python3 -m pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib/"
You can then install all requirements of this tool chain in your home directory by running::
3. pip install -r requirements.txt . --user
Installation into user home directory
=====================================
in pycalibration's root directory.
1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab
2. ``module load anaconda/3`` - load the anaconda/3 environment. If installing into other python environments, this step can be skipped
3. ``pip install -r requirements.txt`` - install all requirements of this tool chain in your home directory
4. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation)
5. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable
After installation, you should make sure that the home directory is in the PATH environment variable::
Copy/paste script:
4. export PATH=$HOME/.local/bin:$PATH
.. code::
Installation using virtual python environment
---------------------------------------------
git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git
cd pycalibration
module load anaconda/3
pip install -r requirements.txt --user
pip install . # `-e` flag for editable install, e.g. `pip install -e .`
export PATH=$HOME/.local/bin:$PATH
Create virtual environment::
module load anaconda/3
python -m venv /path/to/new/virtual/environment
source /path/to/new/virtual/environment/bin/activate
Creating an ipython kernel for virtual environments
===================================================
Clone from git::
To create an ipython kernel with pycalibration available you should (if using a
venv) activate the virtual environment first, and then run:
cd /path/to/packages
git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
cd pycalibration
.. code::
Install the package::
pip install -r requirements.txt
In additional install pyDetLib package, which is required for many notebooks::
cd /path/to/packages
git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git
cd pyDetLib/lib
pip install -r requirements.txt
pip install .
++++++++++++++++++++++++++++++++++++++++++++++++++
Setting an ipython kernel for virtual environments
++++++++++++++++++++++++++++++++++++++++++++++++++
To set a kernel for your virtual environment::
source /path/to/new/virtual/environment/bin/activate
pip install ipykernel
python -m ipykernel install --user --name <virtenv-name> --display-name "virtenv-display-name"
python3 -m pip install ipykernel # If not using a venv add `--user` flag
python3 -m ipykernel install --user --name pcalibration --display-name "pcalibration" # If not using a venv pick different name
This can be useful for Jupyter notebook tools as "max-jhub.desy.de".
Development Installation
------------------------
For a development installation, which automatically
picks up (most) changes, first install the dependencies as above,
but then install the tool-chain separately in development mode (install in home directory using --user, in case of using Anaconda/3)::
pip install -e .
Python Scripted Calibration
===========================
Activate Offline calibration
============================
For using pycalibration package one needs to activate it through::
source activate
from inside of the pycalibration directory. This will automatically load
all needed modules and export the $PATH for the home directory.
**Do not run this on the Maxwell gateway**. Rather, ``salloc`` a node for
yourself first:
.. code::
Python Scripted Calibration
===========================
salloc -p exfel/upex -t 01:00:00
First: do not run this on the Maxwell gateway. Rather, `salloc`
a node for yourself first::
where `-p` gives the partition to use: exfel or upex and `-t` the duration the
node should be allocated. Then `ssh` onto that node.
salloc -p exfel/upex -t 01:00:00
Then activate your environment as described above (or just continue if you are
not using a venv).
where `-p` gives the partition to use: exfel or upex and `-t`
the duration the node should be allocated. Then `ssh` onto
that node.
If running headless (i.e. without X forwarding), be sure to set
``MPLBACKEND=Agg``, via:
(optionally) Set up the environment::
export MPLBACKEND=Agg
module load python3
pip install --user ipython --upgrade
pip install --user ipyparallel --upgrade
pip install --user dill
If running headless (i.e. without X forwarding), be sure to set
`MPLBACKEND=Agg`, via::
Then start an ``ipcluster``. If you followed the steps above this can be done
via:
export MPLBACKEND=Agg
.. code::
Then start an `ipcluster`. If you followed the steps above this can be done
via::
ipcluster start --n=32
~/.local/bin/ipcluster start --n=32
Finally run the script:
Run the script::
.. code::
python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
--output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1
--output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1
Here `--input` should point to a directory of `RAW` files for the detector you
are calibrating. They will be output into the folder specified by `--output`,
which will have the run number or the last folder in the hiearchy of the input
appended. Additionally, you need to specify the number of `--mem-cells` used
for the run, as well as the `--detector`. Finally, you can optionally
specify to only process certain `--sequences` of files, matching the sequence
numbers of the `RAW` input. These should be given as a comma-separated list.
Finally, there is a `--no-relgain` option, which disables relative gain
are calibrating. They will be output into the folder specified by `--output`,
which will have the run number or the last folder in the hierarchy of the input
appended. Additionally, you need to specify the number of `--mem-cells` used for
the run, as well as the `--detector`. Finally, you can optionally specify to
only process certain `--sequences` of files, matching the sequence numbers of
the `RAW` input. These should be given as a comma-separated list.
Finally, there is a `--no-relgain` option, which disables relative gain
correction. This can be useful while we still further characterize the detectors
to provid accurate relative gain correction constants.
to provide accurate relative gain correction constants.
You'll get a series of plots in the output directory as well.

Legal Notice | Contacts