diff --git a/README.rst b/README.rst index 6328ee6186d55220d620f7704c1967fbf829ef1a..9cb0c8b14c56234fd5c50c48b1ca5cb1e09d18ba 100644 --- a/README.rst +++ b/README.rst @@ -1,146 +1,127 @@ +################### Offline Calibration -=================== +################### The offline calibration is a package that consists of different services, responsible for applying most of the offline calibration and characterization for the detectors. -Offline calibration installation -================================ -It's recommended to install the offline calibration (pycalibration) package -over maxwell, using anaconda/3 environment. +Offline Calibration Installation +******************************** -Installation using Anaconda ---------------------------- +It's recommended to install the offline calibration (pycalibration) package over +maxwell, using anaconda/3 environment. -First you need to load the anaconda/3 environment through:: - 1. module load anaconda/3 +Installation using python virtual environment - recommended +=========================================================== -If installing into other python enviroments, this step can be skipped. +1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab +2. ``module load anaconda/3`` - load the anaconda/3 environment +3. ``python3 -m venv .venv`` - create the virtual environment +4. ``source .venv/bin/activate`` - activate the virtual environment +5. ``python3 -m pip install --upgrade pip`` - upgrade version of pip +6. ``python3 -m pip install -r requirements.txt`` - install dependencies +7. ``python3 -m pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation) +8. ``pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib"`` -Then the package for the offline calibration can be obtained from the git repository:: +Copy/paste script: - 2. git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git +.. code:: + git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git + cd pycalibration + module load anaconda/3 + python3 -m venv .venv + source .venv/bin/activate + python3 -m pip install --upgrade pip + python3 -m pip install -r requirements.txt + python3 -m pip install . # `-e` flag for editable install + python3 -m pip install "git+ssh://git@git.xfel.eu:10022/karaboDevices/pyDetLib.git#egg=XFELDetectorAnalysis&subdirectory=lib/" -You can then install all requirements of this tool chain in your home directory by running:: - 3. pip install -r requirements.txt . --user +Installation into user home directory +===================================== -in pycalibration's root directory. +1. ``git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git && cd pycalibration`` - clone the offline calibration package from EuXFEL GitLab +2. ``module load anaconda/3`` - load the anaconda/3 environment. If installing into other python environments, this step can be skipped +3. ``pip install -r requirements.txt`` - install all requirements of this tool chain in your home directory +4. ``pip install .`` - install the pycalibration package (add ``-e`` flag for editable development installation) +5. ``export PATH=$HOME/.local/bin:$PATH`` - make sure that the home directory is in the PATH environment variable -After installation, you should make sure that the home directory is in the PATH environment variable:: +Copy/paste script: - 4. export PATH=$HOME/.local/bin:$PATH +.. code:: -Installation using virtual python environment ---------------------------------------------- + git clone ssh://git@git.xfel.eu:10022/detectors/pycalibration.git + cd pycalibration + module load anaconda/3 + pip install -r requirements.txt --user + pip install . # `-e` flag for editable install, e.g. `pip install -e .` + export PATH=$HOME/.local/bin:$PATH -Create virtual environment:: - module load anaconda/3 - python -m venv /path/to/new/virtual/environment - source /path/to/new/virtual/environment/bin/activate +Creating an ipython kernel for virtual environments +=================================================== -Clone from git:: +To create an ipython kernel with pycalibration available you should (if using a +venv) activate the virtual environment first, and then run: - cd /path/to/packages - git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git - cd pycalibration +.. code:: -Install the package:: - - pip install -r requirements.txt - -In additional install pyDetLib package, which is required for many notebooks:: - - cd /path/to/packages - git clone https://git.xfel.eu/gitlab/karaboDevices/pyDetLib.git - cd pyDetLib/lib - pip install -r requirements.txt - pip install . - -++++++++++++++++++++++++++++++++++++++++++++++++++ -Setting an ipython kernel for virtual environments -++++++++++++++++++++++++++++++++++++++++++++++++++ - -To set a kernel for your virtual environment:: - - source /path/to/new/virtual/environment/bin/activate - pip install ipykernel - python -m ipykernel install --user --name <virtenv-name> --display-name "virtenv-display-name" + python3 -m pip install ipykernel # If not using a venv add `--user` flag + python3 -m ipykernel install --user --name pcalibration --display-name "pcalibration" # If not using a venv pick different name This can be useful for Jupyter notebook tools as "max-jhub.desy.de". -Development Installation ------------------------- - -For a development installation, which automatically -picks up (most) changes, first install the dependencies as above, -but then install the tool-chain separately in development mode (install in home directory using --user, in case of using Anaconda/3):: - - pip install -e . +Python Scripted Calibration +=========================== -Activate Offline calibration -============================ - -For using pycalibration package one needs to activate it through:: - - source activate - -from inside of the pycalibration directory. This will automatically load -all needed modules and export the $PATH for the home directory. +**Do not run this on the Maxwell gateway**. Rather, ``salloc`` a node for +yourself first: +.. code:: -Python Scripted Calibration -=========================== + salloc -p exfel/upex -t 01:00:00 -First: do not run this on the Maxwell gateway. Rather, `salloc` -a node for yourself first:: +where `-p` gives the partition to use: exfel or upex and `-t` the duration the +node should be allocated. Then `ssh` onto that node. - salloc -p exfel/upex -t 01:00:00 +Then activate your environment as described above (or just continue if you are +not using a venv). -where `-p` gives the partition to use: exfel or upex and `-t` -the duration the node should be allocated. Then `ssh` onto -that node. +If running headless (i.e. without X forwarding), be sure to set +``MPLBACKEND=Agg``, via: -(optionally) Set up the environment:: + export MPLBACKEND=Agg - module load python3 - pip install --user ipython --upgrade - pip install --user ipyparallel --upgrade - pip install --user dill - -If running headless (i.e. without X forwarding), be sure to set -`MPLBACKEND=Agg`, via:: +Then start an ``ipcluster``. If you followed the steps above this can be done +via: - export MPLBACKEND=Agg +.. code:: -Then start an `ipcluster`. If you followed the steps above this can be done -via:: + ipcluster start --n=32 - ~/.local/bin/ipcluster start --n=32 +Finally run the script: -Run the script:: +.. code:: python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \ - --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1 + --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1 Here `--input` should point to a directory of `RAW` files for the detector you -are calibrating. They will be output into the folder specified by `--output`, -which will have the run number or the last folder in the hiearchy of the input -appended. Additionally, you need to specify the number of `--mem-cells` used -for the run, as well as the `--detector`. Finally, you can optionally -specify to only process certain `--sequences` of files, matching the sequence -numbers of the `RAW` input. These should be given as a comma-separated list. - -Finally, there is a `--no-relgain` option, which disables relative gain +are calibrating. They will be output into the folder specified by `--output`, +which will have the run number or the last folder in the hierarchy of the input +appended. Additionally, you need to specify the number of `--mem-cells` used for +the run, as well as the `--detector`. Finally, you can optionally specify to +only process certain `--sequences` of files, matching the sequence numbers of +the `RAW` input. These should be given as a comma-separated list. + +Finally, there is a `--no-relgain` option, which disables relative gain correction. This can be useful while we still further characterize the detectors -to provid accurate relative gain correction constants. +to provide accurate relative gain correction constants. You'll get a series of plots in the output directory as well. -