Skip to content
Snippets Groups Projects
Commit 2a79f82c authored by Steffen Hauf's avatar Steffen Hauf
Browse files

Merge branch 'master' into 'fix/larger_data' 

# Conflicts:
#   xfel_calibrate/calibrate.py
#   xfel_calibrate/notebooks.py
parents 030c8ac6 6abdd249
No related branches found
No related tags found
1 merge request!23Fix/larger data
Expand all Hide whitespace changes
Inline Side-by-side
docs/source/index.rst
+ 1
− 1
View file @ 2a79f82c
......@@ -17,7 +17,7 @@ Contents:
workflow
available_notebooks
advanced
tutorial
Indices and tables
......
docs/source/installation.rst
+ 15
− 7
View file @ 2a79f82c
......@@ -6,6 +6,8 @@ done into a Karabo_ Python_ environment. However, there are no explicit
Karabo dependencies, so any Python3 environment is capable of running
the tools.
A more detailed step-by-step intruction can be found in the :ref:`tutorial`.
Prerequisites
-------------
......@@ -36,15 +38,20 @@ by sourcing the activate file in the Karabo path::
source karabo/activate
For other python enviroments, this step can be skipped. You can then install
all prerequisites and this tool chain by running::
For other python enviroments, this step can be skipped. The package for the offline
calibration can be obtained from the git repository::
git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
You can then install all prerequisites and this tool chain by running::
pip install -r requirements.txt .
After installation, you should adjust the :ref:`configuration` and settings to
match your environment.
in th emain directoy pycalibration.
After installation, you should adjust the :ref:`configuration` and settings in the folder
xfel-calibrate to match your environment.
The toolchain is then available via the::
The tool-chain is then available via the::
xfel-calibrate
......@@ -54,7 +61,7 @@ Development Installation
------------------------
For a development installation, which automatically picks up (most) changes,
first install the dependencies as above, but then install the took-chain
first install the dependencies as above, but then install the tool-chain
separately in development mode::
pip install -e .
......@@ -71,4 +78,5 @@ the the tool-chain, without prerequisites, from the package base directory::
Or, in case you are activily developing::
pip install -e .
\ No newline at end of file
pip install -e .
docs/source/tutorial.rst 0 → 100644
+ 96
− 0
View file @ 2a79f82c
.. _tutorial:
Tutorial
========
The goal of this tutorial is to demonstrate the functionality of the offline calibration tool-chain. The main functionality offered by this package is the possibility to run a notebook on the shell with input parameters for the configuration. Extending that concept the package also takes care of starting the necessary jobs on the maxwell cluster, which can be more than one if the notebook makes use of ipyparallel. Finally the pycalibration package will generate a report containing all markup and result cells of the notebook.
The Tutorial consist of this documentation and two very simple notebooks:
1. notebooks/Tutorial/startversion.ipynb
A simple notebook with no knowledge of the requirements of the offline calibration.
2. notebooks/Tutorial/calversion.ipynb
Outcome of adapting the startversion notebook to be able to be run with the offline
calibration tool-chain.
To have a look at those notebooks start from a shell with the karabo environment::
jupyter-notebook
This will open a jupyter kernel running in your browser where you can then open the notebooks in the folder notebooks/Tutorial. If you in addition also start on another shell the ipcluster as instructed in the calversion.ipynb notebook::
ipcluster start --n=4 --profile=tutorial
you can step through the cells and run them.
If you run this notebook using the xfel-calibrate command as explaind at the end of this tutorial you do not need to start the cluster yourself, it will be done by the framework.
Installation and Configuration
------------------------------
The offline calibration tool-chain is optimised to run on the maxwell cluster.
For more information refer to the Maxwell_ documentation.
.. _Maxwell: https://confluence.desy.de/display/IS/Running+Jobs+on+Maxwell
In order to use the offline calibration tool a few steps need to be carried out
to install the necessary packages and setup the environment:
1. Log into max-exfl with you own user name/account.
2. Install karabo in your home directory or under /gpfs/exfel/data/scratch/username
by typing the following commands on you shell::
wget http://exflserv05.desy.de/karabo/karaboFramework/tags/2.2.4/karabo-2.2.4-Release-CentOS-7-x86_64.sh
chmod +x karabo-2.2.4-Release-CentOS-7-x86_64.sh
./karabo-2.2.4-Release-CentOS-7-x86_64.sh
source karabo/activate
3. Get the package pycalibration which contains the offline calibration tool-chain::
git clone https://git.xfel.eu/gitlab/detectors/pycalibration.git
4. Install the necessary requirements and the package itself::
cd pycalibration
pip install -r requirements.txt .
5. Adjust xfel_calibrate/settings.py by changing "karabo_activate_path"
and "ipcluster_path according" to where you installed karabo.
Create your own notebook
------------------------
1. Create a new notebook or re-arrange an existing following the guidelines of this documentation
2. Register you notebook by adding an entry to xfel_calibrate/notebooks.py following
the structure given by the existing notebooks.
Note: Use all capital letters for DETECTOR and TYPE.
3. Load/register the new notebook by updating the installation::
pip install -e .
Running the notebook
--------------------
1. Make sure output folders you want to use exist
2. To run your notebook::
xfel-calibrate Tutorial TEST
You can see your job in the queue with::
squeue -u username
3. Look at the generated report in the chosen output folder.
4. More information on the job run on the cluster can be found in the temp folder.
notebooks/Tutorial/calversion.ipynb 0 → 100644
+ 321
− 0
View file @ 2a79f82c
This diff is collapsed.
notebooks/Tutorial/startversion.ipynb 0 → 100644
+ 134
− 0
View file @ 2a79f82c
%% Cell type:markdown id: tags:
Create a small simple notebook, without thinking about parameters or maxwell
%% Cell type:code id: tags:
``` python
import matplotlib
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
```
%% Cell type:code id: tags:
``` python
sensor_size = [10,20]
random_seed = 2345
np.random.seed = random_seed
fake_data = np.random.random((sensor_size))
```
%% Cell type:code id: tags:
``` python
plt.imshow(fake_data, interpolation="nearest")
plt.colorbar()
```
%% Output
<matplotlib.colorbar.Colorbar at 0x7f9f38551d30>
%% Cell type:markdown id: tags:
Calculate some stuff from the data
%% Cell type:code id: tags:
``` python
my_mean = fake_data.mean()
my_std = fake_data.std()
my_mean, my_std
```
%% Output
(0.5116986708989778, 0.29529469463486308)
%% Cell type:code id: tags:
``` python
```
xfel_calibrate/calibrate.py
+ 13
− 5
View file @ 2a79f82c
......@@ -241,11 +241,11 @@ elif len(sys.argv) >= 3:
else "{}. Default: %(default)s".format(p.comment.replace("#", " ").strip()))
required = (p.comment is not None and "required" in p.comment) and not overwrite_reqs
if p.type == list:
if len(p.value):
if p.type == list or p.name == cvar:
if p.type is list and len(p.value):
ltype = type(p.value[0])
else:
ltype = str
elif p.type is not list:
ltype = p.type
range_allowed = "RANGE ALLOWED" in p.comment.upper() if p.comment else False
parser.add_argument("--{}".format(consolize_name(p.name)),
nargs='+',
......@@ -260,6 +260,7 @@ elif len(sys.argv) >= 3:
default=p.value if not required else None,
help=helpstr,
required=required)
else:
parser.add_argument("--{}".format(consolize_name(p.name)),
type=p.type,
......@@ -531,9 +532,16 @@ def run():
cvals = f(*callargs)
print("Split concurrency into {}".format(cvals))
# get expected type
cvtype = list
for p in parms:
if p.name == cvar:
cvtype = p.type
break
for cnum, cval in enumerate(cvals):
jobid = concurrent_run(run_tmp_path, nb, notebook, args,
cvar, [cval,] if not isinstance(cval, list) else cval,
cvar, [cval,] if not isinstance(cval, list) and cvtype is list else cval,
cnum==len(list(cvals))-1, joblist, fmtcmd,
cluster_cores=cluster_cores, sequential=sequential)
joblist.append(jobid)
......
xfel_calibrate/notebooks.py
+ 9
− 2
View file @ 2a79f82c
......@@ -90,6 +90,13 @@ notebooks = {
"cluster cores": 32},
"extend parms": "extend_parms",
},
}
},
"TUTORIAL": {
"TEST": {
"notebook": "notebooks/Tutorial/calversion.ipynb",
"concurrency": {"parameter": "random_seed",
"default concurrency": None,
"cluster cores": 32},
},
},
}
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

Legal Notice | Contacts