# Advanced Topics
The following tasks should only be carried out by trained staff.
Request dark characterization
-----------------------------
The script runs dark characterization notebook with default parameters
via web service. The data needs to be transferred via the MDC, however,
the web service will wait for any transfer to be completed. The detector
is chosen automatically with respect to given instrument
([\--instrument]). For AGIPD, LPD, Jungfrau runs for the
three gain stages need to be given ([\--run-high],
[\--run-med], [\--run-low]). For FastCCD and
ePIX only a single run needs to be given ([\--run]).
The complete list of parameters is:
-h, --help show this help message and exit
--proposal PROPOSAL The proposal number, without leading p, but with
leading zeros
--instrument {SPB,MID,FXE,SCS,SQS,HED}
The instrument
--cycle CYCLE The facility cycle
--run-high RUN_HIGH Run number of high gain data as an integer
--run-med RUN_MED Run number of medium gain data as an integer
--run-low RUN_LOW Run number of low gain data as an integer
--run RUN Run number as an integer
The path to data files is defined from script parameters. A typical data
path, which can be found in the MDC is:
/gpfs/exfel/exp/MID/201930/p900071/raw
Where [MID] is an instrument name, [201930] is a
cycle, [900071] is a proposal number.
Extending Correction Notebooks on User Request
----------------------------------------------
Internally, each automated correction run will trigger
[calibrate\_nbc.py] to be called anew on the respective
notebook. This means that any changes to save to this notebook will be
picked up for subsequent runs.
This can be useful to add user requests while running. For this:
1. create a working copy of the notebook in question, and create a
commit of the production notebook to fall back to in case of
problems:
> `` ` git add production_notebook_NBC.py git commit -m "Known working version before edits" cp production_notebook_NBC.py production_notebook_TEST.py ``\`
2. add any feature there and *thoroughly* test them
3. when you are happy with the results, copy them over into the
production notebook and save.
???+ warning
Live editing of correction notebooks is fully at your responsibility. Do
not do it if you are not 100% sure you know what you are doing.
1. If it fails, revert to the original state, ideally via git:
git checkout HEAD -- production_notebook_NBC.py
2. Any runs which did not correct do to failures of the live edit can
then be relaunched manually, assuming the correction notebook allows
run and overwrite parameters:
xfel-calibrate ...... --run XYZ,ZXY-YYS --overwrite
Using a Parameter Generator Function
------------------------------------
By default, the parameters to be exposed to the command line are deduced
from the first code cell of the notebook, after resolving the notebook
itself from the detector and characterization type. For some
applications it might be beneficial to define a context-specific
parameter range within the same notebook, based on additional user
input. This can be done via a parameter generation function which is
defined in one of the code cell:
def extend_parms(detector_instance):
from iCalibrationDB import Conditions
import inspect
existing = set()
def extract_parms(cls):
args, varargs, varkw, defaults = inspect.getargspec(cls.__init__)
pList = []
for i, arg in enumerate(args[1:][::-1]):
if arg in existing:
continue
existing.add(arg)
if i < len(defaults):
default = defaults[::-1][i]
if str(default).isdigit():
pList.append("{} = {}".format(arg, default))
elif default is None or default == "None":
pList.append("{} = \"None\"".format(arg))
else:
pList.append("{} = \"{}\"".format(arg, default))
else:
pList.append("{} = 0. # required".format(arg))
return set(pList[::-1]) # mandatories first
dtype = "LPD" if "LPD" in detector_instance.upper() else "AGIPD"
all_conditions = set()
for c in dir(Conditions):
if c[:2] != "__":
condition = getattr(Conditions, c)
parms = extract_parms(getattr(condition, dtype))
[all_conditions.add(p) for p in parms]
return "\n".join(all_conditions)
???+ note
Note how all imports are inlined, as the function is executed outside
the notebook context.
In the example, the function generates a list of additional parameters
depending on the [detector\_instance] given. Here,
[detector\_instance] is defined in the first code cell the
usual way. Any other parameters defined such, that have names matching
those of the generator function signature are passed to this function.
The function should then return a string containing additional code to
be appended to the first code cell.
To make use of this functionality, the parameter generator function
needs to be configured in [notebooks.py], e.g. :
...
"GENERIC": {
"DBTOH5": {
"notebook": "notebooks/generic/DB_Constants_to_HDF5_NBC.ipynb",
"concurrency": {"parameter": None,
"default concurrency": None,
"cluster cores": 32},
"extend parms": "extend_parms",
},
}
...
To generically query which parameters are defined in the first code
cell, the code execution history feature of iPython can be used:
ip = get_ipython()
session = ip.history_manager.get_last_session_id()
first_cell = next(ip.history_manager.get_range(session, 1, 2, raw=True))
_, _, code = first_cell
code = code.split("\n")
parms = {}
for c in code:
n, v = c.split("=")
n = n.strip()
v = v.strip()
try:
parms[n] = float(v)
except:
parms[n] = str(v) if not isinstance(v, str) else v
if parms[n] == "None" or parms[n] == "'None'":
parms[n] = None
This will create a dictionary [parms] which contains all
parameters either as [float] or [str] values.