Update README about running & re-running calibration

README.rst

@@ -349,56 +349,51 @@ unmodified sections of code being looked at.
 Python Scripted Calibration
 ***************************
 
-**Do not run this on the Maxwell gateway**. Rather, ``salloc`` a node for
-yourself first:
+To launch correction or characterisation jobs, run something like this::
 
-.. code::
-
-  salloc -p exfel/upex -t 01:00:00
+    xfel-calibrate AGIPD CORRECT \
+    --in-folder /gpfs/exfel/exp/SPB/202131/p900215/raw --run 591 \
+    --out-folder /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900215-591 \
+    --karabo-id SPB_DET_AGIPD1M-1 --karabo-id-control SPB_IRU_AGIPD1M1 \
+    --karabo-da-control AGIPD1MCTRL00 --modules 0-4
 
-where `-p` gives the partition to use: exfel **or** upex and `-t` the duration
-the node should be allocated. Then ``ssh`` onto that node.
+The first two arguments refer to a *detector* and an *action*, and are used to
+find the appropriate notebook to run. Most of the optional arguments are
+translated into parameter assignments in the notebook, e.g. ``--modules 0-4``
+sets ``modules = [0, 1, 2, 3]`` in the notebook.
 
-Then activate your environment as described above (or just continue if you are
-not using a venv).
+This normally submits jobs to Slurm to do the work; you can check their status
+with ``squeue --me``. If you are working on a dedicated node, you can use the
+``--no-cluster-job`` option to run all the work on that node instead.
 
-If running headless (i.e. without X forwarding), be sure to set
-``MPLBACKEND=Agg``, via:
-
-.. code::
+The notebooks will be used to create a PDF report after the jobs have run.
+This will be placed in ``--out-folder`` by default, though it can be overridden
+with the ``--report-to`` option.
 
-  export MPLBACKEND=Agg
-
-Then start an ``ipcluster``. If you followed the steps above this can be done
-via:
-
-.. code::
-
-  ipcluster start --n=32
-
-
-Finally run the script:
-
-.. code::
+Reproducing calibration
+=======================
 
-  python3 calibrate.py --input /gpfs/exfel/exp/SPB/201701/p002012/raw/r0100 \
-    --output ../../test_out --mem-cells 30 --detector AGIPD --sequences 0,1
+The information to run the calibration code again is saved to a directory next to
+the PDF report, named starting with ``slurm_out_``. It can be run as a new job
+like this::
 
-Here ``--input`` should point to a directory of ``RAW`` files for the detector
-you are calibrating. They will be output into the folder specified by
-``--output``, which will have the run number or the last folder in the hierarchy
-of the input appended. Additionally, you need to specify the number of
-``--mem-cells`` used for the run, as well as the ``--detector``. Finally, you
-can optionally specify to only process certain ``--sequences`` of files,
-matching the sequence numbers of the `RAW` input. These should be given as a
-comma-separated list.
+python3 -m xfel_calibrate.repeat \
+    /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900215-591/slurm_out_AGIPDOfflineCorrection \
+    --out-folder /gpfs/exfel/data/scratch/kluyvert/agipd-calib-900215-591-repro
 
-Finally, there is a ``--no-relgain`` option, which disables relative gain
-correction. This can be useful while we still further characterize the detectors
-to provide accurate relative gain correction constants.
+The information in the directory includes a Pip ``requirements.txt`` file
+listing the packages installed when this task was first set up. For better
+reproducibility, use this to create a similar environment, and pass
+``--python path/to/bin/python`` to run notebooks in that environment.
+Future work will automate this step.
 
-You'll get a series of plots in the output directory as well.
+.. note::
 
+   Our aim here is to run the same code as before, with the same parameters,
+   in a similar software environment. This should produce essentially the same
+   results, but not necessarily exactly identical. The code which runs may
+   use external resources, or involve some randomness, and even different
+   hardware may make small differences.
 
 Appendix
 ********