Skip to content

[DARK] Save paths to old constants instead of copying ([AGIPD][LPD][DSSC])

David Hammer requested to merge feat/agipd-darks-save-old-const-paths into feat/agipd-darks-speedup

Description

One change made in https://git.xfel.eu/gitlab/detectors/pycalibration/merge_requests/465 is to stop copying the data from retrieved old constants to local output folder + /old. This of course breaks the summary notebook which is run for AGIPD, LPD, and DSSC as it expects to find these old files there.

This MR would make the three notebooks simply save information about where the constants are already residing (within the CalCat store on GPFS) in YAML files. The summary notebook will then look for these YAML files.

How Has This Been Tested?

  1. Ran for two AGIPD modules via xfel-calibrate, testing the "AGIPD to summary" path + the "summary to report" path.
  2. Ran LPD notebook with two modules selected, testing the "LPD to summary" path
  3. Ran DSSC notebook with two modules selected, testing the "DSSC to summary" path

For 1, the report at the end looks pretty normal. calibration_metadata.yml as well looks normal; the modules-mapping part which now comes from the combined module metadata files has not broken:

modules-mapping:
  Q2M4: AGIPD_SIV1_AGIPDV11_M313
  Q3M2: AGIPD_SIV1_AGIPDV11_M306

A module_metadata file looks like:

module: Q2M4
old-constants:
  BadPixelsDark:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/agipd-type/agipd_siv1_agipdv11_m313/cal.1618912652.438402.h5
    h5path: //AGIPD_SIV1_AGIPDV11_M313/BadPixelsDark/0
    timestamp: '2021-04-20T10:59:15+02:00'
  Noise:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/agipd-type/agipd_siv1_agipdv11_m313/cal.1618912643.533352.h5
    h5path: //AGIPD_SIV1_AGIPDV11_M313/Noise/0
    timestamp: '2021-04-20T10:59:15+02:00'
  Offset:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/agipd-type/agipd_siv1_agipdv11_m313/cal.1618912632.602872.h5
    h5path: //AGIPD_SIV1_AGIPDV11_M313/Offset/0
    timestamp: '2021-04-20T10:59:15+02:00'
  ThresholdsDark:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/agipd-type/agipd_siv1_agipdv11_m313/cal.1618912666.0721142.h5
    h5path: //AGIPD_SIV1_AGIPDV11_M313/ThresholdsDark/0
    timestamp: '2021-04-20T10:59:15+02:00'
pdu: AGIPD_SIV1_AGIPDV11_M313

For 2, an example module_metadata looks like this:

module: Q3M2
old-constants:
  BadPixelsDark:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312441.4952333.h5
    h5path: //LPD_SIV1_LPDV2_SM006/BadPixelsDark/0
    timestamp: '2020-03-04T09:20:32+01:00'
  Noise:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312434.2626257.h5
    h5path: //LPD_SIV1_LPDV2_SM006/Noise/0
    timestamp: '2020-03-04T09:20:32+01:00'
  Offset:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312426.9552855.h5
    h5path: //LPD_SIV1_LPDV2_SM006/Offset/0
    timestamp: '2020-03-04T09:20:32+01:00'
pdu: LPD_SIV1_LPDV2_SM006

For 3, an example module_metadata file looks like this:

module: Q2M1
old-constants:
  Noise:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/dssc-type/dssc_minisddv1_f2_008/cal.1611527105.1262028.h5
    h5path: //DSSC_MiniSDDV1_F2_008/Noise/0
    timestamp: '2020-11-05T09:40:39+01:00'
  Offset:
    filepath: /gpfs/exfel/d/cal/caldb_store/xfel/cal/dssc-type/dssc_minisddv1_f2_008/cal.1611527101.6753352.h5
    h5path: //DSSC_MiniSDDV1_F2_008/Offset/0
    timestamp: '2020-11-05T09:40:39+01:00'
pdu: DSSC_MiniSDDV1_F2_008

Relevant Documents (optional)

In output_dir, the {AGIPD,LPD,DSSC} DARKS notebook will save module_metadata_{qm}.yml for each QM. This replaces the old module_mapping_{qm}.yml files; they "just" saved the mapping from QM to PDU and these new ones do a superset of that. These may look like this:

module: Q3M2
old-constants:
  BadPixelsDark:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312441.4952333.h5
    h5path: //LPD_SIV1_LPDV2_SM006/BadPixelsDark/0
    timestamp: '2020-03-04T09:20:32+01:00'
  Noise:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312434.2626257.h5
    h5path: //LPD_SIV1_LPDV2_SM006/Noise/0
    timestamp: '2020-03-04T09:20:32+01:00'
  Offset:
    filepath: /gpfs/exfel/d/cal_tst/caldb_store/xfel/cal/lpd-type/lpd_siv1_lpdv2_sm006/cal.1583312426.9552855.h5
    h5path: //LPD_SIV1_LPDV2_SM006/Offset/0
    timestamp: '2020-03-04T09:20:32+01:00'
pdu: LPD_SIV1_LPDV2_SM006

Just like with the old module_mapping_{qm}.yml files, the summary notebook will glob over them, read them into some dict, and unlink them.

Types of changes

  • Refactor

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
    • It might - is this kind of thing in docs?

Reviewers

@calibration

@roscar exciting news: nbqa is struggling on this MR. Not sure why some part of the pipeline expects ASCII. Also not sure which characters upset it. Maybe the "•" that are used in some prints?

Traceback (most recent call last):
  File "/home/gitlab-runner/.cache/pre-commit/repobyi8lc52/py_env-python3/lib/python3.6/site-packages/nbqa/__main__.py", line 615, in _run_on_one_root_dir
    nb_info_mapping[notebook],
  File "/home/gitlab-runner/.cache/pre-commit/repobyi8lc52/py_env-python3/lib/python3.6/site-packages/nbqa/replace_source.py", line 174, in mutate
    pycells = _get_pycells(python_file)
  File "/home/gitlab-runner/.cache/pre-commit/repobyi8lc52/py_env-python3/lib/python3.6/site-packages/nbqa/replace_source.py", line 132, in _get_pycells
    txt = python_file.read_text()
  File "/software/anaconda3/5.2/lib/python3.6/pathlib.py", line 1197, in read_text
    return f.read()
  File "/software/anaconda3/5.2/lib/python3.6/encodings/ascii.py", line 26, in decode
    return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 15932: ordinal not in range(128)
Edited by David Hammer

Merge request reports

Legal Notice | Contacts