Skip to content

Correctly get creation date from files on disk

Cyril Danilevski requested to merge fix/issue_27 into master

Description

The aim of this PR is to close issue #27 , where I described how the path building for getting creation date from disk (as opposed to myMDC) was broken.

How Has This Been Tested?

I've tested this by running the Characterize_AGIPD_Gain_Darks_NBC notebook with the configuration provided in the issue, replicated here (cell number 1 content):

cluster_profile = "noDB" # The ipcluster profile to use
in_folder = "/gpfs/exfel/exp/DETLAB/202031/p900172" # path to input data, required
out_folder = "/gpfs/exfel/data/scratch/danilevc/annotations" # path to output to, required
sequences = [0] # sequence files to evaluate.
modules = [-1]  # list of modules to evaluate, RANGE ALLOWED
run_high = 73 # run number in which high gain data was recorded, required
run_med = 71 # run number in which medium gain data was recorded, required
run_low = 72 # run number in which low gain data was recorded, required

karabo_id = "DETLAB_DET_AGIPD500K2G" # karabo karabo_id
karabo_da = ['-1']  # a list of data aggregators names, Default [-1] for selecting all data aggregators
receiver_id = "{}CH0" # inset for receiver devices
path_template = 'RAW-R{:04d}-{}-S{:05d}.h5' # the template to use to access data
h5path = '/INSTRUMENT/{}/DET/{}:xtdf/image' # path in the HDF5 file to images
h5path_idx = '/INDEX/{}/DET/{}:xtdf/image' # path in the HDF5 file to images
h5path_ctrl = '/CONTROL/{}/MDL/FPGA_COMP' # path to control information
karabo_id_control = "HED_EXP_AGIPD500K2G" # karabo-id for control device '
karabo_da_control = "DA01" # karabo DA for control infromation

use_dir_creation_date = True  # use dir creation date as data production reference date
cal_db_interface = "tcp://max-exfl016:8020" # the database interface to use
cal_db_timeout = 3000000 # timeout on caldb requests"
local_output = True # output constants locally
db_output = False # output constants to database

mem_cells = 0 # number of memory cells used, set to 0 to automatically infer
bias_voltage = 300 # detector bias voltage
gain_setting = 0.1 # the gain setting, use 0.1 to try to auto-determine
acq_rate = 0. # the detector acquisition rate, use 0 to try to auto-determine
interlaced = False # assume interlaced data format, for data prior to Dec. 2017
rawversion = 2 # RAW file format version

thresholds_offset_sigma = 3. # offset sigma thresholds for offset deduced bad pixels
thresholds_offset_hard = [0, 0] # For setting the same threshold offset for the 3 gains. Left for backcompatability. Default [0, 0] to take the following parameters.
thresholds_offset_hard_hg = [3000, 7000] # High-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_offset_hard_mg = [6000, 10000] # Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_offset_hard_lg = [6000, 10000] # Low-gain thresholds in absolute ADU terms for offset deduced bad pixels

thresholds_noise_sigma = 5. # noise sigma thresholds for offset deduced bad pixels
thresholds_noise_hard = [0, 0] # For setting the same threshold noise for the 3 gains. Left for backcompatability. Default [0, 0] to take the following parameters.
thresholds_noise_hard_hg = [4, 20] # High-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_hard_mg = [4, 20] # Medium-gain thresholds in absolute ADU terms for offset deduced bad pixels
thresholds_noise_hard_lg = [4, 20] # Low-gain thresholds in absolute ADU terms for offset deduced bad pixels

thresholds_gain_sigma = 5. # Gain separation sigma threshold

high_res_badpix_3d = False # set this to True if you need high-resolution 3d bad pixel plots. ~7mins extra time for 64 memory cells

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

I've introduced type hinting, which is debatable.
I've also changed the bare except to an IOError catching one.
From my understanding, it's the only error thrown, but I'm not sure.

I have updated the documentation accordingly.

As this is a non-trivial change, and my first MR here, I'd like the feedback from two of @ahmedk @kamile @kluyvert or @tmichela

I'm also keen to create a test, but it requires quite a bit of monkey-patching (ie. time). Moreover, it's mostly buit-ins that are used, which I trust.

Thanks!

Merge request reports