Skip to content
Snippets Groups Projects

Fix/get dir creation date

Merged Karim Ahmed requested to merge fix/get_dir_creation_date into master
3 unresolved threads

Add transparency to get_dir_creation_date() nested methods and unify returned creation date to UTC

Description

This MR started by updating the returned creation_time from myMDC to UTC and ended with splitting the functions for get_dir_creation_date to different functions and adding transparent prints on which method is used at the end to derive the creation date for a run directory.

  1. Unify creation date to UTC
  2. add a separate function to get creation date from myMDC
  3. add a separate function to get creation date from METADATA/creationDate using Extra-data
  4. add tests for both functions and fix old test after unify datetime timezone.
  5. add a fixture to avoid running tests with myMDC connection. as it seems to be the current case at the moment for the CI.

How Has This Been Tested?

By running pytest tests/test_cal_tools locally.

Relevant Documents (optional)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • Refactor (refactoring code with no functionality changes)
  • Test (additional or refactored tests)

Checklist:

  • My code follows the code style of this project.
  • I added tests where appropriate.

Reviewers

@danilevc @roscar @schmidtp

Edited by Karim Ahmed

Merge request reports

Checking pipeline status.

Merged by Karim AhmedKarim Ahmed 3 years ago (Apr 4, 2022 1:52pm UTC)

Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Karim Ahmed
  • 338 raise FileNotFoundError(
    339 "- Failed to read creation time, wrong input folder",
    340 directory) from e
    295 341
    296 342 try:
    297 run_info = get_run_info(proposal, run)
    298 return dateutil.parser.parse(run_info['begin_at'])
    343 return creation_date_metadata_client(proposal, run)
    299 344 except Exception as e:
    300 345 if verbosity > 0:
    301 346 print(e)
    302 347
    303 directory = directory / 'r{:04d}'.format(run)
    304
    305 # Loop a number of times to catch stale file handle errors, due to
    306 # migration or gpfs sync.
  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Karim Ahmed
  • 88 @pytest.mark.requires_mdc
    89 def test_creation_date_metadata_client():
    90
    91 date = creation_date_metadata_client(900113, 9983)
    92 assert isinstance(date, datetime)
    93 assert str(date) == '2020-09-23 13:30:00+00:00'
    94
    95
    96 @pytest.mark.requires_gpfs
    97 def test_creation_date_file_metadata():
    98
    99 folder = '/gpfs/exfel/exp/CALLAB/202031/p900113/raw/r9983'
    100
    101 date = creation_date_file_metadata(RunDirectory(folder))
    72 102 assert isinstance(date, datetime)
    73 assert str(date) == '2019-12-16 08:52:25.196603'
  • Karim Ahmed added 14 commits

    added 14 commits

    Compare with previous version

  • Thomas Kluyver
  • Philipp Schmidt changed milestone to %3.5.0

    changed milestone to %3.5.0

  • removed milestone %3.5.0

  • Philipp Schmidt changed milestone to %3.5.1

    changed milestone to %3.5.1

  • Thank you, @ahmedk and @kluyvert, for double-checking.

    It looks like in our tests @varun was unlucky to get the same second on both (we ignoring milliseconds precision).

    I will let @Djelloul and @Varun know about this discrepancy because DAQ and myMdC should be having the same values.

  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • 4b5e28b0 - avoid duplication in test_cal_tools

    Compare with previous version

  • Cyril Danilevski
  • Karim Ahmed added 1 commit

    added 1 commit

    • 1ea00440 - avoid duplication in test_cal_tools

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • d7fb02a1 - add a comment on parsing file not found error.

    Compare with previous version

  • Thank you for the review!

  • Sweet, thanks! LGTM!

  • Karim Ahmed added 59 commits

    added 59 commits

    • d7fb02a1...94444be8 - 54 commits from branch master
    • 8ef805e1 - first draft for fixing get_dir_creation_date
    • 5f4dc8ec - split functions used by get_dir_creation_date and add tests for them
    • 99df6d18 - flake8
    • c6b4a499 - avoid duplication in test_cal_tools
    • bf21d241 - add a comment on parsing file not found error.

    Compare with previous version

  • merged

  • Karim Ahmed mentioned in commit ea1b0f06

    mentioned in commit ea1b0f06

  • Please register or sign in to reply
    Loading