Skip to content
Snippets Groups Projects

Check for transfer conditional on resolved symlink path

Merged Robert Rosca requested to merge fix/migration-check-repo-only-offline into master

Description

Closes #59

Implemented the changes as we discussed in the standup today. The location of the run directory is resolved, then the response from MyMdC is checked to see if the transfer has been complete or not.

So, even if the GPFS transfer is done, if the resolved path is on PNFS then it will continue waiting.

Overall:

  1. Added cycle and instrument arguments to wait_on_transfer and wait_transfers
  2. Use those vars with config['correct']['in-folder'].format(...) to get the proposal dir
  3. Resolve the path to proposal_dir / f"{run:06d}"
  4. If that path is relative to /gpfs/exfel/d then then check that GPFS transfer is done, else if it is rel. to /pnfs check that DCACHE transfer is done
  5. Update calls to wait_transfers to pass through the cycle and proposal

How Has This Been Tested?

TBC

Relevant Documents (optional)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code follows the code style of this project.

Reviewers

@calibration

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Robert Rosca added 2 commits

    added 2 commits

    • 32fde376 - Make wait_transfers/wait_on_transfers easier to test
    • 753e5cc1 - Update tests for wait_on_transfers

    Compare with previous version

  • Robert Rosca added 3 commits

    added 3 commits

    • f4a30526 - Specifically check if gpfs offline, add tests
    • e12f22d9 - Rename proposal_dir to in_folder for consistency
    • 7ae3db61 - Remove unnecessary mocking

    Compare with previous version

  • Thomas Kluyver resolved all threads

    resolved all threads

  • Robert Rosca added 1 commit

    added 1 commit

    Compare with previous version

  • Robert Rosca added 1 commit

    added 1 commit

    • c6afee50 - Apply 2 suggestion(s) to 2 file(s)

    Compare with previous version

  • Robert Rosca added 1 commit

    added 1 commit

    Compare with previous version

  • Robert Rosca added 1 commit

    added 1 commit

    • 5d338309 - Symlinks for `raw` dir not run dir

    Compare with previous version

  • Robert Rosca resolved all threads

    resolved all threads

  • Thomas Kluyver
  • Robert Rosca resolved all threads

    resolved all threads

  • Robert Rosca added 1 commit

    added 1 commit

    • 3c835fe3 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • Robert Rosca added 2 commits

    added 2 commits

    • 13f1e680 - Return None if resolved path parents not for
    • 000e5c32 - Add test for invalid run dir, test log output

    Compare with previous version

  • Robert Rosca added 1 commit

    added 1 commit

    • b67d5e2a - Fix `test_wait_on_transfer_exceptions` test

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading