Skip to content
Snippets Groups Projects

[Webservice] Use leading zeros in usr/Reports folders

Merged [Webservice] Use leading zeros in usr/Reports folders
feat/leading_zeros_for_reports_runs into master
1 unresolved thread
Merged Karim Ahmed requested to merge feat/leading_zeros_for_reports_runs into master
1 unresolved thread

Description

Adding leading zeros for runs in the correction reports file path for usr/Reports as mentioned in this issue: https://git.xfel.eu/calibration/pycalibration/-/issues/75

How Has This Been Tested?

Relevant Documents (optional)

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

  • configuration (/home/xcal/.config/pycalibration/webservice)for production needs to be updated during the next deployment.

Reviewers

@schmidtp @kluyvert

Edited by Karim Ahmed

Merge request reports

Checking pipeline status.

Merged by Karim AhmedKarim Ahmed 1 year ago (Jan 18, 2024 10:43am UTC)

Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Karim Ahmed added 1 commit

    added 1 commit

    • f2b9b8cb - Update the correct place for adding runs for correct and dark

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • 50d79972 - Update the correct place for adding runs for correct and dark

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • 04b75102 - Update webservice.yaml to have reports_folder for both actions and use it in webservice.py

    Compare with previous version

  • Karim Ahmed changed the description

    changed the description

  • Thomas Kluyver
  • Karim Ahmed added 1 commit

    added 1 commit

    • d6a6db85 - assign the reports_folder separately

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Thomas Kluyver
  • Karim Ahmed added 1 commit

    added 1 commit

    Compare with previous version

  • Thomas Kluyver
    Thomas Kluyver @kluyvert ·
    Owner

    Thanks Karim, LGTM

  • Karim Ahmed changed milestone to %3.11.1

    changed milestone to %3.11.1

  • Karim Ahmed added 1 commit

    added 1 commit

    • 5f8de0be - add new script for renaming report folders

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • 0f84e474 - enable renaming all reports folders for all proposals

    Compare with previous version

  • Philipp Schmidt
    Philipp Schmidt @schmidtp ·
    Owner

    This is a fairly intrusive change. I haven't made up my mind about it completely, but I'm not sure I would like to do it during a run or at least by itself when there's a bit more room.

  • Philipp Schmidt removed milestone %3.11.1

    removed milestone %3.11.1

  • Karim Ahmed resolved all threads

    resolved all threads

  • Philipp Schmidt
    Philipp Schmidt @schmidtp ·
    Owner

    Related to !943 (merged)

  • Philipp Schmidt changed milestone to %3.12.0

    changed milestone to %3.12.0

  • Thomas Kluyver mentioned in merge request !943 (merged)

    mentioned in merge request !943 (merged)

  • Karim Ahmed changed milestone to %3.12.1

    changed milestone to %3.12.1

  • Philipp Schmidt
    Philipp Schmidt @schmidtp started a thread on the diff
    • webservice/useful_scripts/rename_report_folders.py 0 → 100644
    1 import argparse
    2 import os
    3 from pathlib import Path
    4
    5 # Update runs into 4 digits integers e.g. r2 to r0002.
    6
    7
    8 def rename_report_folders_for_instrument(directory_path, really):
    9 for folder_name in directory_path.glob("*/*/usr/Reports/*"):
    10 if folder_name.name.startswith('r') and folder_name.name[1:].isdigit():
    11 new_folder_name = 'r' + folder_name.name[1:].zfill(4)
    12 new_path = folder_name.parent.joinpath(new_folder_name)
    13 if really:
    14 folder_name.rename(new_path)
    • Philipp Schmidt
      Philipp Schmidt @schmidtp ·
      Owner

      What happens when this renames to an existing folder?

    • Thomas Kluyver
      Thomas Kluyver @kluyvert ·
      Owner

      You'd get an error in that case, except if the destination is an empty folder.

      OSError: [Errno 39] Directory not empty: 'foo' -> 'bar'
    • Philipp Schmidt
      Philipp Schmidt @schmidtp ·
      Owner

      It seems to be shutil.move should do the trick, rename if not existing and merge if existing.

    • Thomas Kluyver
      Thomas Kluyver @kluyvert ·
      Owner

      I think if you have an existing directory, it would put the old directory inside it, like r0041/r41.

      I don't think there's any standard function that will do what we want in a single call - if the destination directory exists, I think we'd have to iterate over the contents of the source directory and move them one by one.

    • Philipp Schmidt
      Philipp Schmidt @schmidtp ·
      Owner

      Yes, I think you're right. I actually found shutil.copytree first and incorrectly assumed shutil.move works similar. So to be more precise: shutil.copytree combined with deleting the old folder should do it, but at the cost of not actually renaming.

    • Karim Ahmed
      Karim Ahmed @ahmedk ·
      Author Owner

      Checking the report directories before I ran the script I didn't find the case when there are r0001 and r1 directories in any /usr/Reports/. So I didn't apply a change for this for now.

    • Please register or sign in to reply
  • Karim Ahmed added 331 commits

    added 331 commits

    • 0f84e474...f13b7b5d - 323 commits from branch master
    • a1119a84 - Use leading zeros in usr/Reports folders
    • 692355ac - Update the correct place for adding runs for correct and dark
    • e363fc48 - Update webservice.yaml to have reports_folder for both actions and use it in webservice.py
    • ddae966a - assign the reports_folder separately
    • 10d626cd - assign runs separatly
    • 711a864d - spelling mistake
    • d4cd8ba1 - add new script for renaming report folders
    • cb4fd7f5 - enable renaming all reports folders for all proposals

    Compare with previous version

    Toggle commit list
  • Karim Ahmed added 1 commit

    added 1 commit

    • 0e959ab5 - Add path arg, warn for existing Rep folder, and an extra check

    Compare with previous version

  • Karim Ahmed added 1 commit

    added 1 commit

    • a35f27a5 - Add logging, fix old run number length check and move to cycle year instead of 1 cycle

    Compare with previous version

  • Karim Ahmed
    Karim Ahmed @ahmedk ·
    Author Owner

    The script was used yesterday to update all usr/Reports run folders for all instruments for all cycle years until 2020.

    Edited by Karim Ahmed
  • Karim Ahmed merged

    merged

  • Karim Ahmed mentioned in commit 65b886fa

    mentioned in commit 65b886fa

    • Please register or sign in to reply

    Legal Notice | Contacts