[webservice] Intoduce Global Logger for xfel-calibrate

Description

This merge request introduces a comprehensive logging system for errors and warnings in the offline calibration pipeline. The goal is to improve transparency and provide more detailed feedback to users through myMDC.

graph TD
    A[Start] --> B[Initialize Logging]
    B --> D[Execute xfel-calibrate and wait for processing to finish]
    D --> E{Check if ALL jobs COMPLETED?}
    E -->|Yes| F[Read error logs for all job_ids]
    E -->|No| G[Read error logs for completed job_ids]
    F & G --> H{Any error logs non-empty?}
    H -->|Yes| I[Set Specific Error]
    H -->|No| J[Read warning logs for all job_ids]
    J --> L{Any warning logs non-empty?}
    L -->|No| M[Set Success]
    L -->|Yes| N[Set Warning]
    I & M & N --> O[Update myMDC]
    O --> P[End]

    classDef processNode fill:#f9f,stroke:#333,stroke-width:2px;
    classDef decisionNode fill:#bbf,stroke:#333,stroke-width:2px;
    classDef statusNode fill:#bfb,stroke:#333,stroke-width:2px;
    class B,D,F,G,J processNode;
    class E,H,L decisionNode;
    class I,M,N statusNode;

Related Issues

• Follows and is inspired by the feat/errors-and-warnings branch
• Addresses the issue of propagating warnings/errors back to myMDC: https://git.xfel.eu/calibration/planning/-/issues/149

Changes

graph TD
    A[xfel-calibrate CLI] --> B[Copy `setup_logging`.py to current working dir]
    B --> C[Princess runs setup-logging-nb.py before notebook execution]
    C --> D[Set up Error, Warning, and Info Handlers]
    D --> E[Override IPython Exception Handling]
    E --> F[Execute Calibration Notebooks]

1. New Logging Setup Module

• Introduces a new Python module setup_logging.py for configuring the logging system
• Uses Princess to run this setup module before executing any notebooks

2. Log File Structure

• Creates three separate log files:
  • errors.log: Captures all error-level messages
  • warnings.log: Captures all warning-level messages
  • info.log: Captures DEBUG and INFO level messages, including logs from APIs like calibration_client

3. Implementation Details

• Custom ContextFilter to add notebook name and directory to log records
• Custom log formatter with detailed information:

  %(asctime)s - %(levelname)s - %(filename)s:%(lineno)d - [Notebook: %(notebook)s] - [Directory: %(directory)s] - %(message)s

• Environment variables for notebook name (CAL_NOTEBOOK_NAME) and working directory (CAL_WORKING_DIR)
• Separate file handlers for errors, warnings, and info logs
• Custom error handling with safe_handle_error function
• Custom warning handling with handle_warning function
• Override of IPython's exception handling

4. Next step

• Update webservice:
  • Check error/warning log files.
  • Read errors/warnings, if available.
  • Update myMDC states (start using warning with correction not only errors)

Implementation Steps

1. Copy src/xfel_calibrate/setup_logging.py to CAL_WORKING_DIR as setup-logging-nb.py in calibrate.py
2. Use --run-before Princess command to execute the handler before xfel-calibrate CAL action

How Has This Been Tested?

Use the test notebooks by Philipp from feat/errors-and-warnings to test all types of log levels. (no automated tests done here)

• Add a check in the test to make sure warnings and errors are properly logged separately.

Relevant Documents (optional)

warning_error.pdf

Types of changes

Checklist:

Reviewers