[webservice] feat: Processing calibration logs and displaying errors/warnings on myMDC

Description

This MR addresses this issue and is the next step after storing CAL warnings/errors to log in JSON format

Here process_request_finished is updated. CAL logs are processed after each execution is finished and we check for warnings and in cased of failed execution we check for errors. The log message sent to myMDC is formated in a way to show the messaged for each karabo_id and the number of jobs failed for each message.

WIP: The shown screenshots are for errors. I will add some user warnings for testing as we dont have yet any to show and test how the messages are displayed in case we have multiple messages for the same jobID

%%{init: {'theme': 'base', 'themeVariables': { 'fontSize': '10px', 'fontFamily': 'arial','nodeSpacing': 40, 'rankSpacing': 40}}}%%
graph TD
    GetExec["For each execution:<br/>Get karabo_id & status"] --> CheckExec{Execution<br/>Successful?}
    
    CheckExec -->|Yes| WarnOnly["Check warnings only<br/>warnings[karabo_id][msg] += job_id"]
    
    CheckExec -->|No| StatusCheck{"Check job status<br/>COMPLETED/FAILED/OTHER"}
    
    StatusCheck -->|FAILED| ErrorLogs["Check error logs<br/>errors[karabo_id][msg] += job_id<br/><i>or 'warning message for no error log found'</i>"]
    StatusCheck -->|OTHER| AddStatus["Add status message<br/>errors[karabo_id]['SLURM status'] += job_id"]
    
    ErrorLogs & AddStatus --> CheckWarns["Check warning logs<br/>warnings[karabo_id][msg] += job_id"]
    
    WarnOnly & CheckWarns --> Format["Format messages:<br/>{karabo_id: {<br/>  message: '16 jobs (id1-id16)',<br/>  status: '1 job (id17)'<br/>}}"]
    
    Format --> SendMDC["Send calibration feedback<br/>to myMDC"]

    classDef process fill:#dfd,stroke:#333,stroke-width:2px;
    classDef decision fill:#ffd,stroke:#333,stroke-width:2px;
    classDef data fill:#ddf,stroke:#333,stroke-width:2px;
    
    class GetExec,WarnOnly,ErrorLogs,AddStatus,CheckWarns process;
    class CheckExec,StatusCheck decision;
    class Format,SendMDC data; 

How Has This Been Tested?

Tested on max-exfl-cal002 and results shown in the screenshots.

Relevant Documents (optional)

Correct

https://internal.xfel.eu/test_metadata/proposals/259/runs/63606

Dark

https://internal.xfel.eu/test_metadata/proposals/259#proposal-calibration

Types of changes

Checklist:

Reviewers

@kluyvert @schmidtp @roscar

assigned to @ahmedk

marked this merge request as draft

changed the description

changed title from Draft: feat: First draft for processing calibration logs and adding better... to Draft: feat: Processing calibration logs and displaying errors/warnings on myMDC

added 2 commits

• 35d13fd9 - 1 commit from branch feat/global_xfel-calibrate_logger
• 2c14e369 - feat: First draft for processing calibration logs and adding better...

Compare with previous version

Some thoughts from the dev-team meeting:

• Use CalibrationError(Exception) class to indicate checked exceptions that are reported verbatim in myMdC
• All other exceptions are considered unchecked and quoted indirectly with the recommendation to contact an expert
• Use CalibrationWarning(UserWarning) class to indicate warnings that are logged and reported in myMdC
• Any other warnings are considered minor and only printed to the PDF
• The limit on the text field in myMdC may be 2048 characters
• We should explore whether we can use HTML, markdown or some other formatting in that field
• Messages in myMdC should be frugal
  • No job IDs or other technical SLURM details, at most number of jobs that failed
  • If there is an error, ignore any warnings for the same detector/job

added 1 commit

• 40dc074e - Add exceptions file and change warning name

Compare with previous version

added 5 commits

• 40dc074e...349d27a3 - 3 commits from branch feat/global_xfel-calibrate_logger
• 6de29a7e - feat: First draft for processing calibration logs and adding better...
• f4555bd5 - Improve log display

Compare with previous version

changed milestone to %3.16.0

changed the description