feat [Jungfrau][Dark]: Update notebook to write and read ccvs without using calibrationDBRemote

Description

This addresses the first step in this issue, which is removing calibrationDBRemote usage.

Dark notebook.

• Replace send_to_db and storing local h5 constant file with write_ccv and inject_ccv
• Replace get_to_db with CalibrationData API
• Get pdu using calibration_client and have a da_to_pdu dict.
• Refactor code based on the new da_to_pdu dict and remove unused variables.
• Use begin_at_strategy=prior to get old constants
• Bonus: use calcat_creation_time

calcat_interface2

• new get_full_path method and add JungfrauConditions
• Add begin_at_strategy to from_condition as it was introduce in CALCAT: https://git.xfel.eu/ITDM/calibration_catalog/-/merge_requests/164
• Update extra with the begin_at_strategy if no comments related to this part in this MR.

constants

• Refactor inject_ccv to smaller functions: I have reduced the refactor to reduce the diff - d2782da9
• Enable adding deviations for the stored conditions.
• New warning class CCVAlreadyInjected to catch has already been taken 422 errors while injecting CCVs.

plotting

• Update jungfrau's calibration name for Noise10Hz

Jungfrau dark summary notebook

• Fix access to new data structure for local stored CCVs

How Has This Been Tested?

• Running a JF notebook during development and checking stored local constants
• Running a JF notebook during development and checking injected constants in CALCAT and /gpfs/
• Automated tests. - It is expected to fail as file names changes and structure as well.

Relevant Documents (optional)

first test result through xcaltst:

• Failed: failed_4m_test_injection.pdf Some jobs succeeded and some failed with the following errors:

SSLError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
SSLError: None: Max retries exceeded with url: /test_calibration/api/detectors?
˓→identifier=SPB_IRDA_JF4M (Caused by None)

The above exception was the direct cause of the following exception:
ConnectionError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed
˓→connection without response'))

• Second test: second_test.pdf showed that reinjecting same CCV wasn't handeled the same as we expect.

RuntimeError: {'success': False, 'status_code': 422, 'info': 'Error creating
˓→calibration_constant_version', 'app_info': {'calibration_constant_id': ['has
˓→already been taken'], 'physical_detector_unit_id': ['has already been taken'],
˓→'begin_at': ['has already been taken']}, 'pagination': {}, 'data': {}}

• Third test with already taken error

• Forth and final test: final_report_test.pdf

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

Reviewers

notebooks/Jungfrau/Jungfrau_dark_analysis_all_gains_burst_mode_NBC.ipynb

 import matplotlib.pyplot as plt
 import multiprocessing
 import numpy as np
+import pasha as psh
 import yaml
 from IPython.display import Markdown, display
 from extra_data import RunDirectory
 
-import pasha as psh
 
 from XFELDetAna.plotting.heatmap import heatmapPlot
 from XFELDetAna.plotting.histogram import histPlot
 from cal_tools.constants import inject_ccv, write_ccv
 from cal_tools.enums import BadPixels, JungfrauGainMode
 from cal_tools.jungfrau import jungfraulib
-from cal_tools.restful_config import calibration_client
+from cal_tools.restful_config import (
+    calibration_client,
+    extra_calibration_client,
+)
 from cal_tools.tools import calcat_creation_time
 
 warnings.filterwarnings('ignore')
     gain_setting=gain_setting,
     gain_mode=gain_mode,
 )
+# This is useful to setup caldb_root from the loaded config.
+# e.g. switch to test_calcat
+extra_calibration_client()
 
 cc = calibration_client()
 
     for pdu in resp['data']:
         if pdu['karabo_da'] in karabo_da:  # exclude unselected das
             da_to_pdu[pdu['karabo_da']] = pdu['physical_name']
-            pdu_to_uuid[pdu['physical_name']] = pdu['float_uuid']
+            pdu_to_uuid[pdu['physical_name']] = pdu['uuid']
 
 first_pdu = resp['data'][0]
 detector_info = first_pdu['detector']