feat [Jungfrau][Dark]: Update notebook to write and read ccvs without using calibrationDBRemote

Description

This addresses the first step in this issue, which is removing calibrationDBRemote usage.

Dark notebook.

• Replace send_to_db and storing local h5 constant file with write_ccv and inject_ccv
• Replace get_to_db with CalibrationData API
• Get pdu using calibration_client and have a da_to_pdu dict.
• Refactor code based on the new da_to_pdu dict and remove unused variables.
• Use begin_at_strategy=prior to get old constants
• Bonus: use calcat_creation_time

calcat_interface2

• new get_full_path method and add JungfrauConditions
• Add begin_at_strategy to from_condition as it was introduce in CALCAT: https://git.xfel.eu/ITDM/calibration_catalog/-/merge_requests/164
• Update extra with the begin_at_strategy if no comments related to this part in this MR.

constants

• Refactor inject_ccv to smaller functions: I have reduced the refactor to reduce the diff - d2782da9
• Enable adding deviations for the stored conditions.
• New warning class CCVAlreadyInjected to catch has already been taken 422 errors while injecting CCVs.

plotting

• Update jungfrau's calibration name for Noise10Hz

Jungfrau dark summary notebook

• Fix access to new data structure for local stored CCVs

How Has This Been Tested?

• Running a JF notebook during development and checking stored local constants
• Running a JF notebook during development and checking injected constants in CALCAT and /gpfs/
• Automated tests. - It is expected to fail as file names changes and structure as well.

Relevant Documents (optional)

first test result through xcaltst:

• Failed: failed_4m_test_injection.pdf Some jobs succeeded and some failed with the following errors:

SSLError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
SSLError: None: Max retries exceeded with url: /test_calibration/api/detectors?
˓→identifier=SPB_IRDA_JF4M (Caused by None)

The above exception was the direct cause of the following exception:
ConnectionError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed
˓→connection without response'))

• Second test: second_test.pdf showed that reinjecting same CCV wasn't handeled the same as we expect.

RuntimeError: {'success': False, 'status_code': 422, 'info': 'Error creating
˓→calibration_constant_version', 'app_info': {'calibration_constant_id': ['has
˓→already been taken'], 'physical_detector_unit_id': ['has already been taken'],
˓→'begin_at': ['has already been taken']}, 'pagination': {}, 'data': {}}

• Third test with already taken error

• Forth and final test: final_report_test.pdf

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

Reviewers

src/cal_tools/calcat_interface2.py

@@ -258,13 +258,16 @@ class SingleConstant:
             _have_calcat_metadata=True,
         )
 
-    def dataset_obj(self, caldb_root=None) -> h5py.Dataset:
+    def get_full_path(self, caldb_root=None):
         if caldb_root is not None:
             caldb_root = Path(caldb_root)
         else:
             caldb_root = _get_default_caldb_root()
+        return caldb_root / self.path
+
+    def dataset_obj(self, caldb_root=None) -> h5py.Dataset:
 
-        f = h5py.File(caldb_root / self.path, "r")
+        f = h5py.File(self.get_full_path(caldb_root), "r")
         return f[self.dataset]["data"]
 
     def ndarray(self, caldb_root=None):