feat [Jungfrau][Dark]: Update notebook to write and read ccvs without using calibrationDBRemote

Description

This addresses the first step in this issue, which is removing calibrationDBRemote usage.

Dark notebook.

• Replace send_to_db and storing local h5 constant file with write_ccv and inject_ccv
• Replace get_to_db with CalibrationData API
• Get pdu using calibration_client and have a da_to_pdu dict.
• Refactor code based on the new da_to_pdu dict and remove unused variables.
• Use begin_at_strategy=prior to get old constants
• Bonus: use calcat_creation_time

calcat_interface2

• new get_full_path method and add JungfrauConditions
• Add begin_at_strategy to from_condition as it was introduce in CALCAT: https://git.xfel.eu/ITDM/calibration_catalog/-/merge_requests/164
• Update extra with the begin_at_strategy if no comments related to this part in this MR.

constants

• Refactor inject_ccv to smaller functions: I have reduced the refactor to reduce the diff - d2782da9
• Enable adding deviations for the stored conditions.
• New warning class CCVAlreadyInjected to catch has already been taken 422 errors while injecting CCVs.

plotting

• Update jungfrau's calibration name for Noise10Hz

Jungfrau dark summary notebook

• Fix access to new data structure for local stored CCVs

How Has This Been Tested?

• Running a JF notebook during development and checking stored local constants
• Running a JF notebook during development and checking injected constants in CALCAT and /gpfs/
• Automated tests. - It is expected to fail as file names changes and structure as well.

Relevant Documents (optional)

first test result through xcaltst:

• Failed: failed_4m_test_injection.pdf Some jobs succeeded and some failed with the following errors:

SSLError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
SSLError: None: Max retries exceeded with url: /test_calibration/api/detectors?
˓→identifier=SPB_IRDA_JF4M (Caused by None)

The above exception was the direct cause of the following exception:
ConnectionError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed
˓→connection without response'))

• Second test: second_test.pdf showed that reinjecting same CCV wasn't handeled the same as we expect.

RuntimeError: {'success': False, 'status_code': 422, 'info': 'Error creating
˓→calibration_constant_version', 'app_info': {'calibration_constant_id': ['has
˓→already been taken'], 'physical_detector_unit_id': ['has already been taken'],
˓→'begin_at': ['has already been taken']}, 'pagination': {}, 'data': {}}

• Third test with already taken error

• Forth and final test: final_report_test.pdf

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

Reviewers

src/cal_tools/calcat_interface2.py

@@ -852,6 +852,46 @@ class DSSCConditions(ConditionsBase):
     }
 
 
+@dataclass
+class JUNGFRAUConditions(ConditionsBase):
+    """Conditions for JUNGFRAU detectors"""
+    sensor_bias_voltage: float
+    memory_cells: int
+    integration_time: float
+    gain_setting: int
+    gain_mode: Optional[int] = None
+    sensor_temperature: float = 291
+    pixels_x: int = 1024
+    pixels_y: int = 512
+
+    _params = [
+        "Sensor Bias Voltage",
+        "Memory Cells",
+        "Pixels X",
+        "Pixels Y",
+        "Integration Time",
+        "Sensor temperature",
+        "Gain Setting",
+        "Gain mode",
+    ]
+    calibration_types = {
+        "Offset10Hz": _params,
+        "Noise10Hz": _params,
+        "BadPixelsDark10Hz": _params,
+        "RelativeGain10Hz": _params,
+        "BadPixelsFF10Hz": _params,
+    }
+
+    def make_dict(self, parameters):
+        cond = super().make_dict(parameters)
+
+        # Fix-up some database quirks.
+        if int(cond.get("Gain mode", -1)) == 0:
+            del cond["Gain mode"]
+
+        return cond
+
+
 @dataclass
 class ShimadzuHPVX2Conditions(ConditionsBase):
     burst_frame_count: float