feat [Jungfrau][Dark]: Update notebook to write and read ccvs without using calibrationDBRemote
Description
This addresses the first step in this issue, which is removing calibrationDBRemote usage.
Dark notebook.
- Replace
send_to_db
and storing local h5 constant file withwrite_ccv
andinject_ccv
- Replace
get_to_db
with CalibrationData API - Get pdu using calibration_client and have a
da_to_pdu
dict. - Refactor code based on the new
da_to_pdu
dict and remove unused variables. - Use
begin_at_strategy=prior
to get old constants - Bonus: use
calcat_creation_time
calcat_interface2
- new
get_full_path
method and addJungfrauConditions
- Add
begin_at_strategy
tofrom_condition
as it was introduce in CALCAT: https://git.xfel.eu/ITDM/calibration_catalog/-/merge_requests/164 -
Update extra with the begin_at_strategy
if no comments related to this part in this MR.
constants
- Refactor
inject_ccv
to smaller functions: I have reduced the refactor to reduce the diff - d2782da9 - Enable adding deviations for the stored conditions.
- New warning class
CCVAlreadyInjected
to catchhas already been taken
422 errors while injecting CCVs.
plotting
- Update jungfrau's calibration name for Noise10Hz
Jungfrau dark summary notebook
- Fix access to new data structure for local stored CCVs
How Has This Been Tested?
-
Running a JF notebook during development and checking stored local constants -
Running a JF notebook during development and checking injected constants in CALCAT
and/gpfs/
-
Automated tests. - It is expected to fail as file names changes and structure as well.
Relevant Documents (optional)
first test result through xcaltst:
- Failed: failed_4m_test_injection.pdf Some jobs succeeded and some failed with the following errors:
SSLError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
SSLError: None: Max retries exceeded with url: /test_calibration/api/detectors?
˓→identifier=SPB_IRDA_JF4M (Caused by None)
The above exception was the direct cause of the following exception:
ConnectionError Traceback (most recent call last)
<ipython-input-16-a88110cdc6c3> in <module>
5
6 for (mod, const), (data, timestamp, filepath, h5path) in zip(
----> 7 mod_x_const, old_retrieval_res.get()):
8 old_const.setdefault(mod, {})[const] = data
9 old_mdata.setdefault(mod, {})[const] = {
~/.pyenv/versions/3.8.11/lib/python3.8/multiprocessing/pool.py in get(self, timeout)
769 return self._value
770 else:
--> 771 raise self._value
772
773 def _set(self, i, obj):
ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed
˓→connection without response'))
- Second test: second_test.pdf showed that reinjecting same CCV wasn't handeled the same as we expect.
RuntimeError: {'success': False, 'status_code': 422, 'info': 'Error creating
˓→calibration_constant_version', 'app_info': {'calibration_constant_id': ['has
˓→already been taken'], 'physical_detector_unit_id': ['has already been taken'],
˓→'begin_at': ['has already been taken']}, 'pagination': {}, 'data': {}}
-
Third test with already taken error
-
Forth and final test: final_report_test.pdf
Types of changes
- New feature (non-breaking change which adds functionality)
Checklist:
Reviewers
Edited by Karim Ahmed