Skip to content
Snippets Groups Projects
Commit 35c2bcfd authored by Karim Ahmed's avatar Karim Ahmed Committed by Karim Ahmed
Browse files

add more troubleshooting

parent b16026b2
No related branches found
No related tags found
1 merge request!989[DOC] Updated documentations while preparing for DET calibration training
......@@ -7,15 +7,20 @@
It has been reported more than once through the instruments and DOC that there offline correction showed didn't produce any data. And after the DOC checked the reports there is warning for no trains available to correct for all modules, therefore no processed data available.
This can happen in case the offline calibration is configured to correct illuminated data based on the LitFrameFinder device. This device points to the calibration pipeline which frames are illuminated to process. In case the data didn't have any no data correction will take place and no processed files will be produced. #TODO: display how to confirm that lit frame correction is on.
This can happen in case the offline calibration is configured to correct illuminated data based on the LitFrameFinder device. This device points to the calibration pipeline which frames are illuminated to process. In case the data didn't have any no data correction will take place and no processed files will be produced.
![No LitFrames](../static/troubleshooting/no_litframes.png)
One of the reason for this can be that there were no beam at the time when the data was acquired. This is something the instrument scientists should be aware of and it is usually possible to check the shutter configuration switches for this particular run. As the shutter switches names can be changed. It wont be included here to avoid giving outdated dataset key names.
Below is a reason a table on the shutter switches state for a run acquired with no beam.
## DSSC
![No beam](../static/troubleshooting/shutter_run_state.png)
## Epix100
<!-- ## DSSC
## Epix100 -->
## Gotthard2
......@@ -57,6 +62,12 @@ h5glance /gpfs/exfel/exp/SPB/202401/p005476/raw/r0020/RAW-R0020-GH200-S00000.h5
## Jungfrau
## LPD
### No constants retrieved for burst mode data
To streamline the operation in burst mode, we have decided to exclusively inject fixed gain constants into the database, regardless of the operational mode. Burst mode operates solely at high gain, and the high gain constant remains consistent between fixed and adaptive gain settings. Consequently, we have opted to only incorporate fixed gain constants. The middlelayer responsible for executing dark runs has been adjusted to accommodate this change.
There have been instances where instruments acquired dark runs manually, posing a risk of mistakenly acquiring these runs in adaptive gain mode. Although not anticipated from an offline calibration perspective, such occurrences will be managed by issuing a warning and employing the adaptive gain constant. If dark runs were not performed, an error will be reported due to either the absence of dark runs or the retrieval of burst mode adaptive dark runs that are more than 3 days old.
<!-- ## LPD
## PnCCD
## PnCCD -->
......@@ -67,6 +67,8 @@ To confirm that the slowness in calibration is related to slow migration, one ca
### Allocated jobs are in pending state
![pending correction jobs](../static/troubleshooting/pending_jobs.png)
There are two partitions used by the offline calibration for Active proposals. In these partitions `xcal` has a high priority, hence if all nodes are occupied by different users, `xcal` would be able to take over the node.
This helps in avoiding the issue of not finding resources during user experiments to run dark processing or corrections. However in some cases it would be possible to get a call about `PENDING` calibration jobs for too long.
......@@ -77,8 +79,23 @@ This helps in avoiding the issue of not finding resources during user experiment
## Correction
#### Correction is taking longer than expected (I/O delay)
In case the correction was properly started from myMDC and related jobs are not pending for a long time but rather are processing for longer than expected i.e. in reference to other runs previously corrected in the same proposal. One reason could be is that the data for this run was moved from fast access gpfs to dCache. This movement is expected for proposals after a time window for finished proposals to leave space for new data and active proposals.
Data in dCache has a longer I/O time and for data with many sequence files, data processing can be affected.
To check if the data is on dCache, myMDC can be used.
![mymdc_repositories](../static/myMDC/repositories.png)
This image show that the runs are on gpfs and dCache, there can be other proposals which have some or all runs only on dCache.
#### Correction failed no constants found
For most of the detectors in case the offset dark constant was not retrieved, the correction will not go through.
## Dark processing
<!-- ## Dark processing -->
......
docs/static/myMDC/repositories.png

69.1 KiB

docs/static/troubleshooting/no_litframes.png

27.1 KiB

docs/static/troubleshooting/pending_jobs.png

211 KiB

docs/static/troubleshooting/shutter_run_state.png

33.7 KiB

0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment