Increase proxy request timeout
I am increasing the proxy request timeout from the default 20 seconds to a 2-minute timeout.
This is related to a reported failed correction run for AGIPD after deploying release 3.11.1 for pycalibration, which consists of this change pycalibration!774 (merged)
AGIPD correction now retrieves multiple calibrations for multiple modules in the same query. When the needed CCVs are part of the calibration constants of many versions, CalCat spends more time finding the desired CCVs. This can take more than 20 seconds.
This is logs from the slurm_out of the failed correction job
WARNING:calparrot.proxy:Timeout waiting for response to appear in database; forwarding request for /api/calibration_constant_versions/get_by_detector_conditions?calibration_id=%5B14%2C+2%2C+1%5D&detector_identifier=MID_DET_AGIPD1M-1&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&karabo_da=&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00
ERROR:tornado.application:Uncaught exception GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35766', method='GET', uri='/api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 187, in get
response = await self._get_with_cache(req_path)
File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 171, in _get_with_cache
resp = await self._get_upstream(req_path)
File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 122, in _get_upstream
response = await self.upstream_client.fetch(
File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 83, in fetch
return await self.http_client.fetch(url, headers=headers, **kwargs)
tornado.simple_httpclient.HTTPTimeoutError: Timeout during request
ERROR:tornado.access:500 GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1) 35101.42ms
As Tornado httpClient has a 20-second default timeout, I have increased this timeout after being inspired by this link: https://stackoverflow.com/questions/50971042/tornado-httpclient-httperror-http-599-timeout-during-request
Thank you @manettim for debugging this issue. Please feel free to add any more details
Note: this MR has been tested locally. And there is another MR which changes the same timeout for exfelcalproxy