I am increasing the proxy request timeout from the default 20 seconds to a 2-minute timeout.
This is related to a reported failed correction run for AGIPD after deploying release 3.11.1 for pycalibration, which consists of this change pycalibration!774 (merged)
AGIPD correction now retrieves multiple calibrations for multiple modules in the same query.
When the needed CCVs are part of the calibration constants of many versions, CalCat spends more time finding the desired CCVs. This can take more than 20 seconds.
This is logs from the slurm_out of the failed correction job
WARNING:calparrot.proxy:Timeout waiting for response to appear in database; forwarding request for /api/calibration_constant_versions/get_by_detector_conditions?calibration_id=%5B14%2C+2%2C+1%5D&detector_identifier=MID_DET_AGIPD1M-1&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&karabo_da=&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00ERROR:tornado.application:Uncaught exception GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1)HTTPServerRequest(protocol='http', host='127.0.0.1:35766', method='GET', uri='/api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00', version='HTTP/1.1', remote_ip='127.0.0.1')Traceback (most recent call last): File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute result = await result File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 187, in get response = await self._get_with_cache(req_path) File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 171, in _get_with_cache resp = await self._get_upstream(req_path) File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 122, in _get_upstream response = await self.upstream_client.fetch( File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 83, in fetch return await self.http_client.fetch(url, headers=headers, **kwargs)tornado.simple_httpclient.HTTPTimeoutError: Timeout during requestERROR:tornado.access:500 GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1) 35101.42ms
