Increase proxy request timeout

I am increasing the proxy request timeout from the default 20 seconds to a 2-minute timeout.

This is related to a reported failed correction run for AGIPD after deploying release 3.11.1 for pycalibration, which consists of this change pycalibration!774 (merged)

AGIPD correction now retrieves multiple calibrations for multiple modules in the same query. When the needed CCVs are part of the calibration constants of many versions, CalCat spends more time finding the desired CCVs. This can take more than 20 seconds.

This is logs from the slurm_out of the failed correction job

WARNING:calparrot.proxy:Timeout waiting for response to appear in database; forwarding request for /api/calibration_constant_versions/get_by_detector_conditions?calibration_id=%5B14%2C+2%2C+1%5D&detector_identifier=MID_DET_AGIPD1M-1&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&karabo_da=&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00
ERROR:tornado.application:Uncaught exception GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1)
HTTPServerRequest(protocol='http', host='127.0.0.1:35766', method='GET', uri='/api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 187, in get
    response = await self._get_with_cache(req_path)
  File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 171, in _get_with_cache
    resp = await self._get_upstream(req_path)
  File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 122, in _get_upstream
    response = await self.upstream_client.fetch(
  File "/home/ahmedk/calibration3/.cal3_venv/lib/python3.8/site-packages/calparrot/proxy.py", line 83, in fetch
    return await self.http_client.fetch(url, headers=headers, **kwargs)
tornado.simple_httpclient.HTTPTimeoutError: Timeout during request
ERROR:tornado.access:500 GET /api/calibration_constant_versions/get_by_detector_conditions?detector_identifier=MID_DET_AGIPD1M-1&calibration_id=%5B14%2C+2%2C+1%5D&karabo_da=&event_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00&pdu_snapshot_at=2022-11-09T09%3A20%3A16.279450%2B00%3A00 (127.0.0.1) 35101.42ms

As Tornado httpClient has a 20-second default timeout, I have increased this timeout after being inspired by this link: https://stackoverflow.com/questions/50971042/tornado-httpclient-httperror-http-599-timeout-during-request

Thank you @manettim for debugging this issue. Please feel free to add any more details

@kluyvert

Note: this MR has been tested locally. And there is another MR which changes the same timeout for exfelcalproxy

calparrot/proxy.py

@@ -50,7 +50,7 @@ class XFELOauthClient:
         self.scope = scope
         self.token_url = token_url
 
-        self.http_client = AsyncHTTPClient()
+        self.http_client = AsyncHTTPClient(defaults=dict(request_timeout=120))
         self.oauth_client = BackendApplicationClient(client_id, scope=scope)
 
     async def get_token(self):