voussoir/hnarchive
1
0
Fork 0
Code Issues Activity

Add 2s timeout to get requests.

Browse source 
The hn api usually returns in less than 0.25 seconds, but occasionally
I'm getting huge latency spikes. A quick timeout and retry solves that
for me.

If you are using this tool with a very high baseline ping, 2s may be
too aggressive.
This commit is contained in:
voussoir 2020-10-27 22:28:42 -07:00
parent e0862f9e44
commit 8e5f50323d
1 changed files with 3 additions and 2 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
hnarchive.py
View file

@ -69,7 +69,7 @@ def get(url, retries=1):
while retries > 0: while retries > 0:
log.loud(url) log.loud(url)
try: try:
response = requests.get(url, headers=HEADERS) response = requests.get(url, headers=HEADERS, timeout=2)
response.raise_for_status() response.raise_for_status()
break break
except requests.exceptions.HTTPError as exc: except requests.exceptions.HTTPError as exc:
@ -80,7 +80,8 @@ def get(url, retries=1):
retries -= 1 retries -= 1
log.loud('Request failed, %d tries remain.', retries) log.loud('Request failed, %d tries remain.', retries)
time.sleep(bo.next()) time.sleep(bo.next())
except requests.exceptions.ConnectionError: except (requests.exceptions.ConnectionError, requests.exceptions.ReadTimeout):
log.loud('Request failed, %d tries remain.', retries)
time.sleep(bo.next()) time.sleep(bo.next())
end_time = time.time() end_time = time.time()