Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

processors/github_metadata: retry in case of API request timeout #114

Open
nodiscc opened this issue Jul 26, 2023 · 0 comments
Open

processors/github_metadata: retry in case of API request timeout #114

nodiscc opened this issue Jul 26, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@nodiscc
Copy link
Owner

nodiscc commented Jul 26, 2023

Would prevent having to manually re-run the whole job, e.g. https://github.com/awesome-selfhosted/awesome-selfhosted-data/actions/runs/5662427414/job/15342323076

[...]
INFO:github_metadata.py: Gathering metadata for https://github.com/revive-adserver/revive-adserver from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/revoltchat/revolt from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/roadiz/roadiz from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/robustirc/robustirc from Github API
INFO:github_metadata.py: Gathering metadata for https://github.com/RocketChat/Rocket.Chat from Github API
Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    response = conn.getresponse()
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 461, in getresponse
    httplib_response = super().getresponse()
  File "/usr/lib/python3.10/http/client.py", line 1374, in getresponse
    response.begin()
  File "/usr/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.10/socket.py", line 705, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.10/ssl.py", line 1274, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.10/ssl.py", line 1130, in read
    return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 538, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 370, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/bin/hecat", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/main.py", line 39, in main
    add_github_metadata(step)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/processors/github_metadata.py", line 107, in add_github_metadata
    gh_metadata, latest_commit_date = get_gh_metadata(step, github_url, g, errors)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/hecat/processors/github_metadata.py", line 59, in get_gh_metadata
    gh_metadata = g.get_repo(project)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/MainClass.py", line 363, in get_repo
    headers, data = self.__requester.requestJsonAndCheck("GET", url)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 443, in requestJsonAndCheck
    *self.requestJson(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 580, in requestJson
    return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 685, in __requestEncode
    status, responseHeaders, output = self.__requestRaw(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 719, in __requestRaw
    response = cnx.getresponse()
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/github/Requester.py", line 164, in getresponse
    r = verb(
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/home/runner/work/awesome-selfhosted-data/awesome-selfhosted-data/.venv/lib/python3.10/site-packages/requests/adapters.py", line 532, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='api.github.com', port=443): Read timed out. (read timeout=15)
make: *** [Makefile:25: update_metadata] Error 1
Error: Process completed with exit code 2.
@nodiscc nodiscc added the enhancement New feature or request label Jul 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant