Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

more (20? more exponentially grow) retries on 500s #147

Closed
yarikoptic opened this issue May 11, 2022 · 0 comments · Fixed by #148
Closed

more (20? more exponentially grow) retries on 500s #147

yarikoptic opened this issue May 11, 2022 · 0 comments · Fixed by #148
Assignees

Comments

@yarikoptic
Copy link
Member

was getting logs for a new one -- dandi-archive , eventually crashed with

2022-05-11T18:29:52-0400 [INFO    ] tinuous Downloading logs for cli-integration.yml (Test Integration with dandi-cli) #1050 to 2022/02/10/github/push/master/081e828/Test Integration with dandi-cli/1050
2022-05-11T18:29:53-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:29:54-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:29:59-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:30:09-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:30:25-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:30:51-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:31:28-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:32:18-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:33:23-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
2022-05-11T18:34:44-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs returned 500; waiting & retrying
Traceback (most recent call last):
  File "/mnt/backup/dandi/tinuous-logs/venv/bin/tinuous", line 10, in <module>
    sys.exit(main())
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/__main__.py", line 127, in fetch
    paths = obj.download(Path(path))
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/github.py", line 321, in download
    self.client.download_zipfile(self.logs_url, path)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 143, in download_zipfile
    self.download(path, zippath)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 115, in download
    r = self.get(path, stream=True)
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 107, in get
    r.raise_for_status()
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/1824934158/logs

so it was just about 5 minutes of retries, and wait times between retries seems to not grow as much as it could.

IMHO should try for longer and with steeper growing sleeps

yarikoptic added a commit that referenced this issue May 13, 2022
Adjust retry parameters for non-Pygithub requests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants