Sleep/retry on 502 Bad Gateway

[yarikoptic]

Fresh cron run on smaug for datalad (Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad.lock /mnt/datasets/datalad/ci/logs/tools/cron_job) had

Traceback (most recent call last):                                                                                                                                                   
  File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous", line 33, in <module>                                                                                                 
    sys.exit(load_entry_point('tinuous', 'console_scripts', 'tinuous')())                                                                                                            
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 829, in __call__                                                                  
    return self.main(*args, **kwargs)                                                                                                                                                
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 782, in main                                                                      
    rv = self.invoke(ctx)                                                                                                                                                            
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1259, in invoke                                                                   
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                          
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 1066, in invoke                                                                   
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                   
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/core.py", line 610, in invoke                                                                    
    return callback(*args, **kwargs)                                                                                                                                                 
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func                                                             
    return f(get_current_context().obj, *args, **kwargs)                                                                                                                             
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/__main__.py", line 126, in fetch                                                                                                
    paths = obj.download(Path(path))                                                                                                                                                 
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/github.py", line 318, in download                                                                                               
    r = self.client.get(self.logs_url)                                                                                                                                               
  File "/mnt/datasets/datalad/ci/tinuous/src/tinuous/base.py", line 100, in get                                                                                                      
    r.raise_for_status()                                                                                                                                                             
  File "/home/datalad/miniconda3/envs/tinuous-dev/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status                                                     
    raise HTTPError(http_error_msg, response=self)                                                                                                                                   
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs      

I would assume that it was some intermittent issue and sleep/retry should be sufficient to address it.

I checked mail -- it was the first and only so far incident of this kind. Although seems to be rare, IMHO still worth accounting for

[jwodder]

@yarikoptic A 502 at that location should have been retried unless the maximum number of retries (10) had already been reached. Is the traceback preceded by a bunch of warning messages of the form "Request to %s returned %d; waiting & retrying"?

[yarikoptic]

Sorry for the noise - I guess I was blind, indeed it did retries for 6 minutes:

2021-08-05T10:20:05-0400 [INFO    ] tinuous Found run 3456
2021-08-05T10:20:06-0400 [INFO    ] tinuous Downloading logs for docbuild.yml (Docs) #3456 to 2021/08/05/pr/5874/4f601ad/github-Docs-3456-success
2021-08-05T10:20:16-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:20:27-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:20:41-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:21:00-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:21:27-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:22:02-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:22:48-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:23:47-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 502; waiting & retrying
2021-08-05T10:25:01-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 500; waiting & retrying
2021-08-05T10:26:33-0400 [WARNING ] tinuous Request to https://api.github.com/repos/datalad/datalad/actions/runs/1101686117/logs returned 500; waiting & retrying
Traceback (most recent call last):
  File "/home/datalad/miniconda3/envs/tinuous-dev/bin/tinuous...

not sure if extending it would be productive... Let's assume it was a fluke

[yarikoptic]

Received similar fail in dandi-api and when I clicked on the url in a phone browser got

{
  "message": "Must have admin rights to Repository.",
  "documentation_url": "https://docs.github.com/rest/reference/actions#download-workflow-run-logs"
}

so may be there is more to this story...

logs etc from the run

2021-08-10T18:10:03-0400 [INFO ] tinuous Downloading logs for cli-integration.yml (Test Integration with dandi-cli) #450 to 2021/08/10/github/pr/470/b923159/cli-integration.yml/450-success
2021-08-10T18:10:13-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:10:24-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:10:38-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:10:57-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:11:23-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:11:58-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:12:45-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:13:44-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 500; waiting & retrying
2021-08-10T18:14:58-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 502; waiting & retrying
2021-08-10T18:16:29-0400 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs returned 500; waiting & retrying
Traceback (most recent call last):
File "/home/dandi/cronlib/tinuous-logs/venv/bin/tinuous", line 8, in
sys.exit(main())
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/main.py", line 126, in fetch
paths = obj.download(Path(path))
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/github.py", line 318, in download
r = self.client.get(self.logs_url)
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 100, in get
r.raise_for_status()
File "/home/dandi/cronlib/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://api.github.com/repos/dandi/dandi-api/actions/runs/1117825132/logs

[jwodder]

@yarikoptic Accessing GitHub Actions logs via the API requires a token with admin permissions for the repository in question, which any standard personal token should have (assuming the user in question is an admin for the repo). The message you saw is really irrelevent to any server errors that may be occurring on GitHub's end.

[yarikoptic]

ok, fluke it is I guess