Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github started to 500 on runs/.../logs on many runs #159

Closed
yarikoptic opened this issue Jan 31, 2023 · 4 comments
Closed

github started to 500 on runs/.../logs on many runs #159

yarikoptic opened this issue Jan 31, 2023 · 4 comments
Assignees
@jwodder

Comments

@yarikoptic
Copy link
Member

Started to happen Jan 30th with

2023-01-30T16:30:01-0500 [INFO    ] tinuous tinuous 0.6.0                                                                                                                                                                                                                                                     
2023-01-30T16:30:01-0500 [INFO    ] tinuous Fetching resources from github                                                                                                                                                                                                                                    
2023-01-30T16:30:01-0500 [INFO    ] tinuous Fetching runs newer than 2023-01-30 19:48:09+00:00                                                                                                                                                                                                                
2023-01-30T16:30:02-0500 [INFO    ] tinuous Fetching runs for workflow .github/workflows/cli-integration.yml (Test Integration with dandi-cli)                                                                                                                                                                
2023-01-30T16:30:03-0500 [INFO    ] tinuous Run 2044 not completed; skipping                                                                                                                                                                                                                                  
2023-01-30T16:30:03-0500 [INFO    ] tinuous Found run 2043                                                                                                                                                                                                                                                    
2023-01-30T16:30:03-0500 [INFO    ] tinuous Downloading logs for cli-integration.yml (Test Integration with dandi-cli) #2043 to 2023/01/30/github/pr/1457/c4c77ea/Test Integration with dandi-cli/2043                                                                                                        
2023-01-30T16:30:04-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:30:05-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:30:09-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:30:14-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:30:25-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:30:46-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:31:27-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:32:48-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:35:28-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:40:49-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T16:51:30-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
2023-01-30T17:12:51-0500 [WARNING ] tinuous Request to https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs returned 500; waiting & retrying                                                                                                                                         
Traceback (most recent call last):                                                                                                                                                                                                                                                                            
  File "/mnt/backup/dandi/tinuous-logs/venv/bin/tinuous", line 10, in <module>                                                                                                                                                                                                                                
    sys.exit(main())                                                                                                                                                                                                                                                                                          
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1128, in __call__                                                                                                                                                                                                
    return self.main(*args, **kwargs)                                                                                                                                                                                                                                                                         
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1053, in main                                                                                                                                                                                                    
    rv = self.invoke(ctx)                                                                                                                                                                                                                                                                                     
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1659, in invoke                                                                                                                                                                                                  
    return _process_result(sub_ctx.command.invoke(sub_ctx))                                                                                                                                                                                                                                                   
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 1395, in invoke                                                                                                                                                                                                  
    return ctx.invoke(self.callback, **ctx.params)                                                                                                                                                                                                                                                            
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/core.py", line 754, in invoke                                                                                                                                                                                                   
    return __callback(*args, **kwargs)                                                                                                                                                                                                                                                                        
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/click/decorators.py", line 38, in new_func                                                                                                                                                                                            
    return f(get_current_context().obj, *args, **kwargs)                                                                                                                                                                                                                                                      
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/__main__.py", line 127, in fetch                                                                                                                                                                                              
    paths = obj.download(Path(path))                                                                                                                                                                                                                                                                          
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/github.py", line 356, in download                                                                                                                                                                                             
    self.client.download_zipfile(self.logs_url, path)                                                                                                                                                                                                                                                         
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 143, in download_zipfile                                                                                                                                                                                       
    self.download(path, zippath)                                                                                                                                                                                                                                                                              
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 115, in download                                                                                                                                                                                               
    r = self.get(path, stream=True)                                                                                                                                                                                                                                                                           
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/tinuous/base.py", line 107, in get                                                                                                                                                                                                    
    r.raise_for_status()                                                                                                                                                                                                                                                                                      
  File "/mnt/backup/dandi/tinuous-logs/venv/lib/python3.8/site-packages/requests/models.py", line 960, in raise_for_status                                                                                                                                                                                    
    raise HTTPError(http_error_msg, response=self)                                                                                                                                                                                                                                                            
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs

and since then:

 106 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex.lock /mnt/datasets/datalad/ci/git-annex/tools/cron_job                                                                                                                                       
 109 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job                                                                                                                     
 110 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job                                                                                                         
 111 N   Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandischema && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandischema.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                   
 112 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad.lock /mnt/datasets/datalad/ci/logs/tools/cron_job                                                                                                                                              
 113 N   Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 114 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-bids-specification.lock /mnt/datasets/datalad/ci/bids-specification/tools/cron_job                                                                                                                     
 116 N   Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                               
 117 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-remote-rclone.lock /mnt/datasets/datalad/ci/git-annex-remote-rclone/tools/cron_job                                                                                                           
 130 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex.lock /mnt/datasets/datalad/ci/git-annex/tools/cron_job                                                                                                                                       
 132 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job                                                                                                                     
 133 N   Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job                                                                                                         
 134 N   Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandischema && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandischema.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                   
 136 N   Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 150     Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                               
 152     Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex.lock /mnt/datasets/datalad/ci/git-annex/tools/cron_job                                                                                                                                       
 157     Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job                                                                                                                     
 158     Jan 31 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-git-annex-ci-client-jobs.lock /mnt/datasets/datalad/ci/git-annex-ci-client-jobs/tools/cron_job                                                                                                         
 159     Jan 31 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 162     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                               
 174     Jan 30 Cron Daemon      Cron <datalad@smaug> chronic flock -n -E 0 /home/datalad/.run/tinuous-datalad-extensions.lock /mnt/datasets/datalad/ci/datalad-extensions/tools/cron_job                                                                                                                     
 176     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 177     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                               
 186     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 190     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                               
 200     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-api && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-api.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch                                                                                       
 214     Jan 30 Cron Daemon      Cron <dandi@drogon> cd /mnt/backup/dandi/tinuous-logs/dandi-archive && chronic flock -n -E 0 /home/dandi/.run/tinuous-dandi-archive.lock /mnt/backup/dandi/tinuous-logs/venv/bin/tinuous fetch

Most likely it is just something to report to github and make a note here hoping for a quick resolution, but I can be wrong.
@jwodder
Copy link
Member

jwodder commented Jan 31, 2023

@yarikoptic Is that the only URL that's failing in this way?

@yarikoptic
Copy link
Member Author

it is that single end point for different projects/runs as far as I see:

❯ find Maildir/.Inbox/ -mtime -64 -type f | xargs grep -h '500 Server Error: Internal Server Error for url'
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048205158/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4048885974/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050090426/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050627658/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050197949/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex/actions/runs/4050294149/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-schema/actions/runs/4051266467/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050911516/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050197949/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex/actions/runs/4050192222/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/git-annex-remote-rclone/git-annex-remote-rclone/actions/runs/4051314553/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/bids-standard/bids-specification/actions/runs/4044720448/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad/actions/runs/4046627398/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-schema/actions/runs/4051239387/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050911516/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050197949/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex/actions/runs/4050192222/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-cli/actions/runs/4051244448/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-schema/actions/runs/4051239387/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050911516/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050197949/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex/actions/runs/4050192222/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex/actions/runs/4050192222/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-schema/actions/runs/4051239387/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050911516/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/datalad-extensions/actions/runs/4050197949/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-archive/actions/runs/4049085688/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/dandi/dandi-schema/actions/runs/4051239387/logs
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://api.github.com/repos/datalad/git-annex-ci-client-jobs/actions/runs/4050911516/logs

@jwodder
Copy link
Member

jwodder commented Jan 31, 2023

@yarikoptic Is tinuous able to download any logs at all, or do they all fail with 500?

@yarikoptic
Copy link
Member Author

didn't check, but I think I have not seen this issue for some time (a day). So it was worth complaining -- the gods have heard me ;) if it reemerges - I will reopen

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jwodder jwodder

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@yarikoptic @jwodder