Github API rate limit

[isarkis]

We have a runner deployment consisting of about 360 ephemeral runners. Everything seems to be working just fine except we see the following error in the controller log:

"ERROR actions-runner-controller.runner Failed to check if runner is busy due to Github API rate limit. Retrying in 30s to avoid excessive GitHub API calls"

It seems like the error goes away after rate limit resets, and then comes back after about 20 mins or so. We currently have 5000 api requests per hour. Couple of questions:

• So far we haven't observed any problems - the runners process jobs as expected, pods get deleted, and new runners get created just fine. Is this something we should worry about?
• Why controller is making so many API requests? Is there a way to reduce the number of API requests?

[toast-gear]

It just takes a lot of API requests to maintain the right runner counts. Some work is being done to cache requests #1127 which should help and I'm sure we are doing some API requests that could be consolidated but fundamentally it just takes a lot of requests as the only way ARC can get any information is querying the API. If you use a GitHub app instead of a PAT you can get a rate limited budget of 15,000 requests.

[booleanbetrayal]

@toast-gear - Is there any sort of API call to validate the rate limiting of a GitHub App? I was unable to find any documentation to that effect.