Stuck on "Getting tags"

[grantmcconnaughey]

I have a fairly large repo with 41,000 commits, 2,400 tags, and 15,000 total PRs and issues. I ran gren changelog --generate for 7 hours overnight and it never moved past the "Getting tags" state.

This is my .grenrc.json file:

{
    "username": "REDACTED",
    "repo": "REDACTED,
    "token": "REDACTED",
    "dataSource": "prs",
    "limit": 1,
    "ignoreIssuesWith": [
        "wontfix",
        "duplicate"
    ],
    "groupBy": {
        "Enhancements:": ["enhancement", "internal"],
        "Bug Fixes:": ["bug"]
    }
}

I added "limit": 1 to try to fix this issue and that didn't seem to work.

I'm using gren version 0.17.1. The command I'm running is gren changelog --generate.

[alexcanessa]

Hello. That's a lot of tags lol.

I think limit: 1 would only make it worse because it would make 2,400 calls (with pages of 1 tag each). So in your case, it'd be better to have the highest limit possible (which I can't remember what is).

I guess the quickest thing to do is to add more information about the tags that it's getting like n of total.

Then we need to work on the performance.

[grantmcconnaughey]

Hey Alex! Thanks for the quick reply.

The docs state that the limit argument does this:

Just produce release notes for the last releases.

So I was thinking it would only produce release notes for the last 1 tag, but not change the page size. Does limit also affect the tag page size?

Showing n of total would be great!

[grantmcconnaughey]

I changed limit to 100 (the GitHub page size), but it seems to have maxed out at 30. That appears to be related to this issue: #209

[grantmcconnaughey]

Looks like the limit in the codebase is 99, so when I try to use a limit of 100 it reverts to 30. Using a limit of 99 works.

I'll try that and let you know if it still times out.

[alexcanessa]

Ok thanks.

I still think we should add some more information, at least to understand if it's actually stack, or it's working.

[grantmcconnaughey]

I added paging and confirmed that it is pulling all tag pages, but on the last page it fails then gets stuck spinning and showing "Getting tags."

Before getting stuck the script spits out this error:

⠏ Getting tagsError: 403 error making request get https://api.github.com/repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a: "Forbidden"
    at new ResponseError (/usr/local/lib/node_modules/github-release-notes/node_modules/github-api/dist/components/Requestable.js:50:118)
    at handler (/usr/local/lib/node_modules/github-release-notes/node_modules/github-api/dist/components/Requestable.js:379:18)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  path: '/repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a',
  request: {
    url: 'https://api.github.com/repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a',
    method: 'get',
    params: {},
    data: null,
    headers: {
      Accept: 'application/vnd.github.v3+json',
      'Content-Type': 'application/json;charset=UTF-8',
      Authorization: 'token thetoken',
      'User-Agent': 'axios/0.19.2'
    },
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    responseType: 'json',
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    validateStatus: [Function: validateStatus]
  },
  response: {
    status: 403,
    statusText: 'Forbidden',
    headers: {
      server: 'GitHub.com',
      date: 'Wed, 12 Feb 2020 15:59:40 GMT',
      'content-type': 'application/json; charset=utf-8',
      'transfer-encoding': 'chunked',
      connection: 'close',
      status: '403 Forbidden',
      'retry-after': '1',
      'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type',
      'access-control-allow-origin': '*',
      'x-github-media-type': 'github.v3; format=json',
      'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
      'x-frame-options': 'deny',
      'x-content-type-options': 'nosniff',
      'x-xss-protection': '1; mode=block',
      'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
      'expect-ct': 'max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"',
      'content-security-policy': "default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com www.githubstatus.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com collector.githubapp.com avatars0.githubusercontent.com avatars1.githubusercontent.com avatars2.githubusercontent.com avatars3.githubusercontent.com github-cloud.s3.amazonaws.com; manifest-src 'self'; media-src 'none'; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com",
      vary: 'Accept-Encoding, Accept',
      'x-github-request-id': 'C35F:704A:B44C8:1B4234:5E4420EB'
    },
    config: {
      url: 'https://api.github.com/repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a',
      method: 'get',
      params: {},
      data: null,
      headers: [Object],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      responseType: 'json',
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      validateStatus: [Function: validateStatus]
    },
    request: ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 6,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [TLSSocket],
      _header: 'GET /repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a HTTP/1.1\r\n' +
        'Accept: application/vnd.github.v3+json\r\n' +
        'Content-Type: application/json;charset=UTF-8\r\n' +
        'Authorization: token thetoken\r\n' +
        'User-Agent: axios/0.19.2\r\n' +
        'Host: api.github.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      path: '/repos/OurOrg/OurRepo/git/commits/52f1d7558b8675f407afc2f411737f463d3e026a',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: {
      documentation_url: 'https://developer.github.com/v3/#abuse-rate-limits',
      message: 'You have triggered an abuse detection mechanism. Please wait a few minutes before you try again.'
    }
  },
  status: undefined
}

[grantmcconnaughey]

I've confirmed that that commit does exist.

[v1v]

I've just seen this issue, I haven't had this issue until the last couple of weeks, the process gets stalled in the ⠴ Getting the list of releases it's not related to the tags, but the similar experience of a stalled execution.

Not sure how to debug this further, the initial scripts were based on:

version: 0.17.0
commands: gren release --token="${GREN_GITHUB_TOKEN}" --override -c .grenrc.js -t all
cat .grenrc.js

module.exports = {
    "dataSource": "prs",
    "prefix": "",
    "onlyMilestones": false,
    "ignoreCommitsWith": ["chore", "refactor", "style"],
    "ignoreIssuesWith": ["no-release"],
    "ignoreTagsWith": ["-rc", "-alpha", "-beta", "test", "current"],
    "ignoreLabels": ["closed", "automation", "enhancement", "bug", "fix",
      "internal", "feature", "feat", "docs", "chore", "refactor", "ci",
      "perf", "test", "tests", "style", "groovy", "linux", "master", "mac", "windows"],
    "groupBy": {
        "Enhancements": ["enhancement", "internal", "feature", "feat"],
        "Bug Fixes": ["bug", "fix"],
        "Documentation": ["docs", "question"],
        "No user affected": ["chore", "refactor", "perf", "test", "style"],
        "CI": ["ci"]
    },
    "changelogFilename": "CHANGELOG.md",
    "template": {
        commit: ({ message, url, author, name }) => `- [${message}](${url}) - ${author ? `@${author}` : name}`,
        issue: "- {{labels}} {{name}} [{{text}}]({{url}})",
        label: "[**{{label}}**]",
        noLabel: "closed",
        changelogTitle: "# Changelog\n\n",
        release: "## {{release}} ({{date}})\n{{body}}",
        releaseSeparator: "\n---\n\n",
        group: function (placeholders) {
          var icon = "🙈"
          if(placeholders.heading == 'Enhancements'){
            icon = "🚀"
          } else if(placeholders.heading == 'Bug Fixes'){
            icon = "🐛"
          } else if(placeholders.heading == 'Documentation'){
            icon = "📚"
          } else if(placeholders.heading == 'No user affected'){
            icon = "🙈"
          } else if(placeholders.heading == 'CI'){
            icon = "⚙️"
          }
          return '\n#### ' + icon + ' ' + placeholders.heading + '\n';
        }
    }
}

Any ideas on how I can debug it? I did try with -B but it was not helpful

Thanks

[mickmaccallum]

I'm seeing a similar issue here, but in my case after hanging for a while on "Getting tags," I eventually get the error:

TypeError: Cannot read property 'status' of undefined
at handler (/Users/mick/.config/yarn/global/node_modules/github-api/dist/components/Requestable.js:372:40)
at process._tickCallback (internal/process/next_tick.js:68:7)

[alexcanessa]

🤔 this bug is so weird.

[thomasmichaelwallace]

And another datapoint:

Having just moved to GitHub, we're looking to tidy up our Releases. We have 239 tags/releases on our private repo.

Running yarn gren release --data-source=commits results in an indefinite (>10 minutes) hang, which I guess is maybe fair enough; I wouldn't expect "bulk import" to be something you've focused on.

But narrowing it down: yarn gren release -B --data-source=commits --tags=v5.73.1 --limit=1 (also tried without --limit and from ./node_modules/.bin directly) always results in:

Options:
Tags: v5.73.1
Prefix: empty
Template: [object Object]
Prerelease: false
Generate: false
Quiet: false
Override: false
Debug: true
Ignore labels: empty
Ignore issues with: empty
Ignore commits with: empty
Group by: false
Milestone match: Release {{tag_name}}
Username: username
Repo: repo
Token: token
Limit: 1
Data source: commits
Include messages: commits
Ignore tags with: empty

🤖  - Generate release notes:
===================================
✔ Releases found: 0
⠦ Getting tagsTypeError: Cannot read property 'status' of undefined
    at handler (~/node_modules/github-api/dist/components/Requestable.js:372:40)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
error Command failed with exit code 1.