Renovate: Configuring API Tokens for SCM Platforms

[ghost]

We run an on-premise instance of Renovate, and when it attempts to access GitHub.com to fetch release notes, it quickly exhausts the API limit for a single IP address, and substantially slows the renovation process.

The specific error: API rate limit exceeded for [IP ADDRESS]. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) (403)

I'm not sure how to set a token for our on-premise SCM, and for GitHub.com, at the same time so that we can have a higher API request limit.

Also, we would still be bound by the 5000 upper limit of an authenticated user. Not sure how to work around that for large on-premise installations.

[ghost]

Changing the issue title to reflect the general desire to configure an API token per SCM platform, such as one for GitHub.com, one for GitLab.com, one for out internal SCM platform, etc.

[rarkins]

Currently:

• We support only GitHub as a source for changelogs/commits
• We delete GITHUB_TOKEN from env before querying for changelogs if we detect that GHE has been configured

The quickest solution for now would be to allow both GHE token and github.com token simultaneously.

The ideal long term solution is to explicitly define/configure tokens per-platform, which would require a new configuration concept.

[rarkins]

@destroyerofbuilds I made a slight change in [email protected] where it should now use GITHUB_COM_TOKEN for retrieving release notes, if you have defined it in addition to a GHE endpoint. This issue otherwise remains open, i.e. to implement a way to define authentication explicitly for multiple platforms instead of overloading env.

[rarkins]

Would it look something like this? Kind of like packageRules

• A configuration object of type list/array with name "authentication"
• A "host" field which can be a regex (e.g. for wildcard matching)
• A field like token, but could be expanded one day to include options for username and password, etc

[ghost]

Would it look something like this? Kind of like packageRules

I could see a structure similar to packageRules for configuring platform-specific authentication credentials.

A configuration object of type list/array with name "authentication"

This would not make sense on a per-project basis, except in the rare case that the project depends on a private dependency and specific credentials are needed to fetch Release notes for that dependency.

In the general case I imagine each Renovate instance would configure the credentials at the global level, so as to be used for every project renovated.

On the rare occasion that a project needs to fetch release notes for a private dependency, they will need to be able to _extend_ the list of authentication token used for a given host, and further, to specify which packages that token should be used for (They probably want their private token only used to fetch release notes for their private dependencies, and not for general access to a SCM platform)

A "host" field which can be a regex (e.g. for wildcard matching)

Since each platform, such as GitHub and GitLab generate their own tokens, and do not allow you to set your own, I don't believe there would be a situation where you would want, or could, share a single token with multiple hosts.

Therefore it is probably enough to have one object per fully qualified domain name, and within that object have an array of authentication objects. Each authentication object would contain a token, and potentially an array of regex expressions for packages that the token should be used to fetch release notes for.

A field like token, but could be expanded one day to include options for username and password, etc

Sounds fine with me.

[rarkins]

If we define an "authentication" list object, it could be configured either at the bot level (for self-hosted) and/or augmented per-repository.

And we could have a set of reusable config object names to be used, e.g. username, password, token. A field like token may mean different things to different platforms.

Examples:

  "docker": {
    "authentication": [
      {
        "host": "hub.docker.com",
        "username": "rarkins",
        "password": "foo"
      },
      {
        "host": "internal.company.com",
        "username": "dev",
        "password": "bar"
      }
    ]
  }

[ghost]

@rarkins your example seems fine to me.

[rarkins]

Another real example for configuring tokens for GHE:

    "authentication": [
      {
        "host": "github.com",
        "token": "aaaaaaaaaaaaaaaaaaaaaaaaa"
      },
      {
        "host": "github.company.com",
        "token": "bbbbbbbbbbbbbbbbbbbbbbbbb"
      }
    ]

or similarly for GitLab:

    "authentication": [
      {
        "host": "github.com",
        "token": "aaaaaaaaaaaaaaaaaaaaaaaaa"
      },
      {
        "host": "gitlab.company.com",
        "token": "bbbbbbbbbbbbbbbbbbbbbbbbb"
      }
    ]

The next question is how to pass them to the API layer. e.g. should we have our own extension of got that all datasources use and which can then understand how to use this authentication? Doing that though means it would need to be somewhat service-aware, e.g. how to put an npm token into headers, how to put a github token in, etc.

[kayoub5]

@rarkins what do you think about the word credentials instead of authentication

[rarkins]

Fine by me