Renovate: Allow self-hosted presets

[rtaum]

[rarkins]

This can apply to both GitHub and GitLab. Right now the github> namespace implies github.com. What namespace would we use for self-hosted? Assuming we don't mean arbitrary hosts, then we could make it self> to imply the same host as Renovate is running against. @hutson ?

[mikebryant]

I think self> is a good option, and would satisfy what most people are likely to want.

[rarkins]

After I wrote that I was wondering about “local>” as maybe more self explanatory. What do you think?

[mikebryant]

Hmm. local> is also good. I would be happy with either

[patrickmarabeas]

Alternatively, specifying no namespace uses the RENOVATE_ENDPOINT:

org/repo rather than namespace>org/repo

[rarkins]

@patrickmarabeas that's interesting.. not intended but interesting!

[rarkins]

However we need to ensure that github>a/b always points to github.com/a/b or it will have a lot of unintended consequences.

We'll use local>a/b to mean repo a/b on the same platform/endpoint that Renovate is running on.

[rarkins]

Actually now I’m reconsidering if we should lead the “accidental feature” if it doesn’t conflict with anything else.

[patrickmarabeas]

Specifying a namespace should only be necessary if you're changing context.

I would assume x/y points to the installed context - whether ghe, github.com, gitlab, etc; where as github>x/y will specifically go to github.com regardless of where Renovate is installed.

[rarkins]

Yeah, the main issue is collision/confusion with npm namespace, which came first. But I think it's probably ok

[patrickmarabeas]

On that note - this should also accomodate for self-hosted NPM registries.

Perhaps this should be raised in another issue - Does Renovate consume @scope:registry config (multiple as well)?

[rarkins]

On self-hosted, it should be possible for Renovate to look up private repos for presets if the .npmrc is configured on disk or in the bot's global config.

[rarkins]

I've clarified the current namespacing here: https://github.com/renovatebot/renovate/blob/master/website/docs/config-presets.md#preset-hosting

So it seems like abc/xyz should be technically and logically OK to point to the "local" repo matching that spec.

Next steps:

• [ ] Determine if behaviour is really working completely/reliably already by accident
• [ ] Determine if any feature gaps are there. For example does it work with github.com and GHE, does it work with both self-hosted GitLab as well as gitlab.com?

[jgarec]

+1 for this feature.
How can i help you ?

[rarkins]

@jgarec some help would be appreciated, even if just to test it later on before merge. Which platform do you run your self-hosted bot against? e.g. GitHub Enterprise, GitLab CE/EE, or Bitbucket Server?

[jgarec]

Gitlab CE

[iamstarkov]

I can help with bitbucket server testing

[rgomez90]

@rarkins what is the state of this?

This would be a very nice feature!

[rarkins]

It has not progressed beyond what was documented earlier in this thread, particularly:

The idea is that if you're hosted on something like https://gitlab.company.com and you specify a preset as foo/bar then it would resolve to https://gitlab.company.com/foo/bar. It needs to be verified if this already works, as was mentioned above.

[rgomez90]

@rarkins I am using self hosted Gitlab EE, it is available in our LAN at http://gitlab.ourcompany-d.local.
The Gitlab Instance is NOT available from outside our LAN.

This is the output of CI

DEBUG: File config
       "config": {
         "platform": "gitlab",
         "endpoint": "http://gitlab.ourcompany-d.local/api/v4/",
         "token": "***********",
         "logLevel": "debug",
         "gitAuthor": "Rafael Gómez <r.gomez@*******.com>",
         "assignees": ["rgomez"],
         "labels": ["renovate"],
         "onboarding": true,
         "onboardingConfig": {"extends": ["bots/renovate-shared-config"]},
         "repositories": ["rgomez/t4quickquery"]
       }
....
....
....
 WARN: npm registry failure (repository=rgomez/t4quickquery)
       "err": {
         "name": "HTTPError",
         "host": "registry.npmjs.org",
         "hostname": "registry.npmjs.org",
         "method": "GET",
         "path": "/renovate-config-bots%2Frenovate-shared-config",
         "protocol": "https:",
         "url": "https://registry.npmjs.org/renovate-config-bots%2Frenovate-shared-config",
         "gotOptions": {
           "path": "/renovate-config-bots%2Frenovate-shared-config",
           "protocol": "https:",
           "slashes": true,
           "auth": null,
           "host": "registry.npmjs.org",
           "port": null,
           "hostname": "registry.npmjs.org",
           "hash": null,
           "search": null,
           "query": null,
           "pathname": "/renovate-config-bots%2Frenovate-shared-config",
           "href": "https://registry.npmjs.org/renovate-config-bots%2Frenovate-shared-config",
           "headers": {
             "user-agent": "https://github.com/renovatebot/renovate",
             "cache-control": "no-cache",
             "accept": "application/json",
             "accept-encoding": "gzip, deflate"
           },
           "hooks": {
             "beforeError": [],
             "init": [],
             "beforeRequest": [],
             "beforeRedirect": [],
             "beforeRetry": [],
             "afterResponse": []
           },
           "retry": {"methods": {}, "statusCodes": {}, "errorCodes": {}},
           "decompress": true,
           "throwHttpErrors": true,
           "followRedirect": true,
           "stream": false,
           "form": false,
           "json": true,
           "cache": false,
           "useElectronNet": false,
           "method": "GET",
           "gotTimeout": {"request": 60000}
         },
         "statusCode": 405,
         "statusMessage": "Method Not Allowed",
         "headers": {
           "date": "Fri, 22 Nov 2019 04:22:45 GMT",
           "content-type": "application/json",
           "content-length": "63",
           "connection": "close",
           "set-cookie": [
             "__cfduid=dfc881606f49cbca0f258f0805465c29d1574396565; expires=Sun, 22-Dec-19 04:22:45 GMT; path=/; domain=.npmjs.org; HttpOnly"
           ],
           "cf-ray": "53981fc7ff1797e4-FRA",
           "cf-cache-status": "DYNAMIC",
           "allow": "PUT, POST",
           "expect-ct": "max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
           "vary": "Accept-Encoding",
           "server": "cloudflare"
         },
         "body": {"code": "MethodNotAllowedError", "message": "GET is not allowed"},
         "message": "Response code 405 (Method Not Allowed)",
         "stack": "HTTPError: Response code 405 (Method Not Allowed)\n    at EventEmitter.emitter.on (/usr/src/app/node_modules/got/source/as-promise.js:74:19)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       },
       "regUrl": "https://registry.npmjs.org/",
       "depName": "renovate-config-bots/renovate-shared-config"
DEBUG: Preset fetch error (repository=rgomez/t4quickquery)
       "err": {
         "message": "registry-failure",
         "stack": "Error: registry-failure\n    at Object.getDependency (/usr/src/app/dist/datasource/npm/get.js:222:19)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"
       }

I set bots/renovate-shared-config (the repo containing the shared config) in extends.
If I did nothing wrong (which I may since is my first time trying renovate) it is trying to fetch a package from npmjs with the path /renovate-config-bots%2Frenovate-shared-config

For now, renovate is working perfectly. The only issue I am having is not being able to configure a custom preset.

For now every repo has its own config, but would be awesome to be able to have a preset, since all our repos will have the same config.

[rarkins]

Thanks. Looks like the “local” preset is not correctly detected after all so we’ll need to add some logic to detect that particular use case.

[rgomez90]

Your welcome. Let me know if I can help with something else :)

[oludiro]

Hi, @rarkins is there a branch of renovate bot that I should try to test out hosting renovate preset config in self-hosted GitLab ?

[rarkins]

No, there isn't

[Lucas-C]

I had a look at the code and this looks non-trivial.

In the getPreset function, parsePreset is called to figure out what datasource to query.

Currently the logic is "for 1 preset, we query 1 datasource", whereas with the proposed change abc/xyz could either point to a NPM datasource, or a GitHub/Gitlab datasource.
Hence several datasources will have to be tested in the getPreset function, in a fixed order of priority (NPM before GitHub/Gitlab datasource probably, to be fully backward-compatible).

Does this makes sense to you ?
Do you agree that the next step here would be to make a change to the getPreset function in order to support this ?

[rarkins]

We want to avoid any scenario where there is a potential "namespace" conflict and double lookups. You definitely wouldn't want case where you get an accidental hit on npm which results in the wrong config.

It's been a while since I looked at it, but I had thought that abc/xyz wasn't a used namespace already, but I could be remembering wrong. Alternatively, we can go with a namespace like local>abc/xyz instead. In that case the logic would be:

• Detect that it's a local preset, and strip the local> prefix
• Look up what the current platform and endpoint are
• If the platform is a supported one for getPreset, then attempt to fetch preset from the.end.point/abc/xyz

[countergram]

It does look like abc/xyz looks up on npmjs.com, even without the presence of a @ leading character.

[renovate-release]

:tada: This issue has been resolved in version 19.208.0 :tada:

The release is available on:

• GitHub release
• 19.208.0

Your semantic-release bot :package::rocket:

[mountainash]

Internally our GitLab-ci ran overnight [using Docker Image renovate/renovate:19.208.2]. A few projects/repositories have "Action Required: Fix Renovate Configuration" issues opened on them this morning.

  WARN: Config Warning (repository=new-ci/meet)
        "configError": {
          "validationError": "Cannot find preset's package (github>Init7/renovate-shared-config)"
        },
        "res": "created"

Which was made is public (from a GitLab mirror) at https://github.com/Init7/renovate-shared-config. and has been working OK in previous versions of Renovate.

[viceice]

Can you please check the DEBUG logs, there should be a message

[mountainash]

DEBUG: Repo is onboarded (repository=website/init7.net)
DEBUG: Found renovate.json config file (repository=website/init7.net)
DEBUG: Repository config (repository=website/init7.net)
       "configFile": "renovate.json",
       "config": {"extends": ["github>Init7/renovate-shared-config"]}
DEBUG: migrateAndValidate() (repository=website/init7.net)
DEBUG: No config migration necessary (repository=website/init7.net)
DEBUG: massaged config (repository=website/init7.net)
       "config": {"extends": ["github>Init7/renovate-shared-config"]}
DEBUG: migrated config (repository=website/init7.net)
       "config": {"extends": ["github>Init7/renovate-shared-config"]}
DEBUG: Failed to retrieve default.json from repo (repository=website/init7.net)
       "statusCode": undefined
DEBUG: default.json preset not found - trying renovate.json (repository=website/init7.net)
DEBUG: Failed to retrieve renovate.json from repo (repository=website/init7.net)
       "statusCode": undefined
DEBUG: Preset fetch error (repository=website/init7.net)
       "err": {
         "message": "dep not found",
         "stack": "Error: dep not found\n    at fetchJSONFile (/usr/src/app/dist/config/presets/github.js:27:15)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
       }
 INFO: Throwing preset error (repository=website/init7.net)
 INFO: Repository has invalid config (repository=website/init7.net)
       "error": {
         "validationError": "Cannot find preset's package (github>Init7/renovate-shared-config)"
       }
DEBUG: raiseConfigWarningIssue() (repository=website/init7.net)
DEBUG: getBranchPr(renovate/configure) (repository=website/init7.net)

I'm guessing the "Preset fetch error" has something to do with it, but it doesn't really expand much on the "why".
I just created a new GH token and still got the same error - so I don't think it's a permission issue.

[viceice]

can you please open a new issue with all infos and reference this issue

[mountainash]

Still need another issue created? It looks like you already done all the hard work. Thanks.

[viceice]

no, i found the problem. Need to wait for merge. You can now reference your internal presets from gitlab. See samples below.

{
  "extends": [
    "gitlab>Init7/renovate-shared-config",
    "local>Init7/renovate-shared-config",
    "Init7/renovate-shared-config"
  ]
}

[mountainash]

Yes thanks. That was the plan, so it was pretty good that it actually failed (as it reminds us to update all our repos to point from GH to internally). But I wanted to raise it as it could break many other setups for other users that which to continue referencing a public config.

[mountainash]

For further reference (for other searches) this refers to https://docs.renovatebot.com/config-presets/#local-presets