Renovate: Kustomize require SSH to lookup [email protected] dependencies

Created on 12 Oct 2020 · 6Comments · Source: renovatebot/renovate

What Renovate type, platform and version are you using?

Renovate 23.47.2 via CLI in a Docker container without SSH key.

Describe the bug

The kustomize manager fail to lookup Github dependency that make use of SSH, for example:

bases:
  - [email protected]:moredhel/remote-kustomize.git?ref=v0.0.0

Here is an example of the on-boarding process for 2 different package file (kustomize and terraform). Renovate fail with kustomize but succeed with Terraform.

Relevant debug logs

DEBUG: Datasource unknown error (repository=[redacted])
       "datasource": "git-tags",
       "lookupName": "github.com/[redacted]",
       "err": {
         "task": {
           "commands": ["ls-remote", "github.com/[redacted]"],
           "format": "utf-8"
         },
         "message": "fatal: 'github.com/[redacted]' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
         "stack": "Error: fatal: 'github.com/[redacted]' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n\n    at GitExecutorChain.onFatalException (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:60:87)\n    at GitExecutorChain.<anonymous> (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:51:28)\n    at Generator.throw (<anonymous>)\n    at rejected (/usr/src/app/node_modules/simple-git/src/lib/runners/git-executor-chain.js:6:65)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
       }

To Reproduce

$ docker run -ti \
    -e GITHUB_TOKEN=$GITHUB_TOKEN \
    -v $PWD:/app \
    -v renovate.private-key.pem:/usr/src/app/renovate.private-key.pem \
    -w /app \
    node:14 bash
root@d03b97c8e859:/app# yarn start --log-level debug --token $GITHUB_TOKEN etiennetremel/renovate-kustomize-temp

Additional context

https://github.com/whitesource/renovate-on-prem/issues/121

priority-3-normal bug

Source

etiennetremel

Most helpful comment

In that case, let's merge the current PR if it's otherwise OK and do the refactoring to extract common functions later.

rarkins on 13 Oct 2020

👍2

All 6 comments

Need to determine from logs: what exact depName/lookupName does Renovate extract for this?
Do we need to change the extracted value, or can we improve the git-tags datasource to redirect to github-tags?

rarkins on 12 Oct 2020

Need to determine from logs: what exact depName/lookupName does Renovate extract for this?

I was not able to find documentation around all these depName/depNameShort/lookupName values, do you have a link to share?

Below is a preview of the logs with --log-level=debug using Renovate CLI v23.47.2:

DEBUG: packageFiles with updates (repository=etiennetremel/renovate-kustomize-temp)
       "config": {
         "kustomize": [
           {
             "packageFile": "kustomization.yaml",
             "manager": "kustomize",
             "deps": [
               {
                 "datasource": "git-tags",
                 "depName": "[email protected]:moredhel/remote-kustomize.git",
                 "lookupName": "[email protected]:moredhel/remote-kustomize.git",
                 "currentValue": "v0.0.0",
                 "depIndex": 0,
                 "updates": [
                   {
                     "fromVersion": "v0.0.0",
                     "toVersion": "v0.0.1",
                     "newValue": "v0.0.1",
                     "newMajor": 0,
                     "newMinor": 0,
                     "updateType": "minor",
                     "isSingleVersion": true,
                     "newDigest": "755e3dac2d1696c507fb1ab433434e5f42828b65"
                   }
                 ],
                 "warnings": [],
                 "fixedVersion": "v0.0.0"
               }
             ]
           }
         ],
         "terraform": [
           {
             "packageFile": "main.tf",
             "manager": "terraform",
             "deps": [
               {
                 "depType": "github",
                 "depName": "github.com/moredhel/remote-kustomize",
                 "depNameShort": "moredhel/remote-kustomize",
                 "currentValue": "v0.0.0",
                 "datasource": "github-tags",
                 "lookupName": "moredhel/remote-kustomize",
                 "depIndex": 0,
                 "updates": [
                   {
                     "fromVersion": "v0.0.0",
                     "toVersion": "v0.0.1",
                     "newValue": "v0.0.1",
                     "newMajor": 0,
                     "newMinor": 0,
                     "updateType": "minor",
                     "isSingleVersion": true
                   }
                 ],
                 "warnings": [],
                 "fixedVersion": "v0.0.0",
                 "sourceUrl": "https://github.com/moredhel/remote-kustomize"
               }
             ]
           }
         ]
       }

Then the following logs are after using #7456 :

DEBUG: packageFiles with updates (repository=etiennetremel/renovate-kustomize-temp)
       "config": {
         "kustomize": [
           {
             "packageFile": "kustomization.yaml",
             "manager": "kustomize",
             "deps": [
               {
                 "currentValue": "v0.0.0",
                 "datasource": "github-tags",
                 "depName": "moredhel/remote-kustomize",
                 "depType": "github",
                 "depIndex": 0,
                 "updates": [
                   {
                     "fromVersion": "v0.0.0",
                     "toVersion": "v0.0.1",
                     "newValue": "v0.0.1",
                     "newMajor": 0,
                     "newMinor": 0,
                     "updateType": "minor",
                     "isSingleVersion": true
                   }
                 ],
                 "warnings": [],
                 "fixedVersion": "v0.0.0",
                 "sourceUrl": "https://github.com/moredhel/remote-kustomize"
               }
             ]
           }
         ],
         "terraform": [
           {
             "packageFile": "main.tf",
             "manager": "terraform",
             "deps": [
               {
                 "depType": "github",
                 "depName": "github.com/moredhel/remote-kustomize",
                 "depNameShort": "moredhel/remote-kustomize",
                 "currentValue": "v0.0.0",
                 "datasource": "github-tags",
                 "lookupName": "moredhel/remote-kustomize",
                 "depIndex": 0,
                 "updates": [
                   {
                     "fromVersion": "v0.0.0",
                     "toVersion": "v0.0.1",
                     "newValue": "v0.0.1",
                     "newMajor": 0,
                     "newMinor": 0,
                     "updateType": "minor",
                     "isSingleVersion": true
                   }
                 ],
                 "warnings": [],
                 "fixedVersion": "v0.0.0",
                 "sourceUrl": "https://github.com/moredhel/remote-kustomize"
               }
             ]
           }
         ]
       }

Do we need to change the extracted value, or can we improve the git-tags datasource to redirect to github-tags?

I opened a https://github.com/renovatebot/renovate/pull/7456 which use the github-tags when possible.

etiennetremel on 13 Oct 2020

@viceice @JamieMagee which approach do you prefer?

manager detects github.com tag pattern and extracts github-tags instead of git-tags deps, or
manager extracts git-tags but then the git-tags datasource detects github patterns and redirects the lookups

(1) is as per above and perhaps looks "cleanest"
(2) reduces the amount of duplicated code as this will be used across multiple managers. duplicated coded can perhaps be put into a library though.

So it partly comes down to: where is the best logical location to understand "this tag is from github.com"?

rarkins on 13 Oct 2020

I think we should do it at manager level and extract detection to a reusable utils function.