Renovate: datasource: support private git repositories via hostRules

[slauger]

The git submodule support in combination with with private repositories (via SSH) currently results in a dependency lookup error.

INFO: Failed to look up dependency test-repo (repository=misc/submodule-renovate-test,
packageFile=.gitmodules, dependency=test-repo)

config.js

module.exports = {
 platform: 'gitlab',
 endpoint: 'https://gitlab.example.com/api/v4/',
 token: process.env.RENOVATE_TOKEN,
...
  "git-submodules": { enabled: true },
...
};

.gitmodules:

-bash$ cat .gitmodules 
[submodule "test-repo"]
    path = test-repo
    url = [email protected]:misc/test-repo.git
    branch = master

For authentication i've already tried to add an ssh-agent and a private key (i'm running renovate via Gitlab CI).

.gitlab-ci.yml

---
image: 
  name: renovate/renovate:19.102-slim
  entrypoint: [""]

stages:
  - renovate

before_script:
  - cp config.js /usr/src/app/config.js
  - eval $(ssh-agent -s)
  - echo "${SSH_PRIVATE_KEY}" | tr -d '\r' | ssh-add - > /dev/null
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - ssh-keyscan gitlab.example.com >> ~/.ssh/known_hosts

renovate:
  stage: renovate
  script: 'node /usr/src/app/dist/renovate.js $(cat repos.txt | xargs)'

When using public repositories (e.g. on GitHub) everything just works as expected.

Please provide us an option to renovate private git submodules.

[slauger]

@JamieMagee FYI

[rarkins]

So RENOVATE_TOKEN is enough to authenticate with that repo?

[slauger]

So RENOVATE_TOKEN is enough to authenticate with that repo?

In that case it should, but authentication with RENOVATE_TOKEN is only functional via HTTPS. How should renovate know that it could also checkout this repository via HTTPS?

Most of the submodule setups i've seen use an SSH origin the the .gitmodules file.

Anyways, i've already tried to switch the origin of the submodule to a HTTPS URL. This dosen't change anything.

[rarkins]

In that case it should, but authentication with RENOVATE_TOKEN is only functional via HTTPS. How should renovate know that it could also checkout this repository via HTTPS?

I was going to ask you the same question :)

If your submodule requires SSH authentication, then couldn't you add/mount the required files into your Renovate container ~/.ssh to provide that authentication? If you want to get Renovate to do it on your behalf, it doesn't sound like it would be any less work.

Update: Sorry, I just realised you made some attempts to achieve the same, so I guess we're on the same path. I don't believe that Renovate is doing anything to block that authentication. Are you sure that you're applying the settings into Renovate's container and not instead into the GitLab CI container that is running Renovate inside?

[slauger]

Update: Sorry, I just realised you made some attempts to achieve the same

The git checkout seems to work fine - even via SSH. If the SSH key is not loaded into the agent (or does not have access to the submodule) i recieve the following error message for each submodule.

99 remote: 
100 remote: ========================================================================
101 remote: 
102 remote: The project you were looking for could not be found.
103 remote: 
104 remote: ========================================================================
105 remote: 
106 fatal: Could not read from remote repository.
107 Please make sure you have the correct access rights
108 and the repository exists.

That makes me thinking that the problem is in the depenency lookup step, not in the repository checkout.

[rarkins]

Thanks for that troubleshooting. I guess @JamieMagee will check whether the lookup calls are leveraging git authentication in the container or not.

[JamieMagee]

Thanks for all the information. I'll try to put together a repro repo, and debug through this later.

[JamieMagee]

@slauger So I have a repo JamieMagee/renovate-git-submodule-repro. It's not setup with a private repo, but that shouldn't affect it. SSH access is what's important here.

I wasn't able to repro the exact issue on GitHub or GitLab, but I have some followup questions for you:

1. Can you run git ls-remote [email protected]:misc/test-repo.git master with and without ssh-agent setup
2. Can you run ssh [email protected] with and without ssh-agent setup

The output from the above should help to debug the issue more.

[slauger]

• submodule-renovate-test: the main repository
• test-repo: the submodule (included in the main repository)

My current gitlab-ci.yml:

---
image:
  name: docker.io/renovate/renovate
  entrypoint: [""]

test_ssh_connection:
  stage: test
  script:
    # ssh host key scan
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan git.example.com >> ~/.ssh/known_hosts
    # without ssh-agent
    - echo "test without ssh-agent"
    - git ls-remote [email protected]:misc/test-repo.git || true
    - git ls-remote [email protected]:misc/submodule-renovate-test.git || true
    - ssh [email protected] || true
    # with ssh-agent
    - echo "test with ssh-agent"
    - eval $(ssh-agent -s)
    - echo "${BOT_SSH_KEY}" | tr -d '\r' | ssh-add - > /dev/null
    - git ls-remote [email protected]:misc/test-repo.git
    - git ls-remote [email protected]:misc/submodule-renovate-test.git
    - ssh [email protected]
  only:
    - master

renovate_gitlab:
  variables:
    RENOVATE_PLATFORM: gitlab
    RENOVATE_ENDPOINT: https://git.example.com/api/v4/
  stage: renovate
  script:
    - eval $(ssh-agent -s)
    - echo "${BOT_SSH_KEY}" | tr -d '\r' | ssh-add - > /dev/null
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan git.example.com >> ~/.ssh/known_hosts
    - cp config.js /usr/src/app/config.js
    - node /usr/src/app/dist/renovate.js $(cat repos_gitlab.txt)
  only:
    - master
...

Log Output for test_ssh_connection job:
```
Running with gitlab-runner 12.8.0 (1b659122)
on node01 9f12dcda
Using Docker executor with image docker.io/renovate/renovate ...
WARNING: Container based cache volumes creation is disabled. Will not create volume for "/cache"
Pulling docker image docker.io/renovate/renovate ...
Using docker image sha256:f8ffd2913cbc53ab6ff6a0a4d61aedd37d61dc7799af912563bd8f6058e81df0 for docker.io/renovate/renovate ...
Running on runner-9f12dcda-project-73-concurrent-0 via node01...
00:02
Fetching changes with git depth set to 50...
00:02
Initialized empty Git repository in /builds/misc/renovate-bot/.git/
Created fresh repository.
From https://git.example.com/misc/renovate-bot

• [new ref] refs/pipelines/2619 -> refs/pipelines/2619
• [new branch] master -> origin/master
  Checking out 560bbedc as master...
  Skipping Git submodules setup
  $ mkdir -p ~/.ssh
  00:06
  $ chmod 700 ~/.ssh
  $ ssh-keyscan git.example.com >> ~/.ssh/known_hosts
  # git.example.com:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
  # git.example.com:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
  # git.example.com:22 SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
  $ echo "test without ssh-agent"
  test without ssh-agent
  $ git ls-remote [email protected]:misc/test-repo.git || true
  Warning: Permanently added the ED25519 host key for IP address '192.168.100.100' to the list of known hosts.
  [email protected]: Permission denied (publickey).
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights
  and the repository exists.
  $ git ls-remote [email protected]:misc/submodule-renovate-test.git || true
  [email protected]: Permission denied (publickey).
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights
  and the repository exists.
  $ ssh [email protected] || true
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  [email protected]: Permission denied (publickey).
  $ echo "test with ssh-agent"
  test with ssh-agent
  $ eval $(ssh-agent -s)
  Agent pid 19
  $ echo "${BOT_SSH_KEY}" | tr -d '\r' | ssh-add - > /dev/null
  Identity added: (stdin) ([email protected])
  $ git ls-remote [email protected]:misc/test-repo.git
  98834ae6cc0e489ddef4da27b630f8fc7557c9f9 HEAD
  98834ae6cc0e489ddef4da27b630f8fc7557c9f9 refs/heads/master
  4dd5d391931fa03b8174ff57ff322f29dace06e9 refs/tags/v1.0.0
  98834ae6cc0e489ddef4da27b630f8fc7557c9f9 refs/tags/v1.0.1
  $ git ls-remote [email protected]:misc/submodule-renovate-test.git
  43a91cdd41ffa486027a46653c9e85b9a269259c HEAD
  43a91cdd41ffa486027a46653c9e85b9a269259c refs/heads/master
  319183d7e1dd5294e0090eadb19be362ab0149f3 refs/merge-requests/1/head
  061188cd5cfea9e17389bb89abb23b0860ce92af refs/merge-requests/1/merge
  5a54236c955898b1375df1e9f2e66698f2993505 refs/merge-requests/2/head
  cad713c6cea2a5002baf32dc6dd990b54bb11839 refs/merge-requests/2/merge
  $ ssh [email protected]
  Pseudo-terminal will not be allocated because stdin is not a terminal.
  Welcome to GitLab, @renovate-bot!
  Job succeeded
  ```

Log output for renovate_gitlab job:

... DEBUG: Found 2 package file(s) (repository=misc/submodule-renovate-test) DEBUG: manager.fetchUpdates() (repository=misc/submodule-renovate-test) DEBUG: Error looking up tags in test-repo (repository=misc/submodule-renovate-test) DEBUG: Failed to look up dependency test-repo (repository=misc/submodule-renovate-test, packageFile=.gitmodules, dependency=test-repo) DEBUG: packageFiles with updates (repository=misc/submodule-renovate-test) ...

[slauger]

Any progress or update on this issue @Jamesking56? If you need further information just let me know.

[nejch]

I just want to say that using relative submodule URLs (as is also suggested for GitLab CI, as that makes runners get the URL with job tokens from the superproject URL) works for private submodules for me.

Of course that doesn't solve the initial issue, as some users will have specific reasons for absolute URLs with SSH/GIT protocols in their submodules. Just in case you need a quick workaround @slauger.

Edit: obviously this assumes the repos are on the same instance, so this might not work for you.

[slauger]

I just want to say that using relative submodule URLs (as is also [suggested for GitLab CI]...(https://docs.gitlab.com/ee/ci/git_submodules.html#configuring-the-gitmodules-file)

Wow, a really cool feature! I didn't know that before (even if i use Gitlab CI since ages). This might be useful in some scenarios. Thanks. 👍

But as you already noticed, this don't solve the initial issue.

[Jamesking56]

I'm not sure what information you need from me @slauger ?

[slauger]

I'm not sure what information you need from me @slauger ?

Oops, sorry. I wanted to ping @JamieMagee, not you. Ignore that. Sorry for the confusion.

[hisener]

Any update on this? I think I have a similar issue related to authentication but in my case, it fails because we don't attach any ssh key at all.

IIUC, the following code executes git ls-remote ... but I'd expect a lookup by given GitHub token if the submodule is a GitHub URI. I think it makes sense to extend that functionality to use GitHub API when possible.
https://github.com/renovatebot/renovate/blob/76d8574581f862afbff027a53d2f2153f2192ce2/lib/datasource/git-submodules/index.ts#L26

cc @rarkins

[rarkins]

Your private submodule is on GitHub.com?

[viceice]

I think a better solution is to pass our hostRules via a custom auth helper to git, so we can support any http based git url. Thgis would also solve any other tools we use for updating with external binary tools (eg go).

Maybe in a later step we can support a custom ssh-agent to pass configured ssh keys to any git command. ssh-agent-js

ipc communication over unix sockets on shared docker volume: https://github.com/bbartolome/ipc-test

[rarkins]

Any example of custom http helpers for git? Eg do you mean a .netrc file or something else? Btw if binarySource=Docker we could run it in a container and be more free to manipulate any environment files we need

[viceice]

https://git-scm.com/docs/gitcredentials#_custom_helpers

simple shell script with passes the password from env variable

# or you can specify your own shell snippet
[credential "https://example.com"]
    username = your_user
    helper = "!f() { test \"$1\" = get && echo \"password=$SOME_PASSWORD\"; }; f"

simple stdin/stdout text protocol: https://git-scm.com/docs/git-credential#_typical_use_of_git_credential

[hisener]

Your private submodule is on GitHub.com?

Yes.

I think a better solution is to pass our hostRules via a custom auth helper to git, so we can support any http based git url. Thgis would also solve any other tools we use for updating with external binary tools (eg go).

We do use SSH URLs for git submodules. I think it makes sense to lookup by GitHub API since it's already provided. Having to provide another authentication mechanism is a bit inconvenient.

[rarkins]

Submodules datasource is a little unique today and I hope to bring it more inline with other datasources. Eg using registryUrls as they're intended :)

[nbraem]

Any progress on this? We are also stuck because of this issue.

We're using WhiteSource Renovate App with Github. One private repo has another one of our private repo's as a submodule, using git ssh. Renovate app is configured for both repos. Updating dependencies fails. The log says:

WARN: Unable to initialise git submodule at [path to submodule]

[nejch]

I think the idea with credential helpers and focusing on http-based submodules is a good direction, and you can actually use git's insteadOf to convert SSH/git protocol URLs into HTTP(S) URLs, as well as to inject the PAT into the base URL. So this way you could (for most instances) solve both issues, I think.

You are already doing something similar: https://github.com/renovatebot/renovate/blob/75737805c804f69f427e0f8147c463836b1af448/lib/manager/gomod/artifacts.ts#L21

I use insteadOf a lot with GitLab a lot to "normalize" submodule URL's so that I only need to use the CI_JOB_TOKEN for any kind of private submodule on the same instance. Let me know if I can help here. Here's my example of pre-build steps, but there are different ways of achieving this:

git config --global credential.helper store
echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.example.com" > ~/.git-credentials
git config --global url."https://[email protected]".insteadOf "ssh://[email protected]"
git config --global url."https://[email protected]/".insteadOf "[email protected]:"
git submodule update --init --recursive

See also https://docs.gitlab.com/ee/user/project/#authenticate-git-fetches.

Obviously you might need PATs instead of CI tokens and the format may vary between providers it seems. The only other issue is getting the SSH/git/https URLs of instances where they use non-standard SSH ports/HTTP endpoints served in subdirectories, but this can be added to hostRules or maybe even available via the API.

Edit: also if people want the opposite, to use SSH instead, you could do the reverse I guess.

Edit2: you can also specify a custom path for the credential store if that's relevant. Not sure if this helps with some of the discussions above, I previously kind of assumed it was isolated in a container. But see https://git-scm.com/docs/git-credential-store.

[rarkins]

Right now I don't think we really support ssh, but not sure if anyone's running a custom setup where they're used. Ideally we could simplify renovate by assuming we can rewrite all Github/gitlab dependencies to use the https token format.