Azure-pipelines-agent: Using Git submodules doesn't work with private repository

Created on 6 Sep 2016 · 17Comments · Source: microsoft/azure-pipelines-agent

Agent version and platform

2.105.5
Windows

VSTS type and version

Team Services

What's not working?

Git submodules are not working when using GitHub private repositories.

Agent and Worker's diag log

******************************************************************************
Starting: Build
******************************************************************************
Current agent version: '2.105.5'
******************************************************************************
Starting: Get Sources
******************************************************************************
Syncing repository: user666/MyRepo (GitHub)
Prepending Path environment variable with directory containing 'git.exe'.
git version
git config --get remote.origin.url
git clean -fdx
git reset --hard HEAD
HEAD is now at cb830b0 Fix merge
git submodule foreach git clean -fdx
git submodule foreach git reset --hard HEAD
git config gc.auto 0
git config --get-all http.https://github.com/user666/MyRepo.git.extraheader
git remote set-url origin https://********:********@github.com/user666/MyRepo.git
git remote set-url --push origin https://********:********@github.com/user666/MyRepo.git
git fetch --tags --prune --progress origin
git checkout --progress --force cb830b0c893e1fc47c401f198b0ccd51419b96b4
HEAD is now at cb830b0... Fix merge
git submodule init
git submodule update -f
Cloning into 'C:/agent/_work/2/s/Core'...
bash: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Username for 'https://github.com': Invalid argument
fatal: clone of 'https://github.com/user666/MyRepoCore.git' into submodule path 'C:/agent/_work/2/s/Core' failed
Git submodule update failed with exit code: 1

Source

alxbog

Most helpful comment

Worked. Thank you very much!

For others that might come across the same issue.

.gitmodules looks like this now

[submodule "themes/my-hugo-future-imperfect"]
    path = themes/my-hugo-future-imperfect
    url = ../my-hugo-future-imperfect.git

I just edited the file.

Sarafian on 12 Oct 2016

👍5 👀1 🎉1

All 17 comments

Current workaround:
Install latest Git on Windows agent. Set system.prefergitfrompath to true in build definition variables.

alxbog on 6 Sep 2016

👍1

@alxbog what's version of git.exe did you installed? is your submodule public repository?

TingluoHuang on 6 Sep 2016

@TingluoHuang

System Git version is 2.10.0
VSTS agent embedded Git version from externals folder is 2.9.3

Submodule is a private repository. There's no issues when using public ones.

alxbog on 6 Sep 2016

@alxbog #619 will fix this issue, and we will release a newer version agent with the fix.

TingluoHuang on 1 Oct 2016

619 is merged. Look for it in 2.107.1 agent this next week

bryanmacfarlane on 2 Oct 2016

I have a very similar issue where same unmodified for months project and repository suddenly fails with
Git submodule update failed with exit code: 1

My setup is a private bitbucket repository that has a submodule for a private bitbucket repository.

Hosted Agent successfully assigned.
Configuring connection settings for Hosted Agent
Starting Hosted Agent
Initializing connection to Hosted Agent
******************************************************************************
Starting: Build
******************************************************************************
Current agent version: '2.107.0'
Downloading task: PowerShell
******************************************************************************
Starting: Get Sources
******************************************************************************
Syncing repository: bitbucket.org - sarafian.github.io (Git)
Prepending Path environment variable with directory containing 'git.exe'.
...

git submodule init
Submodule 'themes/my-hugo-future-imperfect' (https://********@bitbucket.org/********/my-hugo-future-imperfect.git) registered for path 'themes/my-hugo-future-imperfect'
git submodule update -f
Cloning into 'C:/a/1/s/themes/my-hugo-future-imperfect'...
bash: /dev/tty: No such device or address
error: failed to execute prompt script (exit code 1)
fatal: could not read Password for 'https://********@bitbucket.org': Invalid argument
fatal: clone of 'https://********@bitbucket.org/********/my-hugo-future-imperfect.git' into submodule path 'C:/a/1/s/themes/my-hugo-future-imperfect' failed
Git submodule update failed with exit code: 1

Last successful build was on 15/9. I can't provide a detailed log compatison, because it's not available by VSTS. Seems like VSTS is cleaning up high level log messages

I think you need to re-open or create new issue.

Sarafian on 11 Oct 2016

@Sarafian can you share your .gitmodules file from your main repository?
We switch to use git.exe instead of libgit2sharp in the build agent, that causing the break. i can work with you to help you unblock.

TingluoHuang on 11 Oct 2016

@Sarafian
https://www.visualstudio.com/sl-si/docs/build/define/repository#what-kinds-of-submodules-can-i-check-out

TingluoHuang on 11 Oct 2016

@TingluoHuang

[submodule "themes/my-hugo-future-imperfect"]
    path = themes/my-hugo-future-imperfect
    url = https://[email protected]/asarafian/my-hugo-future-imperfect.git

If I understand correctly the url is causing the problem because it is an absolut url, when it should be a relative one? If the repository url is https://[email protected]/asarafian/sarafian.github.io.git then based on the above link what should be the url? How does it know how to access a relative path outside the repository I don't understand. I always thought that when adding the submodule with relative path's , it would use the relative path to land on git repository and then extract the repository's full url as per my case.

Sarafian on 11 Oct 2016

@Sarafian
What is your main repository url?
ex: if you main repository url is https://[email protected]/asarafian/sarafian.github.io.git
then: git submodule add "themes/my-hugo-future-imperfect" "../my-hugo-future-imperfect.git"

TingluoHuang on 11 Oct 2016

Worked. Thank you very much!

For others that might come across the same issue.

.gitmodules looks like this now

[submodule "themes/my-hugo-future-imperfect"]
    path = themes/my-hugo-future-imperfect
    url = ../my-hugo-future-imperfect.git

I just edited the file.

Sarafian on 12 Oct 2016

👍5 👀1 🎉1

I'm not able to build a private git repository containing two private git submodules.

Getting following output when the build agent trying to sync the submodules in the Get Sources task:

2019-01-14T07:50:35.3695612Z ##[command]git submodule sync --recursive
2019-01-14T07:50:36.8283300Z ##[command]git -c http.https://github.com.extraheader="AUTHORIZATION: basic ***" submodule update --init --force --recursive
2019-01-14T07:50:37.2194171Z Submodule 'Payments.KlarnaCheckout' (https://github.com/majako/nopcommerce-klarna-checkout.git) registered for path 'Payments.KlarnaCheckout'
2019-01-14T07:50:37.2195979Z Submodule 'Plugins/Majako.Plugin.Misc.ElasticSearch' (https://github.com/majako/nopcommerce-elasticsearch.git) registered for path 'Plugins/Majako.Plugin.Misc.ElasticSearch'
2019-01-14T07:50:37.7340675Z Cloning into 'D:/a/1/s/Payments.KlarnaCheckout'...
2019-01-14T07:50:37.7342372Z remote: Repository not found.
2019-01-14T07:50:37.7342707Z fatal: repository 'https://github.com/majako/nopcommerce-klarna-checkout.git/' not found
2019-01-14T07:50:37.7342989Z fatal: clone of 'https://github.com/majako/nopcommerce-klarna-checkout.git' into submodule path 'D:/a/1/s/Payments.KlarnaCheckout' failed
2019-01-14T07:50:37.7343524Z Failed to clone 'Payments.KlarnaCheckout'. Retry scheduled
2019-01-14T07:50:38.1444508Z Cloning into 'D:/a/1/s/Plugins/Majako.Plugin.Misc.ElasticSearch'...
2019-01-14T07:50:38.1446156Z remote: Repository not found.
2019-01-14T07:50:38.1446480Z fatal: repository 'https://github.com/majako/nopcommerce-elasticsearch.git/' not found
2019-01-14T07:50:38.1446774Z fatal: clone of 'https://github.com/majako/nopcommerce-elasticsearch.git' into submodule path 'D:/a/1/s/Plugins/Majako.Plugin.Misc.ElasticSearch' failed
2019-01-14T07:50:38.1447154Z Failed to clone 'Plugins/Majako.Plugin.Misc.ElasticSearch'. Retry scheduled
2019-01-14T07:50:38.4568329Z Cloning into 'D:/a/1/s/Payments.KlarnaCheckout'...
2019-01-14T07:50:38.4569756Z remote: Repository not found.
2019-01-14T07:50:38.4570115Z fatal: repository 'https://github.com/majako/nopcommerce-klarna-checkout.git/' not found
2019-01-14T07:50:38.4600648Z fatal: clone of 'https://github.com/majako/nopcommerce-klarna-checkout.git' into submodule path 'D:/a/1/s/Payments.KlarnaCheckout' failed
2019-01-14T07:50:38.4600893Z Failed to clone 'Payments.KlarnaCheckout' a second time, aborting
2019-01-14T07:50:38.4760766Z ##[error]Git submodule update failed with exit code: 1
2019-01-14T07:50:38.5144031Z ##[section]Finishing: Get sources

The URLs are correct.

My .gitsubmodule file:

[submodule "Plugins/Majako.Plugin.Misc.ElasticSearch"]
    path = Plugins/Majako.Plugin.Misc.ElasticSearch
    url = ../nopcommerce-elasticsearch.git
[submodule "Payments.KlarnaCheckout"]
    path = Payments.KlarnaCheckout
    url = ../nopcommerce-klarna-checkout.git
    branch = v4.0-1.0.1

How would I solve this?

martingust on 14 Jan 2019

@martingust does your submodules use same credential as your main module?

TingluoHuang on 14 Jan 2019

@TingluoHuang Yes, they do.

martingust on 14 Jan 2019

@TingluoHuang I just realized it is the Azure Pipeline user that clones the repositories. I added the app to my submodules as well and now it works fine. Thanks!

martingust on 14 Jan 2019

@martingust cool, glad you unblock

TingluoHuang on 14 Jan 2019

@martingust what do you mean you added the app to your submodules? I'm facing a similar problem and I'm stuck here.

[EDIT]
Nevermind - I just didn't know how to set the relative path to my submodule repositories. I had to change the url in the .gitmodules file to url = ../submodulerepositoryname