Azure-devops-docs: Documentation of submodules from other projects is insufficient

[jonstelly]

https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/azure-repos-git?view=azure-devops#authorize-access-to-your-repositories

I have 2 private azure devops projects, A and B. B has a submodule that points to a repository that's in project A. I've been digging around in the VSTS UI for over half an hour and I don't see how to do this:

> If the repository is in a different project than your pipeline, you must have read access to the project and repository.

---

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

• ID: 8b846aea-82f7-a988-67d8-1b5e74b61ed2
• Version Independent ID: 2d632729-bb33-c0a0-c996-e1d8e86c2e23
• Content: Build Azure Repos Git repositories - Azure Pipelines
• Content Source: docs/pipelines/repos/azure-repos-git.md
• Product: devops
• GitHub Login: @steved0x
• Microsoft Alias: sdanie

[jonstelly]

I now think I see part of the issue. I'm trying to get this working on a YAML build which means the 'Options' tab in the pipeline edit UI is not present. In the UI it seems that there's an option to set the authorization scope to Project Collection:

But I don't see any documentation of the yaml schema that lets me configure that setting. Is that the problem I'm running into or is it something else?

[jonstelly]

Another update:

Auth Scope and Yaml builds aren't the issue. Even moving the submodule repository to the same project didn't fix it. It seems that submodule checkouts are just broken right now (is it due to the vsts -> azure devops move and organization names now showing up as the user in the git url?)

i.e. https://organization.visualstudio.com/ turned into https://[email protected]/

I've now got 2 repositories in the same Azure Devops project, One has a submodule pointing to the other using HTTPS and the Get Sources step is still failing with:

Submodule 'myorg-framework' (https://[email protected]/myorg/myorg/_git/myorg-framework) registered for path 'myorg-framework'
Cloning into 'D:/a/1/s/myorg-framework'...
fatal: could not read Password for 'https://[email protected]': terminal prompts disabled
fatal: clone of 'https://[email protected]/myorg/myorg/_git/myorg-framework' into submodule path 'D:/a/1/s/myorg-framework' failed
Failed to clone 'myorg-framework'. Retry scheduled
Cloning into 'D:/a/1/s/myorg-framework'...
fatal: could not read Password for 'https://[email protected]': terminal prompts disabled
fatal: clone of 'https://[email protected]/myorg/myorg/_git/myorg-framework' into submodule path 'D:/a/1/s/myorg-framework' failed
Failed to clone 'myorg-framework' a second time, aborting

This is immensely frustrating, I found a developer community post about this but it seems a bit different, I'm not sure if it'll work.

[jonstelly]

For anyone else running into this issue, submodule checkout for private azure devops git repositories just doesn't work right now. I tried opening a ticket with Microsoft support to get this issue elevated to the product team but they suggested that I submit an issue on developer community and there are already a handful of issues out there starting in November.

As a workaround I moved my code to github private repositories and the github source code connector works for private submodules.

[BrandonCookeDev]

+1

[planetmarshall]

I am having the same issue but for git-lfs data rather than submodules.

[BradAF]

+1

[steved0x]

I'm actively tracking this down and hope to have an update soon.

[steved0x]

Just on the off chance that anyone hasn't seen this it, there is this info here: Pipeline options for Git - Checkout submodules

I am still checking this but wanted to post this info.

[jonstelly]

I hadn't seen that relative submodule path information but at some point I did have both repositories in a single Project and it still didn't work. So do submodules from the same project also have to be relative URLs?

And to understand the relative path for the submodules, does the effective directory structure mirror the git repository url:

https://[email protected]/ORG/PROJECTA/_git/PROJECTA
https://[email protected]/ORG/PROJECTB/_git/PROJECTB

So if I wanted to add A as a submodule to B, I'd do this?

git submodule add ../../../PROJECTA/_git/PROJECTA

From the documentation I'm not clear if the _git part of the URL needs to be used and if the PROJECTA/B at the end is considered a file or directory for .. processing, and the example doesn't really feel like a real-world example and also doesn't explain the project/repository structure. If you could beef up the authenticated submodule relative path example, maybe have one example with a submodule from the same project, and another from a different project, that would be really helpful.

Also, I had to go look it up and it looks like git supports relative urls for submodules but I didn't know until seeing your link that was even a valid thing. I'm not a git expert but I'm pretty proficient and I'd be surprised to learn that more than 10% of the git userbase knows they can use relative URLs for submodules. My point is just that it would be nice if the much more common case (absolute URLs) could be supported too.

Also, thanks for digging into this!

[BradAF]

I'm not sure if this is the same issue, but I found that checking out submodules worked fine on private DevOps repositories using Microsoft's hosted pools, but when I used a custom deploy agent, the version of git on my image was too old and produced the exact error messages as above (git 2.7.4). Upgrading to git 2.21.0 fixed it and allowed me to checkout submodules in a build pipeline from a private repo. I had to make sure [email protected] was defined in the .gitmodule URL though.

[code2mars]

@BradAF i am trying the same by adding a sub module using relative path but i am getting not found error. If you could can share the sample .gitmodule file i just wanted to see how to specify the url. Because the documentation says i had use the relative paths but not sure about the syntax of using both URL and relative path when configuring sub module.

[steved0x]

We have updated this topic with more info on submodules - this is what we have right now, hopefully this extra info helps.

[noseratio]

I also experienced this issue fatal: could not read Password for 'https://[email protected]': terminal prompts disabled. Indeed, it got resolved when I changed the submodule URL inside .gitmodules to be a relative URL to the main repo (in my case, all repos belong to the same Azure DevOps project). I.e.:

[submodule "repo1/repo2_as_submodule"]
    path = repo1/repo2_as_submodule
    url = ../repo2

[BradAF]

I apologize for the delay - I am no longer using submodules for my project and I'm not sure if this is still helpful but here's what I had in my .gitmodule that worked fine on git v2.21.0:

[submodule "modules"]
    path = modules
    url = https://[email protected]/myOrg/myProject/_git/modules
    branch = master

EDIT: forgot to use 'code' tag

[AnorZaken]

Had the same problem on git 2.7.x
Upgraded to git 2.24
Made a new repo, imported submodules with relative url from the start (instead of manually editing the .gitmodules file).
The most important point is that the .gitmodules file looks exactly the same - but it magically works now when done with a newer git version.
Honestly I have no idea why it works, .gitmodules looking the same and all, but I figured it might help people to know...

[SharbelWired]

I'm not sure if this is the same issue, but I found that checking out submodules worked fine on private DevOps repositories using Microsoft's hosted pools, but when I used a custom deploy agent, the version of git on my image was too old and produced the exact error messages as above (git 2.7.4). Upgrading to git 2.21.0 fixed it and allowed me to checkout submodules in a build pipeline from a private repo. I had to make sure [email protected] was defined in the .gitmodule URL though.

Dude, thank-you so much for this hint! I was scratching my head wondering why this was giving me so much grief! Updating my git client on our on prem Ubuntu agent solved the issues we were having with regarding to submodule checkout in build pipeline.

Thanks again!