Go: cmd/go: add support for Azure DevOps Git repos to go get

[MrWako]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.11 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

GOHOSTARCH=amd64
GOHOSTOS=windows
GOOS=windows

What did you do?

I created a few public Azure DevOps repos to illustrate the problems

• https://dev.azure.com/markawakefield/GoGetTest/_git/app
• https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
• https://dev.azure.com/markawakefield/GoGetTest/_git/stringutils

Then when trying to get the package with the main function I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/app
Fetching https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1
Parsing meta tags from https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1 (status code 200)
package dev.azure.com/markawakefield/GoGetTest/_git/app: unrecognized import path "dev.azure.com/markawakefield/GoGetTest/_git/app" (parse https://dev.azure.com/markawakefield/GoGetTest/_git/app?go-get=1: no go-import meta tags ())

I think because go get does not know its a Git repo, and there are no meta tags supplied to indicate that it is. If I try to just get one of the other packages with a .git extension I get:

go get -v dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git
# cd .; git ls-remote https://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
Logon failed, use ctrl+c to cancel basic credential prompt.
fatal: could not read Username for 'https://dev.azure.com': terminal prompts disabled
# cd .; git ls-remote git+ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
# cd .; git ls-remote ssh://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt
ssh: connect to host dev.azure.com port 22: Connection timed out
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
package dev.azure.com/markawakefield/GoGetTest/_git/stringfmt.git: cannot download, git://dev.azure.com/markawakefield/GoGetTest/_git/stringfmt uses insecure protocol

I think because the .git extension on the URL is not being passed to the git commands, and therefore it hits an authenticated endpoint.

What did you expect to see?

I expected (hoped) to see the packages add to the GOPATH under the dev.azure.com directory.

What did you see instead?

No files added to the GOPATH

I appreciate that this is not a pure issue with the current Go tool chain, more a miss-match between the conventions of Go and Azure DevOps, and has been noted elsewhere including http://ppanyukov.github.io/2017/02/01/golang-with-vsts-repos.html. Please let me know if a proposal is more appropriate.

For other cloud providers (IBM/Apache/OpenStack) there is logic to accomodate these differences and for Azure DevOps its actually a little worse than just the go get issue as even if this command completed the package ends up under a _git directory and so will be ignored in some instances due to the underscore.

Until now we have used Glide for package management which has allowed us to work around the issue but we would now like to adopt versioned packages so the issue is becoming more pressing.

I'll open a PR with a small revision addressing the issue that can be discussed/reviewed.

Thanks

[gopherbot]

Change https://golang.org/cl/142678 mentions this issue: cmd/go/internal/get: fix go get for Azure DevOps Git repos

[rsc]

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

[rsc]

Note that GitHub generates the \ tags that the go command needs, so that GitHub-hosted code works with go get regardless of domain (e.g., GitHub Enterprise, although github.com has the tags too). At this point it would be nice if hosting sites like dev.azure.com would put the tags in and therefore own their meaning, instead of making the go project source code the source of truth. (That made sense early on when no one had heard of Go and not many sites cared.)

[MrWako]

thanks @rsc - meta tags on the Azure side is definitely another solution route, although due to the .../_git/... in the URLs, it would be more involved than MS just adding these to the existing address. Azure could, or I could, host vanity URLs with the meta tags pointing to the existing repo URLs but this is starting to get more involved and I'd really appreciate a simpler solution.

[MrWako]

Just adding the links to the issues on the Azure DevOps side:

• https://visualstudio.uservoice.com/forums/330519-visual-studio-team-services/suggestions/34888885-support-repos-for-golang-by-providing-a-url-alias
• https://visualstudio.uservoice.com/forums/330519-azure-devops-formerly-visual-studio-team-services/suggestions/20364613-add-golang-support-to-vsts-git

to indicate that the mismatch between Azure DevOps and Go is a real problem for developers that have no choice but to use Azure DevOps, but want to champion Go.

[FiloSottile]

@FiloSottile, it would be nice to say what the decision is when you add the NeedsDecision label.

Will do. In this instance I'd like an owner of cmd/go to decide if we are still expanding the special case mappings, and if so what transformations should be applied to this one, as it seems to be the only one that magically adds a component to the path.

For the record, I do believe adding <meta> tags at the desired path would serve Azure DevOps best, providing them with more control, flexibility (no issues with adding path components at the right spot) and probably compatibility as there might be tools that extracted the mappings from an existing version of Go (although they should use the go tool instead).

[MrWako]

@bketelsen perhaps you could add your experience, thanks.

[crsdrw]

@rsc Whilst I think it would be a good idea for dev.azure.com to add tags, even if they did, I don't think that would solve the problem. As @MrWako says, because dev.azure.com repository paths have a .../_git/... in them go get results in folders that start with an underscore. One solution to this problem would be to "fix" the go tooling like go list so that it can handle folders that start with an underscore but it seems like an easier solution is to change go get. With the added benefit of getting cleaner import paths for dev.azure.com repositories.

It might be a hard sell to convince MS to host vanity urls for dev.azure.com repositories because of the way go tooling handles folders that start with an underscore, although we are trying.

[FiloSottile]

To be clear I'm not suggesting having import paths with _git in them, but serving meta tags from https://dev.azure.com/markawakefield/GoGetTest/app[/*] pointing to https://dev.azure.com/markawakefield/GoGetTest/_git/app.

[crsdrw]

@FiloSottile I doubt MS will be able to serve pages at those exact urls as they might conflict with other pages, for example there already pages like https://dev.azure.com/markawakefield/GoGetTest/_workitems, https://dev.azure.com/markawakefield/GoGetTest/_build.

[FiloSottile]

If they all start with _ they shouldn't ever conflict with Go import paths. Also, the meta tag is not user visible, so it can be served on any page, even unrelated, to describe the import path to repository mapping.

Anyway, I'm not the relevant owner, I was just suggesting that meta tags would be a solution that MS can control, deploy independently, and it would be backwards compatible instead of only working with Go 1.12 and with tools that didn't hardcode the existing set.

[rsc]

What is the name space under (say) dev.azure.com/markawakefield/GoGetTest? What kinds of names can appear after the underscore?

[rsc]

Unless Go users really want to see _git in their import paths, the fix here should really be on dev.azure.com's side.

[MrWako]

Hi @rsc names that can appear after the underscore are basically all the kinds that Azure has for its task tracking/development process, so things like _workitems (for you assigned workitems), _sprints (for sprint related views), _boards (for kanban-ish views), etc.

[MrWako]

Thanks all, this has been a healthy discussion, how do we try and get to a decision?

• if the PR was to be accepted thats great and would be an easy way forward for Azure Go programmers
• if not, then I well understand the arguments and at least we've explored and discounted the option. Either Azure can pick up the challenge or communicate that its up to developers to address.

Coming to a decision is probably nearly as important than what the decision is so that I can make plans what to do next. Thanks,

[crsdrw]

@rsc, @FiloSottile I think perhaps for historical reasons, they also serve pages (or redirect) if you put the name of a project team after the name of the project. For example: dev.azure.com/crsdrw/GoGetTest/BackendTeam/_git/app

[kaylangan]

I'm a Program Manager for Azure DevOps. We've had this request several times over the past few years and I'd love if we could come to a solution for our Go users.

While meta tags don't exist for Azure DevOps, adding those alone will not solve the problem. As others have noted, the crux of the issue is the presence of "_git" in the URLs. We use "_git" as a marker for routing, along with other markers like "_workitems", _dashboards", etc. Our Git URLs specify organization, project, and repo, but have also historically included team name with some of those components being optional (or omitted) if they are identical. Our Git URLs also require "_git" (except for SSH, but I believe that go git doesn't support SSH?).

What is the name space under (say) dev.azure.com/markawakefield/GoGetTest? What kinds of names can appear after the underscore?

@rsc There is a fixed set of names that can come after an underscore (e.g. _git, _workitems, _builds); project names cannot start with an underscore.

To be clear I'm not suggesting having import paths with _git in them, but serving meta tags from https://dev.azure.com/markawakefield/GoGetTest/app[/*] pointing to https://dev.azure.com/markawakefield/GoGetTest/_git/app.

@FiloSottile Unfortunately, this won't work. This scheme matches https://dev.azure.com/{organization}/{project}/{team-name} and redirects to the dashboards directory page: https://dev.azure.com/{organization}/{project}/_dashboards/directory?teamId={team-name}. We've looked at other formats but we run into collisions without "_git".

It sounds like the hesitation is around adding another special case mapping to cmd/go? Is there more information that I could provide that might make a decision on this easier?

[FiloSottile]

This scheme matches https://dev.azure.com/{organization}/{project}/{team-name} and redirects to the dashboards directory page: https://dev.azure.com/{organization}/{project}/_dashboards/directory?teamId={team-name}. We've looked at other formats but we run into collisions without "_git".

I'm not sure I understand. You're in control of the meta tags, can't you just not serve a tag at https://dev.azure.com/{organization}/{project}/_dashboards/...?

If there is no way to map a space without underscores to the correct repositories, then doing the mapping in cmd/go won't help, and you need underscores in import paths, which will break many things we don't have control over. If there is a way to do such a mapping, you can do it by serving meta tags just as well as we can do it in cmd/go, and it will be more backwards compatible, better serving our shared users.

[rsc]

Sorry but it's too late now for Go 1.12. Moving to the Go 1.13 queue and marking as a proposal.

[bcmills]

The original report and associated blog post note that one of the blockers to using <meta> tags is that those endpoints require authentication. Note that we now have a design for HTTPS authentication for Go 1.13: https://github.com/golang/go/issues/26232#issuecomment-461525141.

[bcmills]

It's not clear to me why there would be an issue with redirects: why can't the server either suppress the redirect when it sees the go-get=1 query parameter, or serve the meta tags on the redirected page for all of the import paths that redirect to it?

[gopherbot]

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

[m1o1]

Any update here? I can't seem to figure out how to get Go modules working in Azure DevOps.

With glide, I was able to specify something like:

- package: {organization}.visualstudio.com/{project}/mypackage
  version: v1.1.1
  repo: https://{organization}.visualstudio.com/{project}/_git/mypackage
  vcs: git
  subpackages:
  - anotherpackage
  - some/other/subpackage

This, combined with setting

git config --global url.https://${GIT_ACCESS_TOKEN}@{organization}.visualstudio.com.insteadOf https://{organization}.visualstudio.com;

worked.

However, we wish to switch to Go modules, which doesn't seem to have an equivalent setting (notice how the package name is valid without an underscore in the glide config - the repo field has the underscore). I tried using a replace directive but got this (even when supplying a version):

replacement module without version must be directory path

Playing around with it for a while, I'm not sure how a go.mod file should look to get the same functionality (if it's possible).

[crsdrw]

@adinunzio84 The only workaround I am aware of is to host your own vanity URL service so that you can define your own import path. See here for some example source code. But it seems a sad state of affairs.

[bcmills]

@adinunzio84, this issue timed out awaiting a reply to the question in https://github.com/golang/go/issues/28236#issuecomment-462011842.

We do not plan to add another special case without further evidence for why the general mechanism (go-import tags) cannot be used here.

[brkerr]

Did anything change on the Go side for the repro in the issue description? I can run "go get" with the urls to your public repos and as long as I append ".git" to the end it works for me.

Also what are the specific issues with having the underscore in the path? After I go get one of your repos I can import it in another go file and that works as well.

[m1o1]

@brkerr I cannot get that to work. Do other commands work for you if you do not do "go get" first? With other repositories, go commands work without having to "go get" first (ex. running "go build" will find any missing dependencies - even if they're missing from the "go.mod" file - automatically when scanning the source. Also, "go mod download" should find them too if they're in your "go.mod" file)

[brkerr]

I'll try that out. I do know that if I make 2 public repos and have a go file in one import a go file in another, if I go get the one it correctly fetches the other.

I'm also working on adding go-import tags to repo urls in Azure DevOps and looking at a solution without underscores in them....just trying to understand what the actual issue is with the underscore.

[m1o1]

@brkerr Actually, I just tried it out with my go-imports vanity URL service (which provides the go-import tag pointing requests originally going to /{account}/{project}/_git/{repo} to the correct place in my Azure repos), and I was able to use underscores in import paths too, provided the file itself doesn't start with an underscore (the package and its directory name can though).

I still had to do "go get -insecure" first though, since my vanity URL service is not secure. So maybe it's the other commands that would otherwise go find the dependencies that would break. I'm now curious what would break with underscores too.

[brkerr]

Can you show me what kinds of mappings you have from the _git repo url to the correct place? How does go handle importing nested packages in a git repo? If I wanted to use a package in a nested folder from git, would I "go get" just the repo root first and then import using the nested structure, or would I do that all in one import /{account}/{project}/_git/{repo}/{nested}/{packagename}?

[m1o1]

I have a private sample repo - I assume if you're at Microsoft you can access it, but if not I'll need to find a way to send it to you privately. This demos everything I have to get it working, but the only thing I don't have working yet is submodules (or multi-module repos) (it works if I use a replace directive and point it to the relative path to it, but for realistically testing it (as if it were not a "submodule" of the module I made), it should not use that.

I think the best description is in Module-aware go get. But it sounds like what you're asking is when the go module directory is not at the root of the repository, in which we run into the same issue I mentioned above.

[brkerr]

No we don't have access to random private repos :smile: I think I better understand the problem now and I have a potential solution, it just might take a few weeks to get it released.

[brkerr]

It looks like we will have at least step 1 of the fix in for the next deployment - import tags will be served from the repo root url as well as {repo root}/{other subdirs}/{etc}. Note that these urls will still have _git in them as that was a slightly bigger change that impacted more than just my area. When I understand more about what problems are associated with underscores I can work on that part (I haven't found any issues in testing with public repos) but that should be a much easier fix now that this first one is in place.

[brkerr]

@adinunzio84 This has started rolling out, although it will take about a week to get everywhere. If you don't mind helping me verify what's fixed and what still needs fixing, can you (or anyone else in this thread) email me at my github alias [at]microsoft sometime next week?

[m1o1]

@brkerr absolutely! Thanks for the work you've done on this. Can you give a sample import path (just the syntax/format)? Specifically will the import paths use the <org>.visualstudio.com/<project>/_git/<repo> format or the dev.azure.com/<user>/<project>/_git/<repo> format (or support both)? By the previous comments, it looks like it will use the dev one

[brkerr]

@adinunzio84 For the host format, most of the testing was done with dev.azure.com/<account>/ but routing should work the same with <account>.visualstudio.com/.

For the repo format, we support all existing web repo paths:
.../<project>/<team>/_git/<repo>
.../<project>/_git/<repo>
.../_git/<project> <-- this one is for when project and repo are the same string

And special for go we support subfolders on those paths. Example full path:
go get dev.azure.com/testProject/_git/testRepo/subFolder/packageFolder

If you email me your account name I can let you know specifically when the deployment reaches that account.

[brkerr]

@adinunzio84 It's about halfway deployed at this point

[sebnyberg]

I may be missing something here but I can't get this to work for private repositories over HTTPS.

Since the SSH url does not include the _git segment, I've found that it's easiest to redirect HTTPS -> SSH via git config.

# redirect https -> ssh
git config --global url."[email protected]:v3".insteadOf https://dev.azure.com

# trust keys from ssh.dev.azure.com
ssh-keyscan ssh.dev.azure.com >> ~/.ssh/known_hosts

I initialize my go module with:

go mod init dev.azure.com/{org}/{project}/{repo}.git

Then import (don't forget to add .git at the end):

import (
  "dev.azure.com/{org}/{project}/{repo}.git/path/to/package"
)

For more info check this comprehensive post on different options: https://smartystreets.com/blog/2018/09/private-dependencies-in-docker-and-go

[ablomber]

@brkerr Thank you for working in this feature. I'm coming from the itron account and would like to try testing this out, it will allow me to pull out a bunch of hacky workarounds in my repos for fetching go code from our azure devops repos.

I will give it a trial run and send you feedback once I do.

Best regards,
-Adam

[brkerr]

@sebnyberg I haven't tried via ssh but I had success with private repos by creating a PAT and adding something like this to the gitconfig:

[url "https://<user>:<PAT>@dev.azure.com/<account>/_git/<repo>"] insteadOf = https://dev.azure.com/<account>/_git/<repo>

[psandhu79]

@brkerr do you have any tips on how you can use your suggestion in CI/CD pipeline

[sikemullivan]

Microsoft documented how to get this to work. Thought I would share.

https://docs.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops

[m1o1]

I've found that the git config can work as a prefix too, so you don't need to make one for each repo, just one for each org (as they use different PATs):

[url "https://<org>:<TOKEN>@dev.azure.com/<org>"]
        insteadOf = https://dev.azure.com/<org>

[sikemullivan]

Are you using DevOps pipelines? I'm trying to run a build (Ubuntu Host) and use bash to set the gitconfig with the build's access token. For some reason, it is not working the same. I ran out of time trying to figure it out last night, but I was wondering if git or go fetches the repo differently on ubuntu.

[ablomber]

Hi Michael,

I am using Azure DevOps pipelnes, and presently have a very ugly hack in place to work around the original problem.

Now that this gitconfig solution has been offered, I will attempt to put it in place and will let you know what results I have with it.

Thanks!

-Adam

From: Michael Sullivan [e-mail]
Sent: Wednesday, July 17, 2019 12:00 PM
To: golang/go [e-mail]
Cc: Blomberg, Adam [e-mail]; Comment [e-mail]
Subject: [EXTERNAL] Re: [golang/go] cmd/go: add support for Azure DevOps Git repos to go get (#28236)

Are you using DevOps pipelines? I'm trying to run a build (Ubuntu Host) and use bash to set the gitconfig with the access token. For some reason, it is not working the same. I ran out of time trying to figure it out last night.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_golang_go_issues_28236-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DAMK3FT6CW2ILNJOZ2C56AITP746YBA5CNFSM4F37OOP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2E3VLA-23issuecomment-2D512342700&d=DwMCaQ&c=pqcuzKEN_84c78MOSc5_fw&r=iS-Hk51gmqz0ak8Lvc-WNkKlq81spWMXKffOaGXFz3c&m=cBtp8A8H-ZJKTjsQrPBLLy26-hFRxPebFonpSqvE8eQ&s=aywySDrFJNoOBQsXl6CNXx45xVwuLgvOIti04_x1b1c&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMK3FTZLDNZ7YOMNPCBPDHLP746YBANCNFSM4F37OOPQ&d=DwMCaQ&c=pqcuzKEN_84c78MOSc5_fw&r=iS-Hk51gmqz0ak8Lvc-WNkKlq81spWMXKffOaGXFz3c&m=cBtp8A8H-ZJKTjsQrPBLLy26-hFRxPebFonpSqvE8eQ&s=9qv66gw67A4j6Fy1L3aPAQ02BPbZTUJ7H0IvYdK-UHw&e=.

[sikemullivan]

Here is what I have so far. I threw the git trace in, but it appeared that it wasn't even requesting anything. My thought was, "go build" will automatically install my private repo modules. I'll be working on this later tonight as well.

Edit: I removed the script task for dep, it was confusing. I'm using modules.

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  GIT_TRACE_PERFORMANCE: true
  GIT_TRACE_SETUP: true
  GOVERSION: '1.12.7'
  GOBIN:  '$(GOPATH)/bin' # Go binaries path
  GOROOT: '$(Agent.BuildDirectory)/go' # Go installation path
  GOPATH: '$(Agent.BuildDirectory)/gopath' # Go workspace path
  GOSRC:  '$(GOPATH)/src'
  ACCESSTOKEN: '$(system.AccessToken)'
  modulePath: '$(GOSRC)/github.com/$(build.repository.name)' # Path to the module's code

steps:
- script: |
    wget "https://storage.googleapis.com/golang/go$(GOVERSION).linux-amd64.tar.gz" --output-document "$(Agent.BuildDirectory)/go$(GOVERSION).tar.gz"
    tar -C '$(Agent.BuildDirectory)' -xzf "$(Agent.BuildDirectory)/go$(GOVERSION).tar.gz"
  displayName: 'Install Go 1.12'
- script: |
    mkdir -p '$(GOBIN)'
    mkdir -p '$(GOPATH)/pkg'
    mkdir -p '$(modulePath)'
    shopt -s extglob
    mv !(gopath) '$(modulePath)'
    echo '##vso[task.prependpath]$(GOBIN)'
    echo '##vso[task.prependpath]$(GOROOT)/bin'
  displayName: 'Set up the Go workspace'

- script: |
    git config --global url."https://anything:$(ACCESSTOKEN)@dev.azure.com/mycompany/myproject/".insteadOf "https://dev.azure.com/mycompany/myproject/"

    env GOOS=freebsd GOARCH=amd64 go build
    cp myapp $(Build.ArtifactStagingDirectory)
    if [[ ! -f myapp ]] ; then
        echo 'File "myapp" was not built, aborting.'
        exit 1
    fi
    cp config.release.json $(Build.ArtifactStagingDirectory)/config.json
    cp -rf templates/ $(Build.ArtifactStagingDirectory)/templates/
    cp -rf static/ $(Build.ArtifactStagingDirectory)/static/
  workingDirectory: '$(modulePath)'
  displayName: 'Build'

- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)' 
    artifactName: 'drop'

[jpreese]

For clarity, has the go-imports meta tag been fully deployed? I'm not seeing the tag in our Azure DevOps repository that hosts a Go project.

[sikemullivan]

So git trace was telling the truth, nothing was happening. I was missing GO111MODULE: on. :man_facepalming: I have two modules that are being pulled down from private DevOps' repositories. :spades::hearts::clubs::diamonds::chicken::hamburger::beers:

This works wonderfully for GO 1.12.7

pool:
  vmImage: 'Ubuntu 16.04'

variables:
  GOVERSION: '1.12.7'
  GOBIN:  '$(GOPATH)/bin' # Go binaries path
  GOROOT: '$(Agent.BuildDirectory)/go' # Go installation path
  GOPATH: '$(Agent.BuildDirectory)/gopath' # Go workspace path
  GOSRC:  '$(GOPATH)/src'
  GO111MODULE: on
  ACCESSTOKEN: '$(system.AccessToken)'
  modulePath: '$(GOSRC)/github.com/$(build.repository.name)' # Path to the module's code

steps:
- script: |
    wget "https://storage.googleapis.com/golang/go$(GOVERSION).linux-amd64.tar.gz" --output-document "$(Agent.BuildDirectory)/go$(GOVERSION).tar.gz"
    tar -C '$(Agent.BuildDirectory)' -xzf "$(Agent.BuildDirectory)/go$(GOVERSION).tar.gz"
  displayName: 'Install Go 1.12'
- script: |
    mkdir -p '$(GOBIN)'
    mkdir -p '$(GOPATH)/pkg'
    mkdir -p '$(modulePath)'
    shopt -s extglob
    mv !(gopath) '$(modulePath)'
    echo '##vso[task.prependpath]$(GOBIN)'
    echo '##vso[task.prependpath]$(GOROOT)/bin'
  displayName: 'Set up the Go workspace'

- script: |
    git config --global url."https://anything:$(ACCESSTOKEN)@dev.azure.com/mycompany/myproject/".insteadOf "https://dev.azure.com/mycompany/myproject/"

    env GOOS=freebsd GOARCH=amd64 go build
    cp myapp $(Build.ArtifactStagingDirectory)
    if [[ ! -f myapp ]] ; then
        echo 'File "myapp" was not built, aborting.'
        exit 1
    fi
    cp config.release.json $(Build.ArtifactStagingDirectory)/config.json
    cp -rf templates/ $(Build.ArtifactStagingDirectory)/templates/
    cp -rf static/ $(Build.ArtifactStagingDirectory)/static/
  workingDirectory: '$(modulePath)'
  displayName: 'Build'

- task: PublishBuildArtifacts@1
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)' 
    artifactName: 'drop'

[brkerr]

Hey guys, sorry I've been on vacation for most of July.

@jpreese The tag has been fully deployed for over a month. If you're looking for the tag to be on the page that you load in your browser you won't see it. We are only adding the meta tag if the "go-get" query param is present and set to 1 (which is what Go tacks on when it makes the request).

@sikemullivan Thanks for working out the yaml, good to hear everything is working.

@adinunzio84 Thanks for the follow-up as well.

FYI The follow-up work to get a cleaner URL for public/private has been put on the backburner for now since it fell behind some higher priority features on the backlog. I'm still hoping to get to this but I don't have an ETA yet.

[jpreese]

@brkerr right, but I'm still not seeing it.

go get -u dev.azure.com/myorg/myproject/_git/myrepo

yields

package dev.azure.com/myorg/myproject/_git/myrepo: unrecognized import path "dev.azure.com/myorg/myproject/_git/myrepo" (parse https://dev.azure.com/myorg/myproject/_git/myrepo?go-get=1: no go-import meta tags ())

Running a curl looks like its just a redirect

```
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 615 100 615 0 0 3455 0 --:--:-- --:--:-- --:--:-- 3455

Object moved to
````

[brkerr]

You'll see the redirect if it is a private project, since we redirect you to sign in. You'll need to follow the PAT and gitconfig approach talked about above.

[jpreese]

ah.. we were trying to avoid having to have PATs in the config of the repository. We'll most likely need to end up going the vanity url server if that is the case then. Thanks!

edit: As a follow up in case it was missed, you do not need a PAT when you include .git in the URL, but that looks all sorts of wonky.

[brkerr]

You'll need a PAT unless git had some sort of credential saved on the computer you're testing it for. Unfortunately private repos will always require some sort of authentication, so until Go comes out with their GoAuth feature (planned for the next release) you'll need either the PAT or you can use SSH + SSH key + gitconfig entry

[jpreese]

Yep! From our tests if we use go get with .git in the url (e.g. https://dev.azure.com/myorg/myproject/_git/myproject.git) it will work with a private repo. The Microsoft login window pops up and asks us to login, which is completely acceptable.

The unfortunate part is that including .git in the url makes your binary from go get include ".git" (e.g. myproject.git.exe) and the imports in your project also must have .git if you reference your own modules (e.g. import dev.azure.com/myorg/myproject/_git/myproject.git/someother/submodule)

If we remove the .git from the url, it doesn't work at all. Which in itself is a little confusing because I would expect the error to say "unauthorized" if a PAT must be in the repository. This flat out says it cannot resolve the repository url.

[brkerr]

Yeah so the goofy part is that Go can see a github or gitlab url and know it's a git repo because that's hard coded in the Go source (the original PR for this issue). For Go to know that an Azure DevOps repo is a git repo you need .git or it won't work. It says it can't resolve because of how Go tries to figure out what kind of repo the url is - it will query the url with "go-get" in the params and look for the meta tag, which will hit the redirect and decide that it can't figure it out.

[jpreese]

Right, I thought the PR was to add the meta tag to the Azure DevOps git page so .git wasn't needed. Maybe I misunderstood.

Adding a go-import meta tag to AzDO should fix that.

[brkerr]

Unfortunately private repos will always require some auth workaround since we weren't able to intercept the page before the redirect.

[jpreese]

I see, thanks for the additional information!

[brkerr]

No problem. We want to work on getting "_git" out of the url and merging the original PR for this issue to remove ".git" as well, but those are coming in behind some other higher priority items on the backlog. Eventually we hope to have parity with existing vanity url services.

[prakashpandey]

Can you please provide us estimated timeline when it will be fixed.

[raushan2016]

go get go get -v dev.azure.com/foo/bar/_git/abc.git/pkg/xyz worked perfectly fine with PAT token approach. But when i tried doing
dep ensure -v -add dev.azure.com/foo/bar/_git/abc.git/pkg/xyz it gets suck and eventually fails with the error

ensure Solve(): no valid source could be created: failed to set up sources from the following URLs: https://dev.azure.com/foo/bar/_git/abc.git : remote repository at https://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: ssh://dev.azure.com/foo/bar/_git/abc.git : remote repository at ssh://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: git://dev.azure.com/foo/bar/_git/abc.git : remote repository at git://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128 failed to set up sources from the following URLs: http://dev.azure.com/foo/bar/_git/abc.git : remote repository at http://dev.azure.com/foo/bar/_git/abc.git does not exist, or is inaccessible: : exit status 128

Does dep ensure not use the PAT token url? Any work around for this issue.

[Jonathan34]

If you get a 410 gone error with the PAT solution in .gitconfig:

go: finding dev.azure.com/<org>/<project>/_git/<project>.git latest
go: finding dev.azure.com/<org>/<project>/_git/<project>.git/protos latest
go: downloading dev.azure.com/<org>/<project>/_git/<project>.git v0.0.0-20191007171928-c6a8ebcd1fa5
verifying dev.azure.com/<org>/<project>/_git/<project>[email protected]: dev.azure.com/<org>/<project>/_git/<project>[email protected]: reading https://sum.golang.org/lookup/dev.azure.com/<org>/<project>/_git/<project>[email protected]: 410 Gone

Its reading from sum.golang.org, which does not exist there obviously (it s private).

Setting GOPRIVATE (https://golang.org/cmd/go/#hdr-Module_configuration_for_non_public_modules) to dev.azure.com will download the repo in the mod folder.

[bcmills]

@raushan2016, this issue is about support in go get, not dep ensure.

[MrWako]

Sorry, maybe probably off topic too, but if you are here due to authentification issues sync'ing with Azure DevOps repos with go get, e.g. on a build machine then you can setup a simple git credential helper in your gitconfig file: e.g:

[credential "https://dev.azure.com"]
    helper = "/bin/sh /etc/azure-devops-helper.sh"

with azure-devops-helper.sh something like

#!/bin/sh
echo username=$USERNAME
echo password=$AZURE_DEVOPS_TOKEN

where environment variables containing the username and PAT taken. This proved more reliable for us than switching to SSH for azure repos.

[bcmills]

Reopening due to resumed discussion in #41030.

[artbegolli]

@brkerr Is there any update on the state of affairs here? I think if we can get some clarity on the reason as to why adding the <meta> tags would not be feasible on ADOs end we could justify hardcoding the entry in cmd/go/internal/get. Is the main blocker the auth redirect?

For us the last remaining issue is having to specify .git in our paths.

[vtbassmatt]

:wave: hey folks, I'm the product manager for Azure Repos server. I got clued into this thread because users are now reporting that our documentation for using go get with Azure Repos isn't working anymore, such as https://github.com/MicrosoftDocs/azure-devops-docs/issues/9250.

Brandon added the meta tags back in 2019, and we still serve them. (He's moved on from the product team so I won't bug him here.) I'm hitting our Git protocol endpoint with ?go-get=1 appended to it and see this in the resulting <head>:

<meta name="go-import"
      content="dev.azure.com/mattc-demo/Test1/_git/playground git https://dev.azure.com/mattc-demo/Test1/_git/playground">

Does that look correct for a Git URL https://[email protected]/mattc-demo/Test1/_git/playground?

For private projects, you have to present auth before we'll serve any content other than a redirect, and I probably can't change that. It looks like there was an (abandoned) effort to support our URL formats here. Is anyone willing to resurrect that change and land it?

Alternatively, or in addition, if someone can advise me on what we can fix in that doc, I'd also be happy to make the changes.

Thanks!

[JoseFMP]

@vtbassmatt Thank you.

For us, for private repos is not working.

[artbegolli]

@bcmills So based on @vtbassmatt comment - it seems as though the reason why Go is not recognising private ADO repositories as git repos is that an auth redirect is presented before other content and therefore <meta> tags on the repo itself isn't having the desired effect.

So it seems to me as though the two options are:

1. Hardcoded entry in Go as in this PR or https://github.com/golang/go/pull/41031
2. tag on auth (which I don't think makes as much sense)

[bcmills]

@artbegolli, @vtbassmatt: the go command should be sending credentials from the user's .netrc file. (See https://golang.org/ref/mod#private-module-proxy-auth, but I notice that we have not yet fully documented the use of .netrc credentials for servers other than module proxies. We should fix that.)

Is that not happening, is the server serving the auth redirect (or failing to serve the go-import <meta> tags) despite valid credentials, or is there something else going on?

[JoseFMP]

Over here we are not using a proxy.
Still not working.

We keep getting no go-import meta tags ()) when doing go get.

We tried all the solutions suggested in this thread with no luck.

[bcmills]

@JoseFMP, are you using credentials from a .netrc file, as mentioned in my previous comment?

[JoseFMP]

@bcmills I tried using .netrc as well as other authentication ways here and there, including the ones in https://docs.microsoft.com/en-us/azure/devops/repos/git/go-get?view=azure-devops with no luck so far for private repos. For public works fine.

In private repos I keep getting:

josem@rabbit:~$ go get dev.azure.com/myorg/myproject/pulsaradapter.git@1073
go: downloading dev.azure.com/myorg/myproject/pulsaradapter.git v0.0.0-20201211023249-f2b661c7768f
go get dev.azure.com/myorg/myproject/pulsaradapter.git@1073: dev.azure.com/myorg/myproject/[email protected]: verifying module: dev.azure.com/myorg/myproject/[email protected]: reading https://sum.golang.org/lookup/dev.azure.com/myorg/!myproject/[email protected]: 410 Gone
        server response:
        not found: dev.azure.com/myorg/myproject/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /tmp/gopath/pkg/mod/cache/vcs/2af3f2ae10b32b5e53fd191e041fc1e3751ac7b215c92b3d0b823915f73813be: exit status 128:
                fatal: unable to connect to dev.azure.com:
                dev.azure.com[0: 13.107.42.20]: errno=Connection refused
                dev.azure.com[1: 2620:1ec:21::20]: errno=Connection refused

401 Gone ===> What is this supposed to mean?
Then exist status 128: Unable co to connect to dev.azure.com

[JoseFMP]

Sorry, maybe probably off topic too, but if you are here due to authentification issues sync'ing with Azure DevOps repos with go get, e.g. on a build machine then you can setup a simple git credential helper in your gitconfig file: e.g:

[credential "https://dev.azure.com"]
  helper = "/bin/sh /etc/azure-devops-helper.sh"

with azure-devops-helper.sh something like

#!/bin/sh
echo username=$USERNAME
echo password=$AZURE_DEVOPS_TOKEN

where environment variables containing the username and PAT taken. This proved more reliable for us than switching to SSH for azure repos.

Silly question here. What's supposed to be in USERNAME var ?

[JoseFMP]

Solved now... needed to tell explicitly in GOPRIVATE to not to check checksums of dev.azure.com .

It seems it was really only that.