Athens: private gitlab project support

[pigghost]

We have many common modules in our private gitlab repository, and this gitlab webiste only support http, not https.

How can I use athens to proxy this gitlab repo?

[arschles]

@pigghost does a go get my.internal.gitlab.repo? work, without the athens proxy?

[pigghost]

@arschles go get my.internal.gitlab.repo doesn't work, must add the "-insecure" option, like
go get -insecure my.internal.gitlab.repo. because the repo only support "http"

[manugupt1]

While I have not checked all the details, I think there are some things that we can do

A) Do nothing for unsecure private repositories.
B) Add an env var that says ALLOW_INSECURE and adds that to the go commands inside our code base
C) Add a parameter proxy --allow-insecure on load which essentially does the same as B)
D) The last one is by default append GOFLAGS to all our commands. So for example, a user would set GOFLAGS=insecure in their GOENV which we can add to the go commands that we are using.

There is a similar bug report in cmd/go : https://github.com/golang/go/issues/27332

Thoughts and ideas welcome

[arschles]

My vote is for combining (b) and (c) since they have the same effect. I think our new configuration system allows us to use environment vars and flags at the same time (is that right @rchakra3?), so we could use that?

cc/ @fedepaol @robjloranger @marpio and @marwan-at-work since I think all of you have touched the go get fetcher and/or done some git credentials auth in the recent-ish past

[manugupt1]

(Oh I would like to take this up!)

[rchakra3]

There's no flag -> config parsing at the moment. You can either specify a value in the config file or add an environment variable.
(B) with an additional value in the configuration would be consistent with our current config system.

Regardless of what we choose I think it's important to call out the default behavior users would see with GOFLAGS set. If I understand it right, setting GOFLAGS=-insecure and running the proxy right now would enable the scenario mentioned by @pigghost

[manugupt1]

@rchakra3 Yep!

I have been thinking of a reason not to do GOFLAGS as I really liked the idea of let users add a GOFLAGS=-insecure and we can call it done 💃.

This is the scenario where I like adding a config option or a flag. If the user has a single EC2 instance and they use the same environment to build it and run then by default all their commands would look like this

go -insecure build
go -insecure run
go -insecure mod download

Another option is probably to not to do things on our side is use
GIT_ALLOW_PROTOCOL

As defined in go docs here : https://golang.org/cmd/go/
Defined by Git. A colon-separated list of schemes that are allowed to be used
with git fetch/clone. If set, any scheme not explicitly mentioned will be
considered insecure by 'go get'.

I like this alternative as well.

[marpio]

I don't know how likely this scenario would be but I could imagine some organizations have their repos spread across multiple VCS services i.e. some repos on Github and some on Gitlab. If we use approaches listed above (config, env var, GOFLAGS) would that mean that we set insecure for all of them?
Meaning, we do
go get -insecure my.internal.gitlab.repo
but also
go get -insecure my.internal.**github**.repo

[manugupt1]

@marpio Yes! You are correct! The only thing is that the user is making a decision about marking it as insecure, so I think since a proxy is meant to be in an enterprise setting it does not matter much. It is not ideal though.

I like GIT_ALLOW_PROTOCOL the best
This allows users to specify a list of a protocol that will not have an insecure option and everything else is marked with an -insecure flag. The documentation states that is only git specific so we need to check if other VCS are supported are not or are other vcs just marked as insecure.

I am not sure about other VCS and how to proceed with them though.

PS: @pigghost Do you want to try using the GIT_ALLOW_PROTOCOL and let us know where it leads to as that should satisfy your use case for Gitlab while we are talking about this.

[marpio]

@manugupt1
These are the default secure protocols
https://github.com/golang/go/blob/930ce09ca3a1ab5f886c2327fc143b9ae075807b/src/cmd/go/internal/get/vcs.go#L47-L53
GIT_ALLOW_PROTOCOL can be used but it's specific to git
https://github.com/golang/go/blob/930ce09ca3a1ab5f886c2327fc143b9ae075807b/src/cmd/go/internal/get/vcs.go#L64-L80
I think we should have a generic solution for all VCS supported by go, so I'm not sure if GIT_ALLOW_PROTOCOL is suitable here.

I also think that the proxy should use the insecure option explicitly, meaning it shouldn't be used where it's not required. If we use it globally we might also pull public modules using HTTP and would not notice that, which wouldn't be ideal.

What do you think of having a config/env var where the user could define a list of insecure sources?

[manugupt1]

I am not sure about config / env var anymore for the following reasons

• If you do a go get my.internal.gitlab.repo, it does not really say which protocol it is. I have yet to look at our code to think if we really parse it our not but afaik (this managed by vcs.go), we run
  go get
go list and so on.
• Having an ENV VAR may mean if a user introduces a new config, then they will have to restart the proxy or the environment for it to load depending on how the code is written.

What about this alternative approach?

We always try without the secure flag. If the repo is successful, then we do not do anything.
If it fails, we try by applying the -insecure flag and see if we can run the commands or not.
I think this would be useful for some reasons:

1. the code for go cmd to find out the scheme is in the go repo and we can let it do its job.
2. the failure without the insecure flag is not an expensive one as not a lot of data will be transferred.
3. The user will not have to set and know about their protocol, they can just run the proxy and not worry about it at all.

[michalpristas]

I'm thinking about a second approach but with something like filtering, where you specify a filter of allowed insecure.
another thing is: do we want to store modules coming from unsecured locations?
VCS with self-signed certs or no cert at all: we don't have a circle of trust and yet we provide a mechanism to make them signed, verified and trusted. (for future releases)
I would go for no. @arschles ??

[marpio]

If you do a go get my.internal.gitlab.repo, it does not really say which protocol it is.

Do we need to know the protocol? If we had an list/filter with insecure sources and my.internal.gitlab.repo is on that list then we would apply the -insecure flag.

We always try without the secure flag. If the repo is successful, then we do not do anything.
If it fails, we try by applying the -insecure flag and see if we can run the commands or not.

I'm not sure about that. If we just try the -insecure every time it would be possible for an an attacker to block secure ports (443 for HTTPS and 22 for git+ssh) and serve malicious packages via plain-text HTTP or git protocols.
Or another scenario. Let say I want to add the rsc.io/pdf package but for some reason the rsc.io unexpectedly starts to serve the content using HTTP instead of HTTPS. If we just try -insecure every time we wouldn't notice the unexpected behavior and would just get the module over HTTP

I think we should go with an explicit configurable list of of allowed insecure sources. It could work similarly to the include/exclude filter we already have.

[manugupt1]

@marpio Perfect! Let me start with this now!

[pigghost]

As I know, go get and go download support '-insecure' flag, but go list seems not

[marpio]

@pigghost the go list cmd should still work on a insecure repo though.
Could you try it out with your gitlab repository?
The proxy executes go list -m -versions -json

[pigghost]

@pigghost the go list cmd should still work on a insecure repo though.
Could you try it out with your gitlab repository?
The proxy executes go list -m -versions -json

@marpio I have tried go list -v -m, and not work. The log message shows that go only tries to fetch https://xxx?go-get=1

[marpio]

@pigghost
Could you try the following?
Add this to your $HOME/.gitconfig

[url "http://gitlab"]
insteadOf = https://gitlab

Then please create two files somewhere outside of GOPATH
go.mod with the content module mod
mod.go with the content package mod

and run:
GO111MODULE=on go list -versions -m your.gitlab.repo@latest
Note -versions and not -v

This is just a workaround. I think go should respect the -insecure flag in the long run.

EDIT: After looking at the go code again, I think GIT_ALLOW_PROTOCOL=http should do the trick as well

[manugupt1]

@marpio @michalpristas
After looking at this issue: https://github.com/golang/go/issues/27332 and https://go-review.googlesource.com/c/135735/

Seems like go mod download -insecure is in the works, but has not been released yet, and may not be either.

A) Should we wait until then or should I work off the tip?
B) Finish up implementing the file reading pattern like include/exclude filter and then look at this back again.

@pigghost FYI ^^ Check this CL out. Seems like the support is scheduled for Go1.12

PS: Sorry for editing this message so many times.

[michalpristas]

i would go with A, without a download this feature is not that usable right?
correct me if i'm wrong

[manugupt1]

yep! it won't be usable until late January which is when Go1.12 is scheduled to release.

I think I am in agreement with you guys as well. Can we put it on hold and mark it as won't fix for Go1.11 referencing that one if you guys are cool with it?

[fengxuway]

Go 1.12 is comming! https://golang.org/doc/devel/release.html#go1.12
But go mod download and go list does not have arg -insecure still.

[Haiyung]

Go 1.13 is comming!
But go mod download does not support -insecure still.

[arschles]

@Haiyung I'm catching up on this now. It seems possible that Athens could work with a gitconfig as @marpio showed in https://github.com/gomods/athens/issues/640#issuecomment-423772137, to get this to work. What do you think?

cc/ @manugupt1 @michalpristas

[ZaynJarvis]

With Go 1.14, I think we can try out GOINSECURE env

[arschles]

@ZaynJarvis I agree, and I think that's the best course of action 😄 . The full description of how to set GOINSECURE is in https://github.com/gomods/athens/pull/697#issuecomment-596014257. Here is how you would do that with the environment variable, if you don't want to look at that other issue:

$ export ATHENS_GO_BINARY_ENV_VARS='GOINSECURE=xyz'

Unfortunately, this GOINSECURE environment variable only works with Go 1.14, so we don't have an Athens release yet that supports it. For now, gomods/athens-dev:e1374be supports it and it will available in the next release.

I am going to close this issue now. @pigghost can you please comment here if this doesn't work for you or you have more questions? I will get the notification and will come back to reopen the issue.

Thank you for sticking around for 1.5 years! 😄