Operator-sdk: `operator-sdk new` fails because it is unable to reach git.apache.org

[gzur]

Platform:
macOS Mojave 10.14.6 - just installed via HomeBrew.

$ operator-sdk version
operator-sdk version: v0.10.0, commit: ff80b17737a6a0aade663e4827e8af3ab5a21170

Problem encountered:
I can't run operator-sdk new because git.apache.org seems to be down. (see command output below)

I notice that thrift is also available on github, but I don't know enough about go-lang dependency management to make use of that fact.

$ operator-sdk new test-operator --repo github.com/gzur/test-operator
INFO[0000] Creating new Go operator 'test-operator'.
INFO[0000] Created go.mod
INFO[0000] Created tools.go
INFO[0000] Created cmd/manager/main.go
INFO[0000] Created build/Dockerfile
INFO[0000] Created build/bin/entrypoint
INFO[0000] Created build/bin/user_setup
INFO[0000] Created deploy/service_account.yaml
INFO[0000] Created deploy/role.yaml
INFO[0000] Created deploy/role_binding.yaml
INFO[0000] Created deploy/operator.yaml
INFO[0000] Created pkg/apis/apis.go
INFO[0000] Created pkg/controller/controller.go
INFO[0000] Created version/version.go
INFO[0000] Created .gitignore
INFO[0000] Validating project
go: finding github.com/operator-framework/operator-sdk master
go: finding git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999
go: finding git.apache.org/thrift.git v0.12.0
go: git.apache.org/[email protected]: unknown revision v0.12.0
go: git.apache.org/[email protected]: git fetch -f https://git.apache.org/thrift.git refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /Users/gissur/go/pkg/mod/cache/vcs/83dba939f95a790e497d565fc4418400145a1a514f955fa052f662d56e920c3e: exit status 128:
    fatal: unable to access 'https://git.apache.org/thrift.git/': Failed to connect to git.apache.org port 443: Operation timed out
go: error loading module requirements
Error: failed to exec []string{"go", "build", "./..."}: exit status 1

[jadencodes]

@gzur
Ran into the same issue. For now, I have a workaround:

Let the install get as far as it can. Then:

# add this to the end of the go.mod file
replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0

Then from the actual directory of the operator (for you test-operator) run:

go mod tidy

This will fetch the remaining dependencies

[camilamacedo86]

Hi @gzur,

I tried to reproduce this scenario with the master branch and I could not face the same issue. Shows the dep is in the repo informed and not in the operator-sdk. If you just run the command without inform the repo the project will be created successfully.

$ operator-sdk new test-operator 
INFO[0000] Creating new Go operator 'test-operator'.    
INFO[0000] Created go.mod                               
INFO[0000] Created tools.go                             
INFO[0000] Created cmd/manager/main.go                  
INFO[0000] Created build/Dockerfile                     
INFO[0000] Created build/bin/entrypoint                 
INFO[0000] Created build/bin/user_setup                 
INFO[0000] Created deploy/service_account.yaml          
INFO[0000] Created deploy/role.yaml                     
INFO[0000] Created deploy/role_binding.yaml             
INFO[0000] Created deploy/operator.yaml                 
INFO[0000] Created pkg/apis/apis.go                     
INFO[0000] Created pkg/controller/controller.go         
INFO[0000] Created version/version.go                   
INFO[0000] Created .gitignore                           
INFO[0000] Validating project                           
go: finding github.com/operator-framework/operator-sdk master
go: downloading github.com/operator-framework/operator-sdk v0.10.1-0.20190829173600-c0947baa1fed
go: extracting github.com/operator-framework/operator-sdk v0.10.1-0.20190829173600-c0947baa1fed
INFO[0011] Project validation successful.               
INFO[0011] Project creation complete.                   

Please, let us know if it solved your issue and if has anything that we could do here to help you with.

[camilamacedo86]

HI @joelanford,

WDYT? Has anything else that we should address in this one? Have any action which should be done here or we could close this one since the scaffold project created by master branch shows not face this issue?

PS.: I faced the same issue with a project which was already created and the solution added here solved it.

[joelanford]

@camilamacedo86 I think you would need to delete your Go modules cache and attempt to reproduce it without that package already existing in your cache.

Having said that, I'm not sure there's anything for us to do here since this seems to be a temporary issue with a transitive dependency. I think the workaround that @jadencodes posted (thanks!) is enough to close this issue.

[go-udy]

I run operator-sdk new test-operator but it gets stuck at finding git.apache.org/thrift.git error

but adding replace git.apache.org/thrift.git => github.com/apache/thrift v0.12.0 to go.mod it works fine! thanks @jadencodes

[doltori]

@UdayVishwakarma, what go.mod file did you change?
Command operator-sdk new app-operator --repo github.com/example-inc/app-operator was failed cause by same problem. So I changed go.mod file in app-operator folder. And ran the command again, but I got path already exists. Please use a different project name or delete the existing one error message.

[go-udy]

@doltori update the go.mod and run go mod vendor it fetches the dependencies. Command operator-sdk new app-operator will generate boiler plate and will try to fetch dependencies, so it was failing during updating dependencies. So running go mod vendor should be enough

[estroz]

Note that git.apache.org is down and the cause is being investigated. This has happened a few times in the past few days AFAIK.

[JeremiahDJordan]

As far as I know the ASF recommends people always download through mirrors and not directly from ASF resources. So switching to the github mirror seems like the right thing to do here.

[mlococo]

@joelanford @camilamacedo86 I'm still seeing this in my build. Any chance to reopen? I think the following presents a pretty good case to do so:

1. While it's true that this is "transient", the apache git servers have been up and down very frequently going on 2 weeks now per https://status.apache.org/. I'm not sure if there's a projected time for a permanent resolution, but it's ongoing right now and shows that really big extended outages are possible for the Apache git servers.
2. Jadencodes workaround shows that there is another approach that's viable, and github has a meaningfully better uptime record than we're seeing with Apache. I believe it's accurate to say that Github has never had an outage/degredation of this duration.
3. Jadencodes workaround is ineffective for CI/CD. If I understand correctly, the go.mod file being edited is created during the vendor as part of the boilerplate, and I don't believe there is a facility to add a replace in there before running the job. So while the workaround for local-development isn't to bad, this is really playing havoc with CI jobs for people.
4. Jeremiah is suggesting that relying on github mirrors for dependency download is actually idiomatic and preferred.

Given the above, could we consider re-opening this and switching over to the github repos by default? It doesn't address the fundamental issue of "could dependency download fail" but from a practical standpoint it moves the dependency hosting to much more reliable infrastructure. Especially this week as Apache is having ongoing issues.

[estroz]

I'll submit a PR that adds replace git.apache.org/thrift.git => github.com/apache/thrift to go.mod scaffolds.

[mlococo]

Thanks for the speedy ack, and thanks for all your work making operator-development easier.

[mlococo]

It looks like git.apache.org isn't coming back at all: https://blogs.apache.org/infra/entry/subversion-to-git-service-git The advice Jeremiah gave earlier about github mirrors being the preferred way of exposing things is reiterated at the bottom of that post.

[GrahamDumpleton]

Since no one seems to have explicitly mentioned it, setting:

GOPROXY=https://proxy.golang.org

environment variable also avoids the problem without needing to change anything. You will pick up the package from the proxy cache.

[mlococo]

FWIW, it wasn't in the package cache 2 days ago, but good to hear that it is now. That's probably the best workaround that functions in CI/CD for now, but given the blog post above I'd still advocate eliminating all dependencies pointing toward git.apache.org.

[GrahamDumpleton]

Hmmm, I only had to deal with the problem in the last day. Is lucky then the proxy worked. I am just hoping it doesn't stop working now.

[camilamacedo86]

It has an open pr for it: https://github.com/operator-framework/operator-sdk/pull/1899