Etcd: clientv3: grpc-go (v1.27.0) made API changes to balancer / resolver.

Created on 29 Jan 2020  ·  49Comments  ·  Source: etcd-io/etcd

After the release of grpc-go v1.27.0, an error occurred while pulling etcd / clientv3. The steps to reproduce it are as follows:

  1. go.mod:
go 1.13

require (
    github.com/coreos/etcd v3.3.18+incompatible // indirect
    github.com/coreos/go-systemd v0.0.0-00010101000000-000000000000 // indirect
    github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
    github.com/gogo/protobuf v1.3.1 // indirect
    github.com/google/uuid v1.1.1 // indirect
    go.etcd.io/etcd v3.3.18+incompatible // indirect
    go.uber.org/zap v1.13.0 // indirect
    google.golang.org/grpc v1.27.0 // indirect
)

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0
  1. command: 
$ go get go.etcd.io/etcd/clientv3
# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
  1. PR：https://github.com/etcd-io/etcd/pull/11564
areclientv3
Source
picture eddycjy
👍43 👀6

Most helpful comment

版本不兼容，指定grpc的版本为1.26即可解决问题。
google.golang.org/grpc v1.26.0

picture nieaowei on 30 Jan 2020
👍35 🎉8 ❤7 👀6 😄6 🚀5 😕1

All 49 comments

/cc @jpbetz

xiang90 picture xiang90 on 29 Jan 2020

版本不兼容，指定grpc的版本为1.26即可解决问题。
google.golang.org/grpc v1.26.0

nieaowei picture nieaowei on 30 Jan 2020
👍35 🎉8 ❤7 👀6 😄6 🚀5 😕1

@nieaowei Yes, but we need to solve the problem fundamentally.

eddycjy picture eddycjy on 30 Jan 2020
👍7

Did you find any workaround?

g00nix picture g00nix on 9 Feb 2020

@gun1x You can first downgrade grpc to v1.26.0 to temporarily avoid this problem.

eddycjy picture eddycjy on 10 Feb 2020
👍3

@gun1x You can first downgrade grpc to v1.26.0 to temporarily avoid this problem.

ok, so the actual fix is adding this to go.mod:

require google.golang.org/grpc v1.26.0
replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0
g00nix picture g00nix on 10 Feb 2020

@gun1x can i see your full go.mod for that fix? your suggestion does not fix the issue for me and i'm wondering if i'm missing something.
```go 1.13

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0

require (
github.com/coreos/etcd v3.3.18+incompatible // indirect
github.com/go-log/log v0.2.0 // indirect
github.com/gogo/protobuf v1.3.1 // indirect
github.com/json-iterator/go v1.1.9 // indirect
github.com/lucas-clemente/quic-go v0.14.3 // indirect
github.com/marten-seemann/qtls v0.7.1 // indirect
github.com/micro/go-micro v1.18.0
github.com/miekg/dns v1.1.27 // indirect
github.com/mostfamiliar/shippy-service-consignment v0.0.0-20200212041600-94c659f72b79
github.com/nats-io/jwt v0.3.2 // indirect
github.com/pkg/errors v0.9.1 // indirect
go.uber.org/atomic v1.5.1 // indirect
go.uber.org/multierr v1.4.0 // indirect
go.uber.org/zap v1.13.0 // indirect
golang.org/x/crypto v0.0.0-20200210222208-86ce3cb69678 // indirect
golang.org/x/lint v0.0.0-20200130185559-910be7a94367 // indirect
golang.org/x/tools v0.0.0-20200211205636-11eff242d136 // indirect
google.golang.org/genproto v0.0.0-20200211111953-2dc5924e3898 // indirect
google.golang.org/grpc v1.27.1
)

error:
```# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
mostfamiliar picture mostfamiliar on 12 Feb 2020

@mostfamiliar Your google.golang.org/grpc v1.27.1 should be modified to google.golang.org/grpc v1.26.0.

eddycjy picture eddycjy on 12 Feb 2020

@gun1x can i see your full go.mod for that fix? your suggestion does not fix the issue for me and i'm wondering if i'm missing something.

module mymod

go 1.13

require (
    github.com/coreos/etcd v3.3.18+incompatible
    github.com/coreos/go-systemd v0.0.0-00010101000000-000000000000 // indirect
    github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f // indirect
    github.com/gogo/protobuf v1.3.1 // indirect
    github.com/google/uuid v1.1.1 // indirect
    go.uber.org/zap v1.13.0 // indirect
    google.golang.org/grpc v1.26.0
)

replace github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0
g00nix picture g00nix on 12 Feb 2020

Has this problem been solved?I have to use version 1.27.0 or I can't use micro, but I need to use etcd to get service information.

arCzWw picture arCzWw on 13 Feb 2020
👍2

I have a fix for this issue here: https://github.com/etcd-io/etcd/pull/11580. How do I get a review for it?

arges picture arges on 13 Feb 2020

@arCzWw Micro still uses 1.26.0, so you should be able to pin to that version in your go.mod (this solved the issue for me).

jbhannah picture jbhannah on 14 Feb 2020

遇到同样的问题. meet the same problem

# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:114:78: undefined: resolver.BuildOption
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
vendor\github.com\coreos\etcd\clientv3\balancer\picker\err.go:37:44: undefined: balancer.PickOptions
vendor\github.com\coreos\etcd\clientv3\balancer\picker\roundrobin_balanced.go:55:54: undefined: balancer.PickOptions
yezihack picture yezihack on 16 Feb 2020

if modified to google.golang.org/grpc v1.26.0...
then protobuf build pb error undefined: grpc.SupportPackageIsVersion6..

13642212901 picture 13642212901 on 17 Feb 2020
😕5

@13642212901 The consideration is caused by the high version of protoc.

eddycjy picture eddycjy on 18 Feb 2020

@13642212901 The consideration is caused by the high version of protoc.

I know. I hope etcd-io/etcd support grpc v1.27.*,it,s so easy.

13642212901 picture 13642212901 on 18 Feb 2020

I meet the same issue.

I use the latest version protobuf in https://github.com/protocolbuffers/protobuf/releases

and I go get -u github.com/golang/protobuf/protoc-gen-go

then if I modified to google.golang.org/grpc v1.26.0

protobuf build pb error undefined: grpc.SupportPackageIsVersion6.

if I modified to google.golang.org/grpc v1.27.0

then etcd will report err:

hesongkang picture hesongkang on 18 Feb 2020
👍13

Workaround:

  1. If it is grpc> = 1.27.0, then just change grpc v1.26.0 in the go.mod file, as follows:
require (
    ...
    google.golang.org/grpc v1.26.0
)
  1. If the version of protoc is too high and grpc v1.26.0 is not compatible, you can downgrade protoc.

fix needs to wait for https://github.com/etcd-io/etcd/pull/11564.

eddycjy picture eddycjy on 19 Feb 2020
👍1

遇到同样的问题. meet the same problem

# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:114:78: undefined: resolver.BuildOption
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
vendor\github.com\coreos\etcd\clientv3\balancer\picker\err.go:37:44: undefined: balancer.PickOptions
vendor\github.com\coreos\etcd\clientv3\balancer\picker\roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

resolve method:
https://segmentfault.com/q/1010000021762281

yezihack picture yezihack on 24 Feb 2020
👍1

Workaround:

  1. If it is grpc> = 1.27.0, then just change grpc v1.26.0 in the go.mod file, as follows:
require (
  ...
  google.golang.org/grpc v1.26.0
)
  1. If the version of protoc is too high and grpc v1.26.0 is not compatible, you can downgrade protoc.

fix needs to wait for #11564.

@eddycjy do you know which version of protoc is compatible ? Im stuck as i cant use 26 because of 

internal/storage/storage.pb.go:540:11: undefined: grpc.SupportPackageIsVersion6
internal/storage/storage.pb.go:551:5: undefined: grpc.ClientConnInterface

and cant use 27 because of this issue 😑 Any tips for other work arounds ?

OGKevin picture OGKevin on 25 Feb 2020
👍2

@arCzWw Micro still uses 1.26.0, so you should be able to pin to that version in your go.mod (this solved the issue for me).

@jbhannah Unfortunately this is not the case for me. I get the issue internal/storage/storage.pb.go:540:11: undefined: grpc.SupportPackageIsVersion6 and according to the docs and code found on https://github.com/grpc/grpc-go, v1.27.0 introduced v6 and for v5 i indeed need to pin to v1.26.0. But still the generated code is using v6, i cant seem to get protoc to generate v5 code. Any pointers ?

OGKevin picture OGKevin on 25 Feb 2020

@OGKevin Depends on the structure of your particular project, but I would remove the existing generated files and keep trying to get protoc to regenerate them from the protobuf definitions. I have this comment at the top of a Go file in the same directory as the .proto file and use Go's generate command in my workflow (I'm also using gogoproto):

//go:generate protoc --gogofaster_out=plugins=grpc:. --proto_path=$GOPATH/src:$GOPATH/pkg/mod:. api.proto
jbhannah picture jbhannah on 26 Feb 2020

I have also encountered this problem, i have solved it with @gun1x's approach.

medusar picture medusar on 1 Mar 2020
❤1

@gun1x's approach didn't work for me, if it doesn't for others, try adding:

replace github.com/coreos/etcd v3.3.17+incompatible => github.com/coreos/etcd v3.3.4+incompatible

To your go.mod, didn't need to change grpc or proto versions

SoftTacos picture SoftTacos on 5 Mar 2020
👍6 ❤1

Can a new release be created that can be used for the fix ?

gogrisohil picture gogrisohil on 17 Mar 2020
👍1

版本不兼容，指定grpc的版本为1.26即可解决问题。
google.golang.org/grpc v1.26.0

但是最新版的protoc编译出来的pb文件会提示undefined: grpc.SupportPackageIsVersion6 等问题

phpstudyer picture phpstudyer on 18 Mar 2020

遇到同样的问题. meet the same problem

# github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:114:78: undefined: resolver.BuildOption
vendor\github.com\coreos\etcd\clientv3\balancer\resolver\endpoint\endpoint.go:182:31: undefined: resolver.ResolveNowOption
# github.com/coreos/etcd/clientv3/balancer/picker
vendor\github.com\coreos\etcd\clientv3\balancer\picker\err.go:37:44: undefined: balancer.PickOptions
vendor\github.com\coreos\etcd\clientv3\balancer\picker\roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

resolve method:
https://segmentfault.com/q/1010000021762281

高版本protoc 一样抓瞎 protoc version 3.6.1

phpstudyer picture phpstudyer on 18 Mar 2020

github.com/coreos/etcd v3.3.17+incompatible => github.com/coreos/etcd v3.3.4+incompatible

good job ,thanks

phpstudyer picture phpstudyer on 18 Mar 2020
🚀2 🎉2 👍2 👀1

强制绑定grpc 1.26.0

module kratos-demo

go 1.13

require (
    github.com/bilibili/kratos v0.3.4-0.20200306133340-7c946dc919c1
    github.com/gogo/protobuf v1.3.1
    github.com/golang/protobuf v1.3.5
    github.com/google/wire v0.4.0
    github.com/pkg/errors v0.9.1 // indirect
    google.golang.org/genproto v0.0.0-20200318110522-7735f76e9fa5
    google.golang.org/grpc v1.27.0
)

replace (
    github.com/coreos/go-systemd => github.com/coreos/go-systemd/v22 v22.0.0
    google.golang.org/grpc => google.golang.org/grpc v1.26.0
)
cjb21sj picture cjb21sj on 20 Mar 2020
❤5

Workaround:

  1. If it is grpc> = 1.27.0, then just change grpc v1.26.0 in the go.mod file, as follows:
require (
    ...
    google.golang.org/grpc v1.26.0
)
  1. If the version of protoc is too high and grpc v1.26.0 is not compatible, you can downgrade protoc.

fix needs to wait for #11564.

@eddycjy do you know which version of protoc is compatible ? Im stuck as i cant use 26 because of

internal/storage/storage.pb.go:540:11: undefined: grpc.SupportPackageIsVersion6
internal/storage/storage.pb.go:551:5: undefined: grpc.ClientConnInterface

and cant use 27 because of this issue 😑 Any tips for other work arounds ?

@OGKevin Were you able to figure out a work around for this? I'm facing the same issue and looking for a fix.

shreesha21 picture shreesha21 on 6 Apr 2020

@shreesha21 i removed the GRPC plugin in the protoc command and this seems to generate valid spec.

OGKevin picture OGKevin on 13 Apr 2020

is there any plan to upgrade etcd to support google.golang.org/grpc v1.26.0 and newer?

teone picture teone on 27 Apr 2020

@teone i already use grpc 1.29.1, but you should use replace like that:

replace go.etcd.io/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200329194405-dd816f0735f8
vooon picture vooon on 28 Apr 2020
👍2

Hi I'm facing issue .

github.com/micro/go-micro/v2/registry/etcd
../../../pkg/mod/github.com/micro/go-micro/[email protected]/registry/etcd/etcd.go:79:10: config.LogConfig undefined (type clientv3.Config has no field or method LogConfig)

neetuom picture neetuom on 12 May 2020

module ship-microservice-vessel.com/ship-microservice-vessel

go 1.14

require (
github.com/golang/protobuf v1.4.1
github.com/grpc-ecosystem/grpc-gateway v1.14.5
github.com/micro/go-micro/v2 v2.6.0
google.golang.org/grpc v1.29.1
google.golang.org/protobuf v1.22.0
)

replace github.com/coreos/etcd v3.3.18+incompatible => github.com/coreos/etcd v3.3.4+incompatible

neetuom picture neetuom on 12 May 2020

github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
../../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
../../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption

github.com/coreos/etcd/clientv3/balancer/picker

../../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
../../../pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

above given issue is solved after downgrading the etcd version mentioned below github.com/coreos/etcd v3.3.18+incompatible => github.com/coreos/etcd v3.3.4+incompatible

neetuom picture neetuom on 12 May 2020
👍1

Please help to sort out the issue

neetuom picture neetuom on 12 May 2020

@neetuom I recently managed to upgrade all my dependencies using the following set of replace statements, maybe that's helpful: https://github.com/flexkube/libflexkube/commit/f9cb8bda14eccbb2747eaac2c5007ef5d60d3939#diff-37aff102a57d3d7b797f152915a6dc16L85-R99

invidian picture invidian on 13 May 2020

@neetuom I recently managed to upgrade all my dependencies using the following set of replace statements, maybe that's helpful: flexkube/libflexkube@f9cb8bd#diff-37aff102a57d3d7b797f152915a6dc16L85-R99

Thanks Bro!

neetuom picture neetuom on 18 May 2020
👍1

I'm still struggling here. Anyone have working example I can lurk?

For reference https://github.com/johnbellone/cache-service

johnbellone picture johnbellone on 2 Jun 2020

@invidian replacements did the job for me:

replace (
    github.com/coreos/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
    go.etcd.io/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
)

which is a little odd, since that branch is 6 years old.

refs picture refs on 19 Jun 2020

I ended up nuking my attempt to get Bazel to work as a build environment and went back to simply using go mod and go build. I was able to quickly get it working with only the following:

go 1.14

require (
    github.com/golang/protobuf v1.4.2
    github.com/micro/go-micro/v2 v2.9.0
    google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1
)
johnbellone picture johnbellone on 19 Jun 2020

wish we move etcd dependency out of go-micro, maybe move it to plugins.
this is giving bad experience for first-time go-micro users.

xmlking picture xmlking on 20 Jun 2020
👍11

@invidian replacements did the job for me:

replace (
  github.com/coreos/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
  go.etcd.io/etcd => go.etcd.io/etcd v0.5.0-alpha.5.0.20200425165423-262c93980547
)

which is a little odd, since that branch is 6 years old.

doesn't seem to work, I get conflicts:

go: go.etcd.io/[email protected] used for two different module paths (github.com/coreos/etcd and go.etcd.io/etcd)
PVince81 picture PVince81 on 22 Jun 2020

i'm use this trick -
replace (
google.golang.org/grpc => google.golang.org/grpc v1.27.0
github.com/coreos/etcd => github.com/ozonru/etcd v3.3.20-grpc1.27-origmodule+incompatible
)

vtolstov picture vtolstov on 15 Jul 2020
👍6

hmmmm, right. Thanks for sharing! I'll be using that one.

refs picture refs on 15 Jul 2020

i'm use this trick -
replace (
google.golang.org/grpc => google.golang.org/grpc v1.27.0
github.com/coreos/etcd => github.com/ozonru/etcd v3.3.20-grpc1.27-origmodule+incompatible
)

Thanks mate, seems to have worked for me!

KyteProject picture KyteProject on 23 Jul 2020
👍1

Seems that etcd maintainer did not take serious to fix the issue. I still got this issue, where I do need latest gRPC (1.31+)

github.com/coreos/etcd/clientv3/balancer/resolver/endpoint

/home/alpha/go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:114:78: undefined: resolver.BuildOption
/home/alpha/go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/resolver/endpoint/endpoint.go:182:31: undefined: resolver.ResolveNowOption

github.com/coreos/etcd/clientv3/balancer/picker

/home/alpha/go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/err.go:37:44: undefined: balancer.PickOptions
/home/alpha/go/pkg/mod/github.com/coreos/[email protected]+incompatible/clientv3/balancer/picker/roundrobin_balanced.go:55:54: undefined: balancer.PickOptions

How I get the latest etcd golang module if there is a fix?

zxpan picture zxpan on 30 Sep 2020

@zxpan

The fix is in unreleased 3.5 branch:
https://github.com/etcd-io/etcd/commit/4258cdd2efdf79a145067ae426f93b9f220c05f1#diff-c3bc35973ca604ca1c88d111806c06c4

and backported to 3.4:
https://github.com/etcd-io/etcd/blob/release-3.4/clientv3/balancer/picker/roundrobin_balanced.go

Please try using etcd 3.4

ptabor picture ptabor on 30 Sep 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

WanLinghao picture WanLinghao  ·  4Comments
olalonde picture olalonde  ·  4Comments
r007m4n picture r007m4n  ·  3Comments
primeroz picture primeroz  ·  3Comments
el10savio picture el10savio  ·  4Comments