Grpc-go: While using naming resolver , it can't connect to endpoint services .

[Wang-Kai]

Please answer these questions before submitting your issue.

What version of gRPC are you using?

master

What version of Go are you using (go version)?

go1.10.1 darwin/amd64

What operating system (Linux, Windows, …) and version?

MacOS 10.13.6

What did you do?

If possible, provide a recipe for reproducing the error.

client.go

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "github.com/Wang-Kai/hi"
    "github.com/Wang-Kai/hi/example/pb"
    "google.golang.org/grpc"
    "google.golang.org/grpc/resolver"
)

var (
    etcdLoc = "localhost:2379"
)
var ConnMap = make(map[string]*grpc.ClientConn, 2)

func init() {
    // register resolver
    hiBuilder := hi.NewResolverBuilder([]string{"localhost:2379"})
    resolver.Register(&hiBuilder)

    // build connection of serverB
    serverBConn, err := grpc.Dial(hiBuilder.Scheme()+"://author/serverB", grpc.WithInsecure(), grpc.WithBalancerName("round_robin"))
    if err != nil {
        log.Fatal(err)
    }
    println(serverBConn)
    ConnMap["serverB"] = serverBConn

    // build connection of serverA
    serverAConn, err := grpc.Dial(hiBuilder.Scheme()+"://author/serverA", grpc.WithInsecure(), grpc.WithBalancerName("round_robin"))
    if err != nil {
        log.Fatal(err)
    }
    println(serverAConn)
    ConnMap["serverA"] = serverAConn
}

func main() {
    defer func() {
        for _, cc := range ConnMap {
            cc.Close()
        }
    }()

    for range time.Tick(time.Second * 3) {
        println("Call serverB")
        serverBClient := pb.NewServerBClient(ConnMap["serverB"])

        println("New Client")
        helloResp, err := serverBClient.Hello(context.Background(), &pb.HelloReq{Name: "China"})
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%+v \n", helloResp)

        println("Call serverA")
        serverAClient := pb.NewServerAClient(ConnMap["serverA"])
        hiResp, err := serverAClient.Hi(context.Background(), &pb.HiReq{Name: "kai"})
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%+v \n", hiResp)
    }
}

serverA.go

package main

import (
    "context"
    "flag"
    "fmt"
    "log"
    "net"

    "github.com/Wang-Kai/hi"
    "github.com/Wang-Kai/hi/example/pb"

    "google.golang.org/grpc"
)

var (
    svcName *string
    port    *string
)

func init() {
    svcName = flag.String("name", "serverA", "The name of microservice")
    port = flag.String("port", "10013", "The port of microservice")
    flag.Parse()
}

type serverA struct{}

func (s *serverA) Hi(ctx context.Context, req *pb.HiReq) (*pb.HiResp, error) {
    println("Yeah, it is serverA ...")

    return &pb.HiResp{Echo: "Hi " + req.Name + ", this response comes from ServerA"}, nil
}

func main() {
    lis, err := net.Listen("tcp", fmt.Sprintf(":%s", *port))
    if err != nil {
        log.Fatal(err)
    }

    // register serverA to etcd
    h := hi.NewHi([]string{"localhost:2379"}, "hi")
    err = h.Register(*svcName, fmt.Sprintf("127.0.0.1:%s", *port))
    if err != nil {
        log.Fatal(err)
    }

    s := grpc.NewServer()
    pb.RegisterServerAServer(s, &serverA{})

    println("Hello, I am serverA ...")

    if err := s.Serve(lis); err != nil {
        log.Fatal(err)
    }
}

serverB.go

package main

import (
    "context"
    "fmt"
    "log"
    "net"

    "github.com/Wang-Kai/hi"
    "github.com/Wang-Kai/hi/example/pb"
    "google.golang.org/grpc"
)

var (
    srvName = "serverB"
    port    = ":10015"
)

type serverB struct{}

func (s *serverB) Hello(ctx context.Context, req *pb.HelloReq) (*pb.HelloResp, error) {
    println("Yeah, it is serverB ...")

    return &pb.HelloResp{Echo: "Hello" + req.Name + " ..."}, nil
}

func main() {
    lis, err := net.Listen("tcp", port)
    if err != nil {
        log.Fatal(err)
    }

    h := hi.NewHi([]string{"localhost:2379"}, "hi")
    err = h.Register(srvName, fmt.Sprintf("127.0.0.1%s", port))
    if err != nil {
        log.Fatal(err)
    }

    s := grpc.NewServer()
    pb.RegisterServerBServer(s, &serverB{})

    println("I am serverB ...")
    if err := s.Serve(lis); err != nil {
        log.Fatal(err)
    }
}

What did you expect to see?

I want it can dial to serverA & serverB successfully

What did you see instead?

It block (hang)

[menghanl]

The resolver depends on etcd to get the server addresses, can you double check it gets the correct addresses first?

Also, can you run the client with GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info and get the logs?

[Wang-Kai]

Thank you very much .

（1）I am true i got the correct addresses，in my Builder (implement resolver.Builder) watch function, i print received addresses , it is correct .

func (b *Builder) watch(keyPrefix string) {
    var addrList []resolver.Address

    // first get all address under this keyPrefix
    resp, err := cli.Get(context.Background(), keyPrefix, clientv3.WithPrefix())
    if err != nil {
        log.Fatal(err)
    }

    for _, kv := range resp.Kvs {
        addr := resolver.Address{Addr: strings.TrimPrefix(string(kv.Key), keyPrefix)}
        addrList = append(addrList, addr)
    }

    fmt.Printf("%+v \n", addrList)
    b.cc.NewAddress(addrList)
       .......
}

[{Addr:127.0.0.1:10013 Type:0 ServerName: Metadata:<nil>}]
[{Addr:127.0.0.1:10015 Type:0 ServerName: Metadata:<nil>}]

(2) I don't know how to run client.go with GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info

It is go run client.go GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info ?

[menghanl]

GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info go run client.go, and then you should see logs printed by the client.

[Wang-Kai]

Thank you very much .

When i run GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info go run main.go , i got

INFO: 2018/08/08 09:33:50 ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:10015 0  <nil>}]
INFO: 2018/08/08 09:33:50 ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:10013 0  <nil>}]
INFO: 2018/08/08 09:33:50 base.baseBalancer: got new resolved addresses:  [{127.0.0.1:10013 0  <nil>}]
INFO: 2018/08/08 09:33:50 base.baseBalancer: handle SubConn state change: 0xc4202b2100, CONNECTING
INFO: 2018/08/08 09:33:50 base.baseBalancer: handle SubConn state change: 0xc4202b2100, READY
INFO: 2018/08/08 09:33:50 roundrobinPicker: newPicker called with readySCs: map[{127.0.0.1:10013 0  <nil>}:0xc4202b2100]

My serverA listen on 127.0.0.1:10013, and serverB listen on 127.0.0.1:10015

As my code :

func init() {
    // register resolver
    hiBuilder := hi.NewResolverBuilder([]string{"localhost:2379"})
    resolver.Register(&hiBuilder)

    // build connection of serverB
    serverBConn, err := grpc.Dial(hiBuilder.Scheme()+"://author/serverB", grpc.WithInsecure(), grpc.WithBalancerName("round_robin"))
    if err != nil {
        log.Fatal(err)
    }
    println(serverBConn)
    ConnMap["serverB"] = serverBConn

    // build connection of serverA
    serverAConn, err := grpc.Dial(hiBuilder.Scheme()+"://author/serverA", grpc.WithInsecure(), grpc.WithBalancerName("round_robin"))
    if err != nil {
        log.Fatal(err)
    }
    println(serverAConn)
    ConnMap["serverA"] = serverAConn
}

func main() {
    defer func() {
        for _, cc := range ConnMap {
            cc.Close()
        }
    }()

    time.Sleep(time.Second * 3)

    for range time.Tick(time.Second * 3) {
        println("Call serverB")
        serverBClient := pb.NewServerBClient(ConnMap["serverB"])

        println("New Client")
        helloResp, err := serverBClient.Hello(context.Background(), &pb.HelloReq{Name: "China"})
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%+v \n", helloResp)

        println("Call serverA")
        serverAClient := pb.NewServerAClient(ConnMap["serverA"])
        hiResp, err := serverAClient.Hi(context.Background(), &pb.HiReq{Name: "kai"})
        if err != nil {
            log.Fatal(err)
        }
        fmt.Printf("%+v \n", hiResp)
    }
}

My program is hanging after calling serverB . I don't know what is the problem .

[menghanl]

Short answer is, in your resolver implementation, the watch() method should be on each resolver, not on the builder.

Because

• Builder builds Resolvers.
• Different clients calls the same Builder to build different Resolvers (this happens as part of grpc.Dial().
• So the Builder is shared by all the clients, but different clients should have different Resolvers.

In your resolver implementation, the watch() method and all the updates happen in the Builder, so when you only have one client, things should work fine.
But when you call grpc.Dial the second time, the Builder will be modified by your build function, and things start to go wrong.

A working resolver structure in my mind would look like

type resolver struct {
  cli *etcdClient
  cc  resolver.ClientConn
}
func (*resolver) watch() {
  // Similar to your existing code.
}

type builder struct {...}
func (builder) Build() {
  cli := etcd.New()
  r := &resolver{cli: cli}
  go r.watch() // watch is done by resolver, not the bulder
}

You can also take a look at the dns resolver as an example.

[Wang-Kai]

Thanks for your patient, thanks for your advice , i solved my problem .