Linkerd2: LoadBalancer does not work when "Getting Started" steps applied for a custom deployment with gRPC server

[lwsanty]

Summary I wanted to apply Getting Started steps for a custom deployment with gRPC server, but load-balancing did not happen. I would appreciate any advice regarding to what am I doing wrong.

Environment

• linkerd version stable-2.6.0
• minikube version: v1.5.1
• ubuntu 18.04

Steps
1) setup linkered

curl -sL https://run.linkerd.io/install | sh
linkerd check --pre
linkerd install | kubectl apply -f -
linkerd check

2) install 5 replicas deployment which contain a container with gRPC server

kind: Namespace
apiVersion: v1
metadata:
  name: go-driver
---
kind: Service
apiVersion: v1
metadata:
  name: go-driver-service
  namespace: go-driver
spec:
  selector:
    app: go-driver-app
  ports:
    - protocol: TCP
      port: 9432
      targetPort: grpc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-driver-deployment
  namespace: go-driver
spec:
  replicas: 5
  selector:
    matchLabels:
      app: go-driver-app
  template:
    metadata:
      namespace: go-driver
      labels:
        app: go-driver-app
    spec:
      containers:
        - name: go-driver-server
          image: bblfsh/bblfshd:latest-drivers
          imagePullPolicy: Always
          ports:
            - name: grpc
              containerPort: 9432
          env:
            - name: LOG_LEVEL
              value: debug

3) inject linkered

kubectl get -n go-driver deploy -o yaml \
  | linkerd inject - \
  | kubectl apply -f -

linkerd -n go-driver check --proxy

4) start logger

linkerd -n go-driver top deploy

5) port-forward service port

kubectl -n go-driver port-forward svc/go-driver-service 9432:9432

6) start bombing address localhost:9432 with a gRPC client, which establishes the connection to endpoint and starts sending requests infinitely.

results

• no logs in linkerd -n go-driver top deploy
• from containers logs and linkerd dashboard it is observed that only one pod receives all load

note I had some doubts about port forwarding so I created a separate deployment with the client to send request from inside the cluster, but the result was the same

apiVersion: apps/v1
kind: Deployment
metadata:
  name: johnny
  namespace: go-driver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-driver-app
  template:
    metadata:
      namespace: go-driver
      labels:
        app: go-driver-app
    spec:
      containers:
        - name: johnny
          image: johnny:latest
          imagePullPolicy: Never
          env:
            - name: ADDRESS
              value: go-driver-service.go-driver:9432

[olix0r]

Hi @lwsanty!

note I had some doubts about port forwarding so I created a separate deployment with the client to send request from inside the cluster, but the result was the same

You are correct that port-forwarding bypasses the Linkerd proxy! So that's not a great way to test it. But I would expect your test deployment to work as expected if you inject it with linkerd!

Note that load balancing is all _client side_ -- so an ingress controller or client app like your go-driver is what actually does the load balancing. Injecting the _servers_ is valuable for metrics, etc; but load balancing is not implemented on the server-side of a connection.

Our docs could definitely be a whole lot clearer about this; so sorry for the confusion. Can you try injecting your test application and share the results of, for instance, linkerd stat -n go-driver po ?

[lwsanty]

hi @olix0r !

I have injected only the gprc client deployment

kubectl get -n go-driver deploy johnny -o yaml \
  | linkerd inject - \
  | kubectl apply -f -

the command linkerd stat -n go-driver po emitted the next results:

NAME                                   STATUS   MESHED   SUCCESS   RPS   LATENCY_P50   LATENCY_P95   LATENCY_P99   TCP_CONN
go-driver-deployment-ff5bfdb7-6vwk8   Running      0/1         -     -             -             -             -          -
go-driver-deployment-ff5bfdb7-7d68v   Running      0/1         -     -             -             -             -          -
go-driver-deployment-ff5bfdb7-ht6wr   Running      0/1         -     -             -             -             -          -
go-driver-deployment-ff5bfdb7-mm87v   Running      0/1         -     -             -             -             -          -
go-driver-deployment-ff5bfdb7-nrtdz   Running      0/1         -     -             -             -             -          -
johnny-fdcfbddf4-kcrfr                Running      1/1         -     -             -             -             -          -

Calls metrics

However I still cannot get if load balancing works.

---

Update
I have also injected go-driver-deployment, restarted client deployment(deleted a pod to force RC to recreate it) and it looks like only one pod is loaded

things that could possibly affect: I have some troubles with internal endpoints so client uses ClusterIP instead of go-driver-service.go-driver:9432 but IMO that's not the reason.

[grampelberg]

Linkerd uses the service name to do service, so ClusterIP will skip all load balancing.

[olix0r]

Though, ClusterIP support is working on a branch right now... it should be in an upcoming edge.

[lwsanty]

Thank you, gentlemen!
Everything works with service address go-driver-service.go-driver:9432

Closing.