Istio: Istio ingress gateway with AWS NLB has high target reset count

[ediezh]

Istio ingress gateway with AWS NLB has high target reset count(400-500 per minute)

[ ] Docs
[ ] Installation
[X] Networking
[ ] Performance and Scalability
[ ] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior

AWS NLB metrics show reasonable target reset count

Steps to reproduce the bug
Deploy istio ingress gateway with NLB

Version (include the output of istioctl version --remote and kubectl version --short and helm version if you used Helm)

 20-09-17 15:09  ~  istioctl version --remote
1.7.0
data plane version: 1.7.0 (49 proxies)

 20-09-17 15:09  ~  kubectl version --short
Client Version: v1.18.6
Server Version: v1.16.13-eks-2ba888

How was Istio installed?
istioctl

Environment where bug was observed (cloud vendor, OS, etc)
AWS

Istio operator configuration

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
          hpaSpec:
            minReplicas: 2

[ediezh]

Hmm..looks like setting ingress gateway service externalTrafficPolicy to Local fixed the problem.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-type: nlb
            service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
          hpaSpec:
            minReplicas: 2
          service:
            externalTrafficPolicy: Local

[ediezh]

But I'm still curious why the NLB target reset count is high with the default config?