Skywalking: Failed to enable envoy access log service with istio 1.9

[yckbilly1929]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• [ ] Question or discussion
• [x] Bug
• [ ] Requirement
• [ ] Feature or performance improvement

---

Bug

• Which version of SkyWalking, OS, and JRE?
  gke: 1.17
  istio: 1.9
  skywalking: (installed by helm)

helm -n istio-system install skywalking skywalking \ 
               --set oap.storageType='h2'\
               --set ui.image.tag=8.4.0 \
               --set oap.image.tag=8.4.0-es7 \
               --set oap.replicas=1 \
               --set oap.env.SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS=k8s-mesh \
               --set oap.env.JAVA_OPTS='-Dmode=' \
               --set oap.envoy.als.enabled=true \
               --set elasticsearch.enabled=false

• What happened?
  Was following the installation guideline from https://skywalking.apache.org/blog/2020-12-03-obs-service-mesh-with-sw-and-als/#installing-istio and trying to integrate skywalking with a running istio cluster but error logs reported in istio-ingressgateway and failed to start.

2021-02-18T15:46:40.561327Z info    ads Full push, new service istio-system/skywalking-oap.istio-system.svc.cluster.local
2021-02-18T15:46:40.661964Z info    ads Push debounce stable[18] 1: 100.125768ms since last change, 100.125573ms since last push, full=true
2021-02-18T15:46:40.663166Z info    ads XDS: Pushing:2021-02-18T15:46:40Z/7 Services:15 ConnectedEndpoints:0  Version:2021-02-18T15:46:40Z/7
2021-02-18T15:48:18.752721Z info    validationController    Reconcile(enter): update event (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) istiod-istio-system
2021-02-18T15:48:18.758575Z info    validationController    Successfully updated validatingwebhookconfiguration istiod-istio-system (failurePolicy=Fail,resourceVersion=14734618)
2021-02-18T15:48:18.758618Z info    validationController    Reconcile(enter): update event (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) istiod-istio-system
2021-02-18T15:48:18.759047Z info    validationController    validatingwebhookconfiguration istiod-istio-system (failurePolicy=Fail, resourceVersion=14734618) is up-to-date. No change required.
2021-02-18T15:48:23.008201Z info    ads Push debounce stable[19] 2: 100.159192ms since last change, 100.164101ms since last push, full=true
2021-02-18T15:48:23.009400Z info    ads XDS: Pushing:2021-02-18T15:48:23Z/8 Services:15 ConnectedEndpoints:0  Version:2021-02-18T15:48:23Z/8
2021-02-18T15:48:23.630276Z info    ads ADS: new connection for node:router~10.52.1.164~istio-ingressgateway-8446d77c49-r8rhb.istio-system~istio-system.svc.cluster.local-8
2021-02-18T15:48:23.632436Z info    ads CDS: PUSH for node:istio-ingressgateway-8446d77c49-r8rhb.istio-system resources:33 size:29.3kB
2021-02-18T15:48:23.703734Z info    ads EDS: PUSH for node:istio-ingressgateway-8446d77c49-r8rhb.istio-system resources:26 size:6.3kB empty:5 cached:18/26
2021-02-18T15:48:23.708009Z info    ads LDS: PUSH for node:istio-ingressgateway-8446d77c49-r8rhb.istio-system resources:2 size:12.4kB
2021-02-18T15:48:23.712470Z warn    ads ADS:LDS: ACK ERROR router~10.52.1.164~istio-ingressgateway-8446d77c49-r8rhb.istio-system~istio-system.svc.cluster.local-8 Internal:Error adding/updating listener(s) 0.0.0.0_8443: V2 (and AUTO) xDS transport protocol versions are deprecated in log_name: "listener_envoy_accesslog"
grpc_service {
  envoy_grpc {
    cluster_name: "envoy_accesslog_service"
  }
}
filter_state_objects_to_log: "wasm.upstream_peer"
filter_state_objects_to_log: "wasm.upstream_peer_id"
filter_state_objects_to_log: "wasm.downstream_peer"
filter_state_objects_to_log: "wasm.downstream_peer_id"
. The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition.

Not sure whether the issue originates from istio side or skywalking side. Suspected if this line of code will cause the issue https://github.com/apache/skywalking/blob/v8.4.0/oap-server/server-receiver-plugin/envoy-metrics-receiver-plugin/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/AccessLogServiceGRPCHandler.java#L22.

[wu-sheng]

This error seems Istio's side setting. The codes you are pointing to, are for supporting v2 and v3 at the same time.

[wu-sheng]

And I am not reading this error as error, is that blocking the deployment?

[wu-sheng]

@yckbilly1929 @kezhenxu94 Take a look at https://github.com/istio/istio/issues/30773

[yckbilly1929]

And I am not reading this error as error, is that blocking the deployment?

Yes, deployment is blocked. Istio-ingressgateway becomes unavailable as istio-proxy's status will always be unready after adding these 2 lines in the config:

meshConfig.enableEnvoyAccessLogService=true
meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800

I will go check again from istio's side.

[wu-sheng]

And I am not reading this error as error, is that blocking the deployment?

Yes, deployment is blocked. Istio-ingressgateway becomes unavailable as istio-proxy's status will always be unready after adding these 2 lines in the config:

meshConfig.enableEnvoyAccessLogService=true
meshConfig.defaultConfig.envoyAccessLogService.address=skywalking-oap.istio-system:11800

I will go check again from istio's side.

If this blocks the Istio deployment, I think this more likely belongs Istio's field, could you submit an issue at Istio side and linked here? @yckbilly1929

[yckbilly1929]

@wu-sheng issue submitted at https://github.com/istio/istio/issues/30939

[wu-sheng]

This has been located and fixed on Istio.