Istio: TLS connection failures after Istio 1.3 upgrade

[lhotrifork]

Bug description
After upgrading to Istio 1.3 our services are losing their database connections after what seems to be fixed intervals. This causes errors in the application, as it cannot write data using the already open connection - causing a RollBack of the transaction.
Downgrading to istio 1.2.5 resolves the problem.
Our setup is as follows.
AWS EKS.
Istio 1.3 with mTLS and global.outboundTrafficPolicy.mode=REGISTRY_ONLY.
AWS RDS Postgresql 9.4.
Pods access RDS via a VirtualService and ServiceEntry:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
spec:
  hosts:
  - fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
  - fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp-5432
    number: 5432
    protocol: tcp
  resolution: NONE

And VirtualService to allow access to the RDS instances are via SNI inspection.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
spec:
  hosts:
  - fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
  - fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
  tls:
  - match:
    - port: 5432
      sniHosts:
      - fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
    route:
    - destination:
        host: fut-trifork-rds-aurora.cluster-ro-hash.eu-west-1.rds.amazonaws.com
        port:
          number: 5432
  - match:
    - port: 5432
      sniHosts:
      - fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
    route:
    - destination:
        host: fut-trifork-rds-aurora.cluster-hash.eu-west-1.rds.amazonaws.com
        port:
          number: 5432

Logs from istio-proxy for affected services:
Service A: Custom JPA java service

[{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59968","duration":"287272","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:44.315Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59966","authority":"-","path":"-"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59926","duration":"291214","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:40.373Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59924","authority":"-","path":"-"}
,{"upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:60066","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:60068","duration":"283127","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:48.461Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:60046","duration":"285386","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:46.203Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:60044","authority":"-","path":"-"}
,{"upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59954","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59956","duration":"288130","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:43.459Z","method":"-","request_id":"-"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59922","duration":"291252","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:40.336Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59920","authority":"-","path":"-"}
,{"start_time":"2019-09-17T09:18:38.459Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-","bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59888","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59890","duration":"293131","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-"}
,{"bytes_sent":"3400","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59852","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59854","duration":"295434","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:36.155Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"1075","istio_policy_status":"-"}
,{"downstream_remote_address":"REDACTED_IP1:59816","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59818","duration":"297304","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:34.286Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"3424","istio_policy_status":"-","bytes_sent":"6445","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
,{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59806","duration":"297837","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:18:33.752Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5193","istio_policy_status":"-","bytes_sent":"5952","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59804","authority":"-","path":"-"}
,{"start_time":"2019-09-17T09:18:33.461Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"190283","istio_policy_status":"-","bytes_sent":"136744","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP1:59800","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP1:59802","duration":"298128","downstream_local_address":"REDACTED_IP2:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-"}]

Service B: Keycloak IdP

2019-09-17T08:28:58.915759Z info    FLAG: --applicationPorts="[8080,7600]"
2019-09-17T08:28:58.915794Z info    FLAG: --binaryPath="/usr/local/bin/envoy"
2019-09-17T08:28:58.915801Z info    FLAG: --concurrency="2"
2019-09-17T08:28:58.915806Z info    FLAG: --configPath="/etc/istio/proxy"
2019-09-17T08:28:58.915812Z info    FLAG: --connectTimeout="10s"
2019-09-17T08:28:58.915817Z info    FLAG: --controlPlaneAuthPolicy="MUTUAL_TLS"
2019-09-17T08:28:58.915823Z info    FLAG: --controlPlaneBootstrap="true"
2019-09-17T08:28:58.915828Z info    FLAG: --customConfigFile=""
2019-09-17T08:28:58.915832Z info    FLAG: --datadogAgentAddress=""
2019-09-17T08:28:58.915837Z info    FLAG: --disableInternalTelemetry="false"
2019-09-17T08:28:58.915845Z info    FLAG: --discoveryAddress="istio-pilot.istio-system:15011"
2019-09-17T08:28:58.915849Z info    FLAG: --dnsRefreshRate="300s"
2019-09-17T08:28:58.915854Z info    FLAG: --domain="our_ns.svc.cluster.local"
2019-09-17T08:28:58.915859Z info    FLAG: --drainDuration="45s"
2019-09-17T08:28:58.915862Z info    FLAG: --envoyAccessLogService=""
2019-09-17T08:28:58.915865Z info    FLAG: --envoyMetricsServiceAddress=""
2019-09-17T08:28:58.915869Z info    FLAG: --help="false"
2019-09-17T08:28:58.915875Z info    FLAG: --id=""
2019-09-17T08:28:58.915879Z info    FLAG: --ip=""
2019-09-17T08:28:58.915887Z info    FLAG: --lightstepAccessToken=""
2019-09-17T08:28:58.915892Z info    FLAG: --lightstepAddress=""
2019-09-17T08:28:58.915896Z info    FLAG: --lightstepCacertPath=""
2019-09-17T08:28:58.915900Z info    FLAG: --lightstepSecure="false"
2019-09-17T08:28:58.915904Z info    FLAG: --log_as_json="false"
2019-09-17T08:28:58.915908Z info    FLAG: --log_caller=""
2019-09-17T08:28:58.915913Z info    FLAG: --log_output_level="default:info"
2019-09-17T08:28:58.915917Z info    FLAG: --log_rotate=""
2019-09-17T08:28:58.915921Z info    FLAG: --log_rotate_max_age="30"
2019-09-17T08:28:58.915926Z info    FLAG: --log_rotate_max_backups="1000"
2019-09-17T08:28:58.915930Z info    FLAG: --log_rotate_max_size="104857600"
2019-09-17T08:28:58.915934Z info    FLAG: --log_stacktrace_level="default:none"
2019-09-17T08:28:58.915945Z info    FLAG: --log_target="[stdout]"
2019-09-17T08:28:58.915949Z info    FLAG: --mixerIdentity=""
2019-09-17T08:28:58.915954Z info    FLAG: --parentShutdownDuration="1m0s"
2019-09-17T08:28:58.915958Z info    FLAG: --pilotIdentity=""
2019-09-17T08:28:58.915964Z info    FLAG: --proxyAdminPort="15000"
2019-09-17T08:28:58.915968Z info    FLAG: --proxyComponentLogLevel="misc:error"
2019-09-17T08:28:58.915972Z info    FLAG: --proxyLogLevel="warning"
2019-09-17T08:28:58.915977Z info    FLAG: --serviceCluster="keycloak.our_ns"
2019-09-17T08:28:58.915981Z info    FLAG: --serviceregistry="Kubernetes"
2019-09-17T08:28:58.915985Z info    FLAG: --statsdUdpAddress=""
2019-09-17T08:28:58.915990Z info    FLAG: --statusPort="15020"
2019-09-17T08:28:58.915993Z info    FLAG: --templateFile=""
2019-09-17T08:28:58.915997Z info    FLAG: --trust-domain=""
2019-09-17T08:28:58.916003Z info    FLAG: --zipkinAddress="jaeger-collector.jaeger.svc.cluster.local:9411"
2019-09-17T08:28:58.916037Z info    Version [email protected]/istio-1.3.0-c2bd59595ce699b31d0f931885f023028ff7902b-Clean
2019-09-17T08:28:58.943630Z info    Obtained private IP [REDACTED_IP2]
2019-09-17T08:28:58.943721Z info    Proxy role: &model.Proxy{ClusterID:"", Type:"sidecar", IPAddresses:[]string{"REDACTED_IP2", "REDACTED_IP2"}, ID:"keycloak-79c6949b47-rkq54.our_ns", Locality:(*core.Locality)(nil), DNSDomain:"our_ns.svc.cluster.local", TrustDomain:"cluster.local", PilotIdentity:"", MixerIdentity:"", ConfigNamespace:"", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:labels.Collection(nil), IstioVersion:(*model.IstioVersion)(nil)}
2019-09-17T08:28:58.943744Z info    PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2019-09-17T08:28:58.944424Z info    Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istio-pilot.istio-system:15011
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: keycloak.our_ns
statNameLength: 189
tracing:
  zipkin:
    address: jaeger-collector.jaeger.svc.cluster.local:9411

2019-09-17T08:28:58.944445Z info    Monitored certs: []string{"/etc/certs/key.pem", "/etc/certs/root-cert.pem", "/etc/certs/cert-chain.pem"}
2019-09-17T08:28:58.944457Z info    waiting 2m0s for /etc/certs/key.pem
2019-09-17T08:28:58.944476Z info    waiting 2m0s for /etc/certs/root-cert.pem
2019-09-17T08:28:58.944489Z info    waiting 2m0s for /etc/certs/cert-chain.pem
2019-09-17T08:28:58.944596Z info    PilotSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-pilot-service-account"}
2019-09-17T08:28:58.947534Z info    Starting proxy agent
2019-09-17T08:28:58.947830Z info    Opening status port 15020

2019-09-17T08:28:58.948228Z info    watching /etc/certs for changes
2019-09-17T08:28:58.948241Z info    Received new config, resetting budget
2019-09-17T08:28:58.948620Z info    Reconciling retry (budget 10)
2019-09-17T08:28:58.948640Z info    Epoch 0 starting
2019-09-17T08:28:58.995583Z info    Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster keycloak.our_ns --service-node sidecar~REDACTED_IP2~keycloak-79c6949b47-rkq54.our_ns~our_ns.svc.cluster.local --max-obj-name-len 189 --local-address-ip-version v4 --allow-unknown-fields -l warning --component-log-level misc:error --concurrency 2]
[2019-09-17 08:28:59.375][17][warning][config] [external/envoy/source/server/options_impl.cc:193] --allow-unknown-fields is deprecated, use --allow-unknown-static-fields instead.
[2019-09-17 08:28:59.942][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, no healthy upstream
[2019-09-17 08:28:59.942][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:50] Unable to establish new stream
2019-09-17T08:29:00.314844Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-17T08:29:02.686142Z info    Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-09-17 08:29:02.969][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:03.026][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:05.601][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:29:05.681][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T08:29:05.938502Z info    Envoy proxy is ready
{"bytes_sent":"380382","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45437","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:43976","duration":"85433","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:39.511Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"41400","istio_policy_status":"-"}
{"duration":"4489","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:31:04.940Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52733","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55605","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:31:46.178Z","method":"-","request_id":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33211","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:32:25.920Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:33:03.264Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49435","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45225","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:33:45.970Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:34:28.333Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45403","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55973","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:35:10.447Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:35:41.326Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52953","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54037","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:36:12.330Z","method":"-","request_id":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42125","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:36:48.683Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35561","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:37:19.359Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:59319","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:37:45.857Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"start_time":"2019-09-17T08:38:20.362Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48135","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
[2019-09-17 08:39:01.187][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 08:39:01.396][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:01.399][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:03.980][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:39:04.074][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40159","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:38:53.569Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55745","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:39:37.187Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"127.0.0.6:54749","duration":"630932","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:16.427Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"11008","istio_policy_status":"-","bytes_sent":"1630862","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:54298","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"response_flags":"-","start_time":"2019-09-17T08:29:47.915Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"7954","istio_policy_status":"-","bytes_sent":"6804","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:54038","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54040","duration":"599445","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54220","duration":"591253","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:56.107Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"50656","istio_policy_status":"-","bytes_sent":"80635","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:54218","authority":"-","path":"-"}
{"downstream_remote_address":"REDACTED_IP2:54058","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:54060","duration":"598377","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:29:48.983Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"46839","istio_policy_status":"-","bytes_sent":"109279","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42353","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:39:56.699Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:40:18.921Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43153","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55957","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:40:56.963Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:41:29.230Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48199","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"start_time":"2019-09-17T08:42:08.310Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38977","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51779","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:42:46.318Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:43:26.624Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49625","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
[2019-09-17 08:44:04.179][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 08:44:04.553][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:04.556][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:06.980][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:44:07.074][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"downstream_remote_address":"REDACTED_IP2:46989","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:43:59.052Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"response_flags":"UF,URX","start_time":"2019-09-17T08:44:15.164Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:50565","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"duration":"288926","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:40:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758666","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:60416","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:53535"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:47849","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:44:43.287Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54909","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:45:06.671Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:35132","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"407","upstream_local_address":"-","duration":"408","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T08:45:35.612Z","method":"GET","request_id":"b96a034e-126e-9da6-967c-9ebf840bbdc8"}
{"upstream_local_address":"-","duration":"410","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T08:45:35.612Z","method":"GET","request_id":"2d50ff24-d14e-99e8-a3eb-a52e4e0e9362","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:35134","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"410"}
{"upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:45:37.769Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58129","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43687","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:46:05.550Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60717","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:46:37.382Z","method":"-","request_id":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:16.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52239","authority":"-","path":"-"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33501","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:32.458Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53405","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:47:59.695Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55685","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:48:43.431Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 08:49:07.179][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 08:49:07.322][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:07.326][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:10.078][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:49:10.172][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:59579","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:49:19.431Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"requested_server_name":"-","bytes_received":"1981","istio_policy_status":"-","bytes_sent":"4304","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:44344","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:44346","duration":"257828","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:35.619Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-"}
{"bytes_sent":"181114","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:43784","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:43786","duration":"286962","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:06.485Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"71237","istio_policy_status":"-"}
{"bytes_sent":"758696","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:34884","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:58615","duration":"292027","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:45:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34675","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:49:44.478Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43731","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:50:13.377Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48033","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:50:56.574Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58363","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:51:27.790Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55183","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:51:43.506Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53303","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:52:22.047Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:52:47.413Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49921","authority":"-","path":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:53:23.653Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55607","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
[2019-09-17 08:54:10.279][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 08:54:10.398][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:10.473][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:13.075][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:54:13.079][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T08:54:00.007Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53253","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:54:26.540Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:54817","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:37594","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:51887","duration":"294920","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:50:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758732"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:54:59.622Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40555","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:55:34.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33221","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
[2019-09-17 08:56:04.944][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 08:56:05.402][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:05.405][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:07.782][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 08:56:07.874][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T08:55:58.154Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:39047","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35527","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:56:38.221Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"303518","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:40302","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:55337","duration":"109727","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:55:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"2048","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46505","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:57:20.669Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:57:36.995Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:57243","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55567","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:58:02.340Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_local_address":"-","duration":"9","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.2","response_flags":"-","start_time":"2019-09-17T08:58:34.216Z","method":"GET","request_id":"7728adbe-b55b-96f2-b9f2-ae62ce916117","upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.64.219","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.64.219:0","authority":"saml.fut.trifork.com","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"8"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.75.43","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1462","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.43:0","authority":"saml.fut.trifork.com","path":"/auth/realms/master/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"215","upstream_local_address":"-","duration":"215","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.2","response_flags":"-","start_time":"2019-09-17T08:58:34.281Z","method":"GET","request_id":"be41771d-9537-9411-9e00-aecdd3aa2400"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:58:39.033Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35205","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55051","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:59:04.941Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T08:59:49.911Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40611","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:00:06.252Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40747","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"response_flags":"-","start_time":"2019-09-17T09:00:16.688Z","method":"GET","request_id":"cc4e4316-bd98-9070-bdfe-0e2bfd6004f4","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1789","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.34.96:49514","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/.well-known/uma2-configuration","protocol":"HTTP/1.1","upstream_service_time":"28","upstream_local_address":"-","duration":"28","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)"}
{"upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"72","istio_policy_status":"-","bytes_sent":"2651","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.34.96:49514","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"914","upstream_local_address":"-","duration":"915","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:21.996Z","method":"POST","request_id":"7a26ffbe-7e33-9f3e-91d8-314fde3e3a2d"}
{"downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"8","upstream_local_address":"-","duration":"9","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:25.809Z","method":"GET","request_id":"e3e42a11-7827-9ad4-b080-bbed2fdb5538","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local"}
{"bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"378","upstream_local_address":"-","duration":"379","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:25.847Z","method":"POST","request_id":"95072a05-a3d8-924f-8314-c8dfc48ea165","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-"}
{"bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"441","upstream_local_address":"-","duration":"441","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.242Z","method":"GET","request_id":"6afa974c-f86e-907b-9f15-0dfd3bb54143","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"5","upstream_local_address":"-","duration":"5","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:26.700Z","method":"GET","request_id":"649bbb76-436c-9d27-a4b8-c9fe7e0ff515","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"91","upstream_local_address":"-","duration":"91","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.760Z","method":"POST","request_id":"2af6d861-a2ad-9d3f-b71d-da876696071f","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-","bytes_sent":"1825"}
{"downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"17","upstream_local_address":"-","duration":"17","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:26.859Z","method":"GET","request_id":"6d6de6a0-b6e3-9284-a9f1-564ca1892766","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local"}
{"protocol":"HTTP/1.1","upstream_service_time":"4","upstream_local_address":"-","duration":"4","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:27.475Z","method":"GET","request_id":"274ddff2-47f0-952e-81a2-ee1fee60da88","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs"}
{"bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"122","upstream_local_address":"-","duration":"123","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.526Z","method":"POST","request_id":"0cf41600-100c-9ac7-9ef4-305a2e6bc324","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-"}
{"bytes_sent":"540","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"21","upstream_local_address":"-","duration":"22","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.657Z","method":"GET","request_id":"6683f498-32b4-9467-affe-53f3001d1a10","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44454","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"4","upstream_local_address":"-","duration":"4","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Java/11.0.4","response_flags":"-","start_time":"2019-09-17T09:00:27.708Z","method":"GET","request_id":"90dcf838-ad89-9b77-a12a-fb1687b1a47d","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-"}
{"response_flags":"-","start_time":"2019-09-17T09:00:27.775Z","method":"POST","request_id":"15130842-5e0f-91da-8bf2-f42ae6a1a66f","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"95","istio_policy_status":"-","bytes_sent":"1825","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"133","upstream_local_address":"-","duration":"133","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.75.125:44490","authority":"keycloak.our_ns","path":"/auth/admin/realms/ehealth/users/ca6139ff-0dae-4422-91ac-4cbb9c300abc","protocol":"HTTP/1.1","upstream_service_time":"15","upstream_local_address":"-","duration":"16","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Apache-HttpClient/4.5.2 (Java/11.0.4)","response_flags":"-","start_time":"2019-09-17T09:00:27.917Z","method":"GET","request_id":"6c745f50-88fd-92af-92ea-f0e1dbf0d445","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"0","istio_policy_status":"-","bytes_sent":"540"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56977","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:00:43.563Z","method":"-","request_id":"-"}
[2019-09-17 09:01:07.975][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:01:08.081][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:01:08.086][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:01:10.372203Z info    Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://127.0.0.1:15000/server_info: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2019-09-17 09:01:10.478][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:01:10.573][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:01:10.673142Z info    Envoy proxy is ready
{"duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:07.027Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52197","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:33.254Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55639","authority":"-","path":"-"}
{"bytes_sent":"758816","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:41402","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:41907","duration":"292326","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:57:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-"}
{"downstream_remote_address":"REDACTED_IP2:58898","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:58900","duration":"199439","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T08:58:34.308Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"51089","istio_policy_status":"-","bytes_sent":"42209","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:01:56.357Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45489","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:02:35.383Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36387","authority":"-","path":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58539","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:03:13.660Z","method":"-","request_id":"-"}
{"upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:03:52.751Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56261","authority":"-","path":"-","protocol":"-","upstream_service_time":"-"}
{"upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.64.219:0","authority":"saml.fut.trifork.com","path":"/auth/realms/ehealth/protocol/openid-connect/token","protocol":"HTTP/1.1","upstream_service_time":"347","upstream_local_address":"-","duration":"348","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"PostmanRuntime/7.15.2","response_flags":"-","start_time":"2019-09-17T09:04:39.855Z","method":"POST","request_id":"ea4c4f82-19b1-94f8-b66b-008b9b5a236b","upstream_host":"127.0.0.1:8080","x_forwarded_for":"85.191.124.254,10.150.64.219","requested_server_name":"outbound_.80_._.keycloak.our_ns.svc.cluster.local","bytes_received":"62","istio_policy_status":"-","bytes_sent":"2576"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:33007","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:04:36.017Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:37641","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:05:13.459Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:05:35.612Z","method":"GET","request_id":"24030340-b20d-9809-afb1-3cf57cc3211d","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:45980","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"2","upstream_local_address":"-","duration":"3","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default"}
{"start_time":"2019-09-17T09:05:35.612Z","method":"GET","request_id":"15eaf9ec-b9c6-9a8f-a2d6-19947382984e","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:45978","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"10","upstream_local_address":"-","duration":"10","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-"}
{"downstream_remote_address":"REDACTED_IP2:55527","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:05:30.766Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
[2019-09-17 09:06:10.578][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:06:10.615][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:10.618][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:12.974][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:06:12.978][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:32895","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:06:12.071Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38191","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:06:34.317Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"136362","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:04:39.889Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5155","istio_policy_status":"-","bytes_sent":"11328","upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:37478","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:37480"}
{"requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758846","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:44114","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:49777","duration":"294831","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:02:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38037","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:17.270Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48681","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:37.730Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43129","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:07:59.789Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:47021","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:08:40.069Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:41619","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:09:20.058Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:43947","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:09:55.122Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34237","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:10:11.248Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:10:38.885Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53343","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
[2019-09-17 09:11:13.079][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:11:13.237][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:13.242][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:15.576][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:11:15.672][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:49673","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:11:21.573Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"758869","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:46816","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:59963","duration":"297426","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:07:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36457","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10006","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:11:58.589Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"response_flags":"UF,URX","start_time":"2019-09-17T09:12:41.650Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:48467","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46187","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:09.464Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42705","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:27.491Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:52555","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:13:57.743Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:53757","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:14:17.473Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51633","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"9997","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:14:49.186Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35791","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:15:33.250Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56953","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:15:53.640Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:16:15.683][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:16:16.092][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:16:16.095][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:16:18.371980Z info    Envoy proxy is NOT ready: failed to get server info: failed retrieving Envoy stats: Get http://127.0.0.1:15000/server_info: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
[2019-09-17 09:16:18.380][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:16:18.473][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-09-17T09:16:18.574380Z info    Envoy proxy is ready
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56963","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:16:17.372Z","method":"-","request_id":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55795","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:16:45.723Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"upstream_cluster":"outbound|5432||fut-trifork-rds-aurora.cluster-ro-HASH.eu-west-1.rds.amazonaws.com","downstream_remote_address":"REDACTED_IP2:49116","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"REDACTED_IP2:49118","duration":"115170","downstream_local_address":"10.150.175.18:5432","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:15:06.477Z","method":"-","request_id":"-","upstream_host":"10.150.175.18:5432","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"3923","istio_policy_status":"-","bytes_sent":"4711"}
{"upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:49518","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:41719","duration":"300227","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:12:01.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5376","istio_policy_status":"-","bytes_sent":"796906"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60021","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:17:11.890Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"downstream_remote_address":"REDACTED_IP2:58899","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10011","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:17:30.788Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56157","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:18:06.637Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:18:41.270Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56171","authority":"-","path":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:34729","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:19:18.033Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:57949","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:19:59.122Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:51397","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:20:19.743Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:60499","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:20:44.023Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:21:18.575][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:21:18.900][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:18.972][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:21.278][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:21:21.374][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"start_time":"2019-09-17T09:21:24.904Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58719","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:45621","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:21:42.869Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:17:16.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"5120","istio_policy_status":"-","bytes_sent":"759108","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:52366","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:60615","duration":"288127","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-"}
{"start_time":"2019-09-17T09:22:21.314Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:56619","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35463","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:22:42.451Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:22:59.781Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:32907","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:41053","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:23:20.542Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0"}
{"duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:23:37.246Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36351","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-"}
{"downstream_remote_address":"REDACTED_IP2:42211","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:24:09.564Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46049","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:24:46.707Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
[2019-09-17 09:25:16.841][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:25:17.084][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:17.087][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:19.582][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:25:19.677][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
{"response_flags":"UF,URX","start_time":"2019-09-17T09:25:22.159Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:46273","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-"}
{"upstream_local_address":"-","duration":"20","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:25:35.612Z","method":"GET","request_id":"5aa1e428-4b46-925c-b5d0-7c6fb588f4a4","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"1450","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:56816","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/ehealth/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"19"}
{"bytes_sent":"2658","upstream_cluster":"inbound|80|http|keycloak.our_ns.svc.cluster.local","downstream_remote_address":"10.150.21.91:56814","authority":"keycloak.our_ns.svc.cluster.local","path":"/auth/realms/nemlogin/protocol/openid-connect/certs","protocol":"HTTP/1.1","upstream_service_time":"30","upstream_local_address":"-","duration":"30","downstream_local_address":"REDACTED_IP2:8080","upstream_transport_failure_reason":"-","route_name":"default","response_code":"200","user_agent":"Go-http-client/1.1","response_flags":"-","start_time":"2019-09-17T09:25:35.612Z","method":"GET","request_id":"c178f8fd-ea29-936a-8c46-8528d52b52d7","upstream_host":"127.0.0.1:8080","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:58027","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:25:38.035Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0"}
{"bytes_sent":"607347","upstream_cluster":"InboundPassthroughClusterIpv4","downstream_remote_address":"10.150.17.125:55066","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"127.0.0.6:45605","duration":"226507","downstream_local_address":"REDACTED_IP2:15090","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"-","start_time":"2019-09-17T09:22:16.420Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP2:15090","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"4096","istio_policy_status":"-"}
{"downstream_remote_address":"REDACTED_IP2:43933","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:26:19.820Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local"}
{"requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:42991","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10003","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:26:54.796Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:38733","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:27:38.326Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:01.526Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:40857","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10001","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-"}
{"bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:35657","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10002","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:20.225Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-"}
{"upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:36679","authority":"-","path":"-","protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10004","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:28:45.935Z","method":"-","request_id":"-"}
{"protocol":"-","upstream_service_time":"-","upstream_local_address":"-","duration":"10000","downstream_local_address":"REDACTED_IP1:7600","upstream_transport_failure_reason":"-","route_name":"-","response_code":"0","user_agent":"-","response_flags":"UF,URX","start_time":"2019-09-17T09:29:23.707Z","method":"-","request_id":"-","upstream_host":"REDACTED_IP1:7600","x_forwarded_for":"-","requested_server_name":"-","bytes_received":"0","istio_policy_status":"-","bytes_sent":"0","upstream_cluster":"outbound|7600||keycloak-headless.our_ns.svc.cluster.local","downstream_remote_address":"REDACTED_IP2:55843","authority":"-","path":"-"}
[2019-09-17 09:30:19.778][17][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-09-17 09:30:20.153][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-09-17 09:30:20.156][17][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/

One thing that jumps out at me is that durations for the closed are very similar (~287000 for Service A, ~10000 for Service B). Also they seem to be followed up by a reconfiguration of Envoy due to gRPC config stream closed: 13,

Affected product area (please put an X in all that apply)

[ ] Configuration Infrastructure
[ ] Docs
[ ] Installation
[x] Networking
[ ] Performance and Scalability
[ ] Policies and Telemetry
[ ] Security
[ ] Test and Release
[ ] User Experience
[ ] Developer Infrastructure

Expected behavior
Connections should not close unexpectedly

Steps to reproduce the bug

Version (include the output of istioctl version --remote and kubectl version)

$ istioctl version # after downgrade
client version: 1.2.4
citadel version: 1.2.5
galley version: 1.2.5
ingressgateway version: 1.2.5
pilot version: 1.2.5
policy version: 1.2.5
sidecar-injector version: 1.2.5
telemetry version: 1.2.5 
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.2", GitCommit:"f6278300bebbb750328ac16ee6dd3aa7d3549568", GitTreeState:"clean", BuildDate:"2019-08-05T16:54:35Z", GoVersion:"go1.12.7", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.10-eks-5ac0f1", GitCommit:"5ac0f1d9ab2c254ea2b0ce3534fd72932094c6e1", GitTreeState:"clean", BuildDate:"2019-08-20T22:39:46Z", GoVersion:"go1.11.13", Compiler:"gc", Platform:"linux/amd64"}

How was Istio installed?
Helm

Environment where bug was observed (cloud vendor, OS, etc)
AWS EKS.

[lhotrifork]

May be related to https://github.com/istio/istio/issues/15773, but I'm not sure.

[jmagnusson]

Sounds exactly like the issue we’ve been experiencing since the upgrade to Istio 1.3.1 yesterday. Although we’re connected to Google Cloud SQL via private IP (No ServiceEntry resources defined).

[chadlwilson]

Thanks for reporting this @lhotrifork - we have observed what looks like the same problem since Istio 1.2.5 -> Istio 1.3.1 upgrade. We're on EKS too; Istio installed via Helm chart. ServiceEntrys without VirtualService using CIDR ranges to allow outbound MESH_EXTERNAL traffic.

In some log analysis we also seem the same pattern (red herring or not) of the gRPC config stream closed: 13 errors; then almost exactly 45 seconds later; logs from our service complaining about network errors. It's not immediately clear what the timeouts/keepalives might be for this service; so unclear whether the proxy warning is before or after the connections are detected as dead.

[jmagnusson]

Arr you having similar issues with other external services? For us Redis also sees sporadic connectivity issues since istio 1.3.1 (through Google Cloud Memorystore)

[chadlwilson]

We've seen MySql connections being dropped and we also have Redis usage; but I haven't looked into either of them in detail with respect to this issue. Probably because on app side we have quite robust recovery mechanisms within their connection pools.

The reason I've been looking into this issue is that one of our services is a non-DB, non-cache TLS/TCP custom protocol and the app-layer connection/session pooling mechanism is not as robust as standard DB/redis pools so the Istio instability underneath the app had exposed an app-level problem for us.

To make it a bit more visual in a relatively low usage environment - the graph below from Istio Workload shows instability starting at exactly the point that we upgraded to Istio 1.3 in this environment :-(

[jmagnusson]

Ran a buckload of requests for ~20 minutes to one of our services and the SSL error ((psycopg2.OperationalError) SSL SYSCALL error: EOF detected) always popped up every five minutes (+/- a few seconds). Isn't that peculiar?

[incfly]

/cc @incfly

[howardjohn]

When looking at a user that had somewhat similar issues, I noticed they did not have container ports specified. The behavior changed in 1.3 around this -- before this meant inbound traffic bypassed Envoy, but now it does go through Envoy.

Note that this should still work, but if you guys are seeing this issue and have the same lack of container ports, then that means the problem is not Envoy 1.2 -> Envoy 1.3 behavior change, but rather No Envoy -> Envoy behavior change

We have some docs at https://istio.io/faq/traffic-management/#controlling-inbound-ports

[chadlwilson]

When looking at a user that had somewhat similar issues, I noticed they did not have container ports specified. The behavior changed in 1.3 around this -- before this meant inbound traffic bypassed Envoy, but now it does go through Envoy.

Thanks for your reply @howardjohn. I have read through that; as well as the changlog and am still struggling to see what we might tweak here.

Excuse my possible ignorance; however since we are talking about external Egress here from app -> Envoy -> MESH_EXTERNAL I didn't think containerPorts were relevant? If my understanding is correct, and the traffic wasn't going through Envoy in 1.2.x, the the sidecar/mesh wouldn't have been able to perform any kind of Egress control at all.

If it helps, our Mesh is installed with

proxy:
    # snip

    includeIPRanges: "*"
    excludeIPRanges: ""
    excludeOutboundPorts: ""

    includeInboundPorts: "*"
    excludeInboundPorts: ""

We don't have a VirtualService - only

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
spec:
  addresses:
  - REDACTED_IP1
  - REDACTED_IP2
  hosts:
  - alias.for.non.dns.external.service
  location: MESH_EXTERNAL
  ports:
  - name: tcp
    number: 2223
    protocol: tcp

I tried disabling the new outboundProtocolDetectionfeature to see if that had anything to do with this, but no luck.

Is there anything to the comment at https://github.com/istio/istio/issues/15773#issuecomment-536214875 on LDS/CDS config updates causing connections to be dropped; or something we could do to validate this theory? (noting that the linked issue was raised against a much older Istio version)

[howardjohn]

@chadlwilson good point, probably not the same issue. They were seeing similar error messages so I thought I would mention that in case others were running into that.

Maybe similar issue to https://github.com/istio/istio/issues/17383#issuecomment-535459139, can you try setting that option in pilot mentioned there. Note my warning there - pilot will use about 10x CPU when you set that

[chadlwilson]

@howardjohn thanks! I note that the semantics observed (5 to 25 minute resets) are identical to my observations (posted in Slack before I came across this ticket) so this looks promising.

I have changed this PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true setting in one of our clusters and will observe over today.

[chadlwilson]

@howardjohn This appears to have caused the problem to go away. Connections are stable since I made the suggested change in https://github.com/istio/istio/issues/17383#issuecomment-535459139

After disabling XDS_MARSHALING_TO_ANY

Before disabling

No connection drops reported by our health checks or logging since I changed the pilot configuration.

Pilot memory and CPU went up, as expected and consistent with the linked ticket.

[howardjohn]

@chadlwilson great we figured it out, but very disappointed this is the root cause did to the pilot performance impact..

As mentioned in the other issue @nrjpoddar is working on getting a fix into 1.3 so hopefully you could help verify that - I have not been able to reproduce this myself unfortunately

[jmagnusson]

Setting env var PILOT_DISABLE_XDS_MARSHALING_TO_ANY to true on the istio-pilot deployment did indeed resolve this issue.

Seeing some incredible CPU spikes every 5 minutes though in a zero traffic staging cluster. With just a request per second or so, the pilot HPA maxes out the number of pods for a minute or so, and then back down.

Looking forward to @nrjpoddar's fix so we don't have to do this in production 😅

(PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true added around 00:40)

[howardjohn]

@chadlwilson @jmagnusson on the master branch we do have this change. Can one of you try out master images? That is, gcr.io/istio-release/proxyv2:master-latest-daily and with the default PILOT_DISABLE_XDS_MARSHALING_TO_ANY=false.

I don't want to go through all the effort + waiting for a 1.3 release to find out it doesn't fix the problem.

Also if anyone has a way I could easily reproduce this on my own cluster that would be very much appreciated

[chadlwilson]

@howardjohn I've removed the PILOT_DISABLE_XDS_MARSHALING_TO_ANY override and changed to use this image in one of our envs now.

Excuse my likely ignorance; but do I need to change the proxy image injected into app pods; or just the one within pilot itself - or change mesh-wide?

[howardjohn]

Excuse my likely ignorance; but do I need to change the proxy image injected into app pods; or just the one within pilot itself - or change mesh-wide?

Into app pods. you can do this mesh wide with --set global.proxy.image if using helm or if you just want to apply to a few proxies can override with https://preliminary.istio.io/docs/reference/config/annotations.html sidecar.istio.io/proxyImage

Pilot does have a proxy next to it but its not really doing much at all, it has a static config.

[chadlwilson]

Into app pods. you can do this mesh wide with --set global.proxy.image if using helm or if you just want to apply to a few proxies can override with https://preliminary.istio.io/docs/reference/config/annotations.html sidecar.istio.io/proxyImage

Got it; thank you sir. Rolling out now.

istio-proxy:
    Image:         gcr.io/istio-release/proxyv2:master-latest-daily
    Image ID:      docker-pullable://gcr.io/istio-release/proxyv2@sha256:d722cfa118f7bac5120a249c3cbd0ca57542b61b768c2099662ccc657b62b39b

[howardjohn]

Sorry if my original message was confusing -- PILOT_DISABLE_XDS_MARSHALING_TO_ANY should be false. We have established that if we set it to true the issue is resolved, but pilot takes a major performance hit. What we want to verify is if we set it to false with the new images if its resolved as well.

Thanks for trying this out!

[chadlwilson]

Sorry if my original message was confusing -- PILOT_DISABLE_XDS_MARSHALING_TO_ANY should be false.

Ack; understood. Possibly my visual diff was confusing. Above I am deleting the workaround/override for pilot's env to go back to the default value (false) and changing the proxy image.

[howardjohn]

perfect !

[nrjpoddar]

It would be good to understand how to reproduce it and if possible we can write some tests. This issue feels like something that can prop up again if serialization is not stable.

[chadlwilson]

So far things are looking good with the daily build mentioned above after ~7 hours. No connection drops observed.

I made the change at ~ 08:55 below - (i.e the blue 87444c67d replicaset) and restarted deployments after that. Very low usage environment; and relatively small mesh; so can just see less spikey pilot CPU usage.

[denniswebb]

I'm not sure if I'm experiencing the same issue or not. After upgrading to 1.3.1 from 1.2.2, we started getting intermittent SSL failures accessing services through the istio-ingressgateway. Curl testing showed that ever 100 or so(rough estimate) requests would return curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to[REDACTED]:443

I was able to recreate this on a separate cluster running 1.3.0.

I tried all 3 combinations of the recommendations above:

1. istio-ingressgateway using nightly image
2. pilot with env var _PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true_
3. both 1 and 2 set together

Every combination still resulted in the random SSL_connect error. I'm sure it's pure coincidence, but combination 3(both solutions in place) resulted in it taking many more requests, probably 300, before the issue occurred.

For the people way smarter about Istio than myself, is it likely this is the same issue as in this issue, or something separate I should open a new issue on?

Thanks.

Edit: For simplicity, in all tests I have reduced the number of ingress-gateway pods to 1

[prune998]

Using the latest Envoy proxy image everything seems to be fine.
Proxy (Ingress Gateway) keeps beeing deconnected from Pilot without any effect on the current connexions :

istio-ingressgateway-57bcb4d647-rrjwj istio-proxy [2019-10-03 17:35:33.484][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
istio-ingressgateway-57bcb4d647-rrjwj istio-proxy [2019-10-03 17:35:33.910][20][info][upstream] [external/envoy/source/common/upstream/cds_api_impl.cc:63] cds: add 282 cluster(s), remove 3 cluster(s)

Waiting for next release of Istio which include the new Envoy now :)

[jmagnusson]

@denniswebb your app pods needs the gcr.io/istio-release/proxyv2:master-latest-daily image as well. Most easily fixed by applying the helm charts again with global.proxy.image: gcr.io/istio-release/proxyv2:master-latest-daily (if you’re using helm that is)

[jmagnusson]

Also you need to restart the pods afterwards for the new istio-proxy image to be used

[chadlwilson]

So far things are looking good with the daily build mentioned above after ~7 hours. No connection drops observed.

Things still looking good after 24 hours. No connection drops observed.

I'm not sure if I'm experiencing the same issue or not. After upgrading to 1.3.1 from 1.2.2, we started getting intermittent SSL failures accessing services through the istio-ingressgateway. Curl testing showed that ever 100 or so(rough estimate) requests would return curl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to[REDACTED]:443

@denniswebb I'm no expert, but my sense is that your issue is something different. This issue was about dropping of long-lived connections app -> sidecar -> mesh_external; whereas yours seems to be about broken connection attempts app -> ingress_proxy -> sidecar or app -> ingress_proxy

[chadlwilson]

Just a note for those keen to get this fix into their envs - looks like this might only make Istio 1.3.3 since 1.3.2 is reserved for the HIGH severity security patch to Envoy coming 8th Oct

• https://discuss.istio.io/t/upcoming-security-updates-in-istio-1-1-16-1-2-7-1-3-2/3905
• https://groups.google.com/forum/#!topic/envoy-announce/TOkx6pYFong

[lhotrifork]

I can confirm that upgrading envoy to gcr.io/istio-release/proxyv2:master-latest-daily as mentioned in https://github.com/istio/istio/issues/17139#issuecomment-537736840 seems to fix the issue. We're using the default value for PILOT_DISABLE_XDS_MARSHALING_TO_ANY
Will keep monitoring over time, but so far I've not seen any errors.

[howardjohn]

Thank you all for testing! just to be clear, our daily images are not
really suited for production usage, and change every day, so they may not
be stable. I recommended them only to test out this issue - use at your own
risk.

On Fri, Oct 4, 2019, 4:11 AM Lasse Højgaard [e-mail]
wrote:

I can confirm that upgrading envoy to
gcr.io/istio-release/proxyv2:master-latest-daily as mentioned in #17139
(comment)
https://github.com/istio/istio/issues/17139#issuecomment-537736840
seems to fix the issue. We're using the default value for
PILOT_DISABLE_XDS_MARSHALING_TO_ANY
Will keep monitoring over time, but so far I've not seen any errors.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/17139?email_source=notifications&email_token=AAEYGXKD33RKY7ZZRLGXH73QM4QGHA5CNFSM4IXOLVIKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEALKGIA#issuecomment-538354464,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEYGXNZ5UENU2BBVOD7KXTQM4QGHANCNFSM4IXOLVIA
.

[vadimeisenbergibm]

@lhotrifork So should this issue be closed?

[lhotrifork]

@vadimeisenbergibm I'm not sure how it's usually done with Istio. Should we wait until the fix is actually released or just close it now? Fine by me either way.

[vadimeisenbergibm]

@lhotrifork I think once the issue is fixed in master, it should be closed.

[lhotrifork]

@vadimeisenbergibm cool! We'll close it now then.

[kyessenov]

Did we identify the root cause of why config generation is unstable? Struct support is being removed once again, so we can no longer rely on that workaround soon.

[howardjohn]

Reopening since this is a P0 issue we need to make sure gets into a release-1.3 backport

@kyessenov https://github.com/istio/envoy/pull/109 is the fix, which seems to include a regression test for this?

[kyessenov]

That fix may or may not work depending on the root cause. The "fix" is basically doing text print out and hashing that. But any structural differences (e.g. list order changes) would still cause a reload.

[denniswebb]

@chadlwilson Thanks. I opened #17660

[lhotrifork]

@kyessenov what kind of logs would be helpful in order to figure out the root cause?

[kyessenov]

@lhotrifork Getting envoy config dump for LDS before and after observed reloads without config changes in between would help. If we look at LDS dump, we can probably find what code is responsible for producing it, and what part of it is unstable with respect to some random ordering.

[prune998]

I did that (somewhat) in the other issue : https://github.com/istio/istio/issues/17383#issuecomment-535142546

Does it dump LDS conf ? (I think so...)
Maybe it's just the way Envoy receives it, not the content ?

[lhotrifork]

@kyessenov can you use the logs from https://github.com/istio/istio/issues/17139#issuecomment-539463425 ? Otherwise I will try to get some logs for you - but it's difficult because it will impact our development cluster and in turn all development processes.

[ckcd]

update istio to 1.3.3 but still get errors: my grpc client <--> server still get some erorr_msg:OS Error and tcp Reset, and the logs of istio-proxy still has:

[2019-10-15 07:05:59.038][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-10-15 07:39:07.474][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-10-15 08:11:41.801][24][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,

[hzxuzhonghu]

@ckcd This log is normal. The grpc connection proxy----pilot disconnects at a period of 30 minutes by default

[chadlwilson]

On our low traffic envs we've removed the PILOT_DISABLE_XDS_MARSHALING_TO_ANY: true workaround and upgraded to 1.3.3 and seems to be working fine for the last 12 hours with no connection drops.

Wonder if this issue can be closed now?

[howardjohn]

Yes thanks for the verification

[sergii-s]

I'm experiencing the same problem after upgrading istio to 1.3.1
Problem still persist after upgrade to 1.3.3
My gke version 1.14.7-gke.10

For high-load applications cpu consumption looks ok

[prune998]

Upgrade to 1.3.4, that's where the issue is solved.

[sergii-s]

@prune998
Thank you, didn't notice 1.3.4 was already available ! (it's in the bottom of the page https://istio.io/news/2019/ )

[howardjohn]

Thanks, will make sure the news page gets fixed so it's at the top

[sergii-s]

Unfortunately the same for 1.3.4

Example for 6-pods application (normally 1 pod is enough to handle 120 rps, but the deployment is scaled to 6 pods due to istio cpu consumption, and 6 is a max of the HPA) :

3 pods are ok and 3 pods consuming too much cpu

[prune998]

At least the connexions are not closed anymore ?
Are you sure you updated everything (ingress, pilot, proxies) to 1.3.4 and restarted everything ?
did you removed PILOT_DISABLE_XDS_MARSHALING_TO_ANY ?

I'm testing with 4000 GRPC cnx and the Istio-proxy is "only" taking 0.3 CPU (on a single pod)

Of course this will depend on what's going on in your connexions, if using mTLS (I'm not)...

What should not happen is having 6 pods where only 3 shows high CPU...

[howardjohn]

@sergii-s I think there are two issues with the same symptoms of high CPU usage, this one (fixed) and another https://github.com/istio/istio/issues/18229. We have (at least) 2 people looking at that one as a top priority. Any info you can add there would be helpful

[sergii-s]

I didn't override PILOT_DISABLE_XDS_MARSHALING_TO_ANY settings.
I'm sure I've updated all components + rollout to re-inject sidecar . I've checked the version of all docker images, and they are all 1.3.4.
I'm not using mTLS
As for connections, I don't really know. My pods do almost nothing, just exposing some metrics to scrap by prometheus. Istio logs :

istio-proxy | 04.11.2019, 5:09:16 PM | [2019-11-04 16:09:16.403][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
-- | -- | --
istio-proxy | 04.11.2019, 4:41:15 PM | [2019-11-04 15:41:15.244][20][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,

@howardjohn thanks for update. I will check if I can add something new to #18229

[JoshuaFox]

Is this issue fixed in Envoy 1.4.3?

I see the discussion of a fix in a 1.3 patch](https://github.com/istio/istio/issues/17139#issuecomment-539148941), but I am not finding this fix (apparently commit 647aea1) in the history of v.1.4.3.

[chadlwilson]

@JoshuaFox At least from a user perspective, we have not had this issue resurface on any of the Envoy versions used by Istio 1.4.[23456789] or 1.5.[457] versions we have deployed, and we would notice it due to the way our application services and connection pooling work.

[raswinraaj]

@chadlwilson @JoshuaFox Even I am seeing the same issue on 1.4.3. I even tried by setting --set pilot.env.PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true. As soon as i restarted one of my pods which was using the ingress gateway, I got the same error. Is there any other workaround on this?

[chadlwilson]

@raswinraaj I'm not clear from your comment what your particular issue is - what is "the same error" in this context?

This issue was specifically about outgoing long-lived TLS connections that rely on L3/L4 proxying being dropped periodically by the proxy when config was being re-pushed to them in the background. There was no particular error logged in proxies or pilot; just observation from an application perspective that connections were being incorrectly dropped.

That doesn't seem to relate to ingress gateway usage?

[raswinraaj]

@chadlwilson Apologies for not stating clearly. It is primarily related to status code 503 upstream connect error or disconnect. I thought it was related to https://github.com/istio/istio/issues/17383 which in turn lead me to this ticket. I am getting this often randomly whenever a deployment of a new version of my pod happens. On recreating the Kubernetes service object the issue gets resolved or I have to wait for about 30 mins for the endpoint to be up and running again. Not sure if these two issues are related

[chadlwilson]

@raswinraaj That sounds like something rather different - perhaps some sort of endpoint discovery issue. You may want to consider asking on https://discuss.istio.io/ and sharing some details about what you have tested and your cluster configuration, Service configuration etc, and what you observe when accessing with/without istio on the client side.

[raswinraaj]

Sure thanks Chad!!

On Wed, Aug 26, 2020, 9:17 PM Chad Wilson [e-mail] wrote:

@raswinraaj https://github.com/raswinraaj That sounds like something
rather different - perhaps some sort of endpoint discovery issue. You may
want to consider asking on https://discuss.istio.io/ and sharing some
details about what you have tested and your cluster configuration, Service
configuration etc, and what you observe when accessing with/without istio
on the client side.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/istio/istio/issues/17139#issuecomment-680963127, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AL77WLT6V4LBYSD4B6RMDILSCUVBHANCNFSM4IXOLVIA
.

[courcelm]

I'm having a similar issue on Istio 1.5.8. TCP connections are dropped for various service types (SSL, postgres, redis).
I get the gRPC config stream closed: 13 error previously reported every 30 min.

I tried the PILOT_DISABLE_XDS_MARSHALING_TO_ANY=true config but I get high CPU usage and the following error in the proxy:
gRPC config stream closed: 13, grpc: error while marshaling: proto: repeated field Resources has nil element

[courcelm]

After more investigation, I'm not affected by this issue.

[omerfsen]

Also saw

2020-09-03T17:11:14.577530Z     info    Subchannel Connectivity change to CONNECTING
2020-09-03T17:11:14.577614Z     info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-03T17:11:14.577728Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc000aa2370, {CONNECTING <nil>}
2020-09-03T17:11:14.577761Z     info    Channel Connectivity change to CONNECTING
2020-09-03T17:11:14.577764Z     info    Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-09-03T17:11:14.601045Z     info    Subchannel Connectivity change to READY
2020-09-03T17:11:14.601099Z     info    pickfirstBalancer: HandleSubConnStateChange: 0xc000aa2370, {READY <nil>}
2020-09-03T17:11:14.601110Z     info    Channel Connectivity change to READY

[puneetloya]

I do see similar logs with Istio 1.6.9

[marziman]

Hi @howardjohn , @nrjpoddar @raswinraaj

I am not sure why this was closed, because
we are seeing this issue still with Istio 1.6.8.
We have connection drops and our services stop the DB connections. How can this be fixed properly (beside the pilot property with high CPU). As this is in production for us, we would be thankful if anyone can help.

Observed Behavior:

After services are inactive for a period of time (e.g 1 hour), first attempt to access mongo database fails with exception (Got socket exception on connection)

Versions:
Istio 1.6.8
K8s: 1.18

Logs are the following for istio-proxy of the service:

kubectl logs -n production serviceXYZ istio-proxy -f

es gRPC config stream closed: 13, 
2020-09-15T10:11:52.663825Z info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-15T10:11:52.664529Z info    Subchannel Connectivity change to CONNECTING
2020-09-15T10:11:52.664711Z info    Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-09-15T10:11:52.665358Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {CONNECTING <nil>}
2020-09-15T10:11:52.665429Z info    Channel Connectivity change to CONNECTING
2020-09-15T10:11:52.685936Z info    Subchannel Connectivity change to READY
2020-09-15T10:11:52.686039Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {READY <nil>}
2020-09-15T10:11:52.686054Z info    Channel Connectivity change to READY
2020-09-15T10:14:14.199478Z warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13, 
2020-09-15T10:38:57.236959Z info    Subchannel Connectivity change to CONNECTING
2020-09-15T10:38:57.237118Z info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-15T10:38:57.237074Z info    Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-09-15T10:38:57.237120Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {CONNECTING <nil>}
2020-09-15T10:38:57.237338Z info    Channel Connectivity change to CONNECTING
2020-09-15T10:38:57.275256Z info    Subchannel Connectivity change to READY
2020-09-15T10:38:57.275341Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {READY <nil>}
2020-09-15T10:38:57.275355Z info    Channel Connectivity change to READY
2020-09-15T10:47:09.502962Z warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13, 
2020-09-15T11:07:12.806011Z info    Subchannel Connectivity change to CONNECTING
2020-09-15T11:07:12.806124Z info    transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-09-15T11:07:12.806526Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {CONNECTING <nil>}
2020-09-15T11:07:12.806576Z info    Channel Connectivity change to CONNECTING
2020-09-15T11:07:12.806588Z info    Subchannel picks a new address "istiod.istio-system.svc:15012" to connect
2020-09-15T11:07:12.841464Z info    Subchannel Connectivity change to READY
2020-09-15T11:07:12.841559Z info    pickfirstBalancer: HandleSubConnStateChange: 0xc000e8aaf0, {READY <nil>}
2020-09-15T11:07:12.841617Z info    Channel Connectivity change to READY
2020-09-15T11:10:24.355084Z info    sds resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-09-15T11:10:24.359140Z warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamSecrets gRPC config stream closed: 2, resource:default Close connection to proxy "sidecar~100.65.243.177~jx-conugate-ns-dashboard-dc4647df6-z68hl.production~production.svc.cluster.local-26"
2020-09-15T11:10:24.858618Z info    sds resource:default new connection
2020-09-15T11:10:24.858977Z info    sds Skipping waiting for ingress gateway secret
2020-09-15T11:10:25.412900Z info    cache   GenerateSecret default
2020-09-15T11:10:25.413270Z info    sds resource:default pushed key/cert pair to proxy
2020-09-15T11:17:32.927403Z warning envoy config    [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 13, 

[chadlwilson]

@marziman The issue was was closed because the root cause was a specific regression introduced from 1.2.5 into 1.3.0/1.3.1 within Envoy code and then confirmed to be fixed with 1.3.4. All of that context is above, which I'd encourage you to read.

It's also worth noting that as mentioned in many of these issues gRPC config stream closed: 13 is normal with respect to the pilot connections. What was abnormal in this ticket was noticing a correlation between pilot reconnects and traffic consistently being dropped after these reconnects. I can't make out such a correlation in your logging above but perhaps I am missing something?

What you are describing sounds like a different problem that may have broadly similar observed semantics; but there are a number of reasons a connection could be dropped by a proxy, or any networked system.

This problem: connections dropped after periodic pilot config push-to-proxy regardless of usage
Your problem as described: connections dropped when they are idle/unused after what looks like a timeout

Unless you are saying something was working on Istio version x and then stopped working on 1.6.8 i.e a change in semantics that you have observed OR that the Pilot config variable fixed your issue (which seems unlikely to me), then it sounds like the issues is a different one.

If you think it's to do with idle connections (and you don't see the same behaviour when a connection is consistently used) I wonder if you have tried/considered health checks or other pings on your (presumably pooled) Mongo connections so they are not idle from a TCP perspective? If not, perhaps you should consider adding them to avoid being at the mercy of whatever TCP maxConnectionTime and/or keepAlive settings that might exist in any infrastructure between your app and MongoDB. I can't recall off the top of my head what the default semantics may be in Istio or Envoy, but I prefer to not rely on such settings in lower layers for long-lived connections if I can avoid it? Alternatively, you could consider setting tcpKeepAlives in DestinationRules to at least see if that changes the behaviour?

[marziman]

@chadlwilson ,
many thanks for your answer.
I completeley studied the whole thread.

The problem: We didnt have this issue at all with Istio 1.2.x. After our update to Istio 1.6.8 last week this started to happen. So I dont think this is on Application levle, as it was running fine almost one year long.

We only see this Connection drops after ca. 30minutes and the first requests to our services fail. The second ones, after a warmup call, so to say than work fine. We are using health checks etc., but this really seems not normal to me.

I somehow have still the feeling our issue is related to this.

BR Mehmet

[marziman]

Maybe one more thing, we have the similiar state as https://github.com/istio/istio/issues/17139#issuecomment-684118616

I am wondering if the fix for this made it to the 1.6.x branch 😞

[chadlwilson]

@marziman OK, although the way you initially described it as being when the connections were idle which is a bit different to this. If you are seeing it regardless of the connections being used/idle, and you can see a correlation to when pilot is pushing config (based on proxy logs) and then a drop shortly after that then it _might_ be something similar.

We haven't moved to 1.6.x yet (still on 1.5.x since our migrations must involve zero downtime thus have to be done carefully) so I can't speak to whether the issue might have been re-introduced there, but it seems unlikely to be a branch issue to me because the fix was in envoy itself, which master would be regularly updated, and the release branches are regularly cut from master.

The commenter in https://github.com/istio/istio/issues/17139#issuecomment-684118616 also said that they weren't affected by this issue in a subsequent comment. @courcelm - are you perhaps able to share what your issue was in the end, and how you resolved it, if it wasn't caused by this issue?

In any case, I feel it might be more productive to open & describe your issue independently with appropriate context as to your config, even if you refer to this issue as appearing similar. You may also want to describe your upgrade path and whether you restarted everything cleanly, because to my knowledge skip-version upgrades (especially over that many versions - you've moved over a year of releases there) are not really supported/validated, so depending on how you executed the upgrade I wonder if the control plane and/or proxies got into a messy state.

[a8j8i8t8]

We're on Istio 1.7.2. We're getting database connection issue to external AWS RDS instance. We're getting SSL error ((psycopg2.OperationalError) SSL SYSCALL error: EOF detected) randomly. When searched for a bit we stumbled upon this thread. We're still analysing the issue. But our initial assumption is Istio may be the reason for this error.

Is there a way to set traffic.sidecar.istio.io/excludeOutboundPorts: 5432 globally for our mesh.

_Cheers,
~Ajit_

[marziman]

Hi all,
I am really confused about the handling of this issue.
We for now solved this by a bad workaround and ping our DB every minute, which is a really shitty solution in Production.

There is a pretty similiar thread here https://github.com/istio/istio/issues/19321 and I ve the feeling both have been closed pretty fast. We really tried also to check if this is by TCP settings in OS, K8s etc. but it didnt change anything.

"Something" in Istio is currently forcing TCP disconnects and affects the communication between services.
Can you guys maybe make a double check on this, too? It shall be easily reproducable.

[howardjohn]

@marziman you are unlikely to get proper help on a year old closed issue with a completely different symptoms. I strongly suggest you open a new issue with details on your issue and how to dreproduce.

[Shahard2]

We're on Istio 1.7.2. We're getting database connection issue to external AWS RDS instance. We're getting SSL error ((psycopg2.OperationalError) SSL SYSCALL error: EOF detected) randomly. When searched for a bit we stumbled upon this thread. We're still analysing the issue. But our initial assumption is Istio may be the reason for this error.

Is there a way to set traffic.sidecar.istio.io/excludeOutboundPorts: 5432 globally for our mesh.

_Cheers, ~Ajit_

me too.
Any solution or workaround?

[a8j8i8t8]

@Shahard2 For us, using annotation traffic.sidecar.istio.io/excludeOutboundPorts is working, for now.

[Shahard2]

@Shahard2 For us, using annotation traffic.sidecar.istio.io/excludeOutboundPorts is working, for now.

cool, did you put it in Istio yaml's itself or in each deployment of your services?

[a8j8i8t8]

@Shahard2 In each deployment of our services. I don't think there is any Istio config to do this at mesh level. For that I've opened a feature request.