Envoy: Envoy + Encrypted Redis Cluster errors

[sabiurr]

Title: Envoy + Encrypted Redis Cluster errors

Description:

I have enabled encryption (Encryption in-transit (TLS)) on my AWS elasticache cluster and now it seems that envoy is stuck in PRE_INITIALIZING state. I believe this is during cluster discovery/initialization. I believe I have to enable using SSL with the redis protocol (rediss).

Does envoy support proxying to encrypted redis clusters?

Spinoff issue from an older issue -> https://github.com/envoyproxy/envoy/issues/8223

Config:

 listeners:
  - name: redis_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 1999
    filter_chains:
    - filters:
      - name: envoy.redis_proxy
        config:
          stat_prefix: egress_redis
          prefix_routes:
            catch_all_route:
               cluster: redis_cluster
          settings:
            op_timeout: 5s
  clusters:
  - name: redis_cluster
    connect_timeout: 0.25s
    dns_lookup_family: V4_ONLY
    lb_policy: CLUSTER_PROVIDED
    upstream_connection_options:
      tcp_keepalive:
        keepalive_time: 60
        keepalive_probes: 1
        keepalive_interval: 5
    hosts:
      - socket_address:
          address: 
          port_value: 6379
    cluster_type:
      name: envoy.clusters.redis
      typed_config:
        "@type": type.googleapis.com/google.protobuf.Struct
        value:
          cluster_refresh_rate: 360s
          cluster_refresh_timeout: 4s
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

Logs:

[2019-09-13 17:27:45.871][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:839] adding TLS initial cluster redis_cluster
[2019-09-13 17:27:45.871][6][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:75] cm init: adding: cluster=redis_cluster primary=1 secondary=0
[2019-09-13 17:27:45.871][6][info][config] [source/server/configuration_impl.cc:71] loading 1 listener(s)
[2019-09-13 17:27:45.871][6][debug][config] [source/server/configuration_impl.cc:73] listener #0:
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:485] begin add/update listener: name=redis_listener hash=8613620242285538431
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:57]   filter #0:
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:58]     name: envoy.redis_proxy
[2019-09-13 17:27:45.871][6][debug][config] [source/server/listener_manager_impl.cc:61]   config: {"prefix_routes":{"catch_all_route":{"cluster":"redis_cluster"}},"stat_prefix":"egress_redis","settings":{"op_timeout":"5s"}}
[2019-09-13 17:27:45.873][6][debug][config] [source/server/listener_manager_impl.cc:376] add active listener: name=redis_listener, hash=8613620242285538431, address=0.0.0.0:1999
[2019-09-13 17:27:45.873][6][info][config] [source/server/configuration_impl.cc:96] loading tracing configuration
[2019-09-13 17:27:45.873][6][info][config] [source/server/configuration_impl.cc:116] loading stats sink configuration
[2019-09-13 17:27:45.873][6][info][main] [source/server/server.cc:516] starting main dispatch loop
[2019-09-13 17:27:45.887][6][debug][connection] [source/common/network/connection_impl.cc:704] [C0] connecting to <IP>:6379
[2019-09-13 17:27:45.887][6][debug][connection] [source/common/network/connection_impl.cc:713] [C0] connection in progress
[2019-09-13 17:27:45.888][6][debug][connection] [source/common/network/connection_impl.cc:552] [C0] connected
[2019-09-13 17:27:49.891][6][debug][connection] [source/common/network/connection_impl.cc:101] [C0] closing data_to_write=0 type=1
[2019-09-13 17:27:49.891][6][debug][connection] [source/common/network/connection_impl.cc:190] [C0] closing socket: 1

Envoy Info
{ "version": "e349fb6139e4b7a59a9a359be0ea45dd61e589c5/1.11.1/Clean/RELEASE/BoringSSL", "state": "PRE_INITIALIZING", "command_line_options": { "base_id": "0", "concurrency": 4, "config_path": "/etc/envoy.yaml", "config_yaml": "", "allow_unknown_fields": false, "admin_address_path": "", "local_address_ip_version": "v4", "log_level": "debug", "component_log_level": "", "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v", "log_path": "/tmp/envoy.log", "hot_restart_version": false, "service_cluster": "", "service_node": "", "service_zone": "", "mode": "Serve", "max_stats": "0", "max_obj_name_len": "0", "disable_hot_restart": false, "enable_mutex_tracing": false, "restart_epoch": 0, "cpuset_threads": false, "file_flush_interval": "10s", "drain_time": "600s", "parent_shutdown_time": "900s" }, "uptime_current_epoch": "11s", "uptime_all_epochs": "11s" }

[HenryYYang]

Sorry for the delayed response on the other thread. Based on your observation, it does sounds like it's a bug in the Redis code. It seems that the Redis code was able to correctly read the TLS context and open the TLS connection, but somehow the connection was closed by upstream. This seems like a bug.

[HenryYYang]

Can we add the bug and help wanted tag please?

[tinder-jonathanmayehobbs]

also experiencing this bug in v1.15