Contour: Add sticky session annotation

[davecheney]

At the moment the lb_strategy for each cluster defaults to whatever Envoy chooses. We should make this configurable via an annotation on the ingress object. Specifically i've been asked if we can support sticky session load balancing.

• [ ] choose an annotation name; contour.heptio.com/lb-strategy: sounds good.
• [ ] decide if it applies to the whole vhost, or only the routes mentioned in the ingress, hence you can have several ingresses with the same vhost and alter the lb strategy on a per path basis.

[rosskukulinski]

We should probably add this feature off the IngressRoute design instead. Related to #289 which some websocket-related libraries need sticky-sessions. Moving outside 0.6.0 as non-essential to closing out IngressRoute support.

[davecheney]

I hoped that we could hide this in a load balancing strategy option. I
haven't looked to see if there is one natively supported by envoy.

On 10 August 2018 at 02:18, Ross Kukulinski [e-mail]
wrote:

We should probably add this feature off the IngressRoute design instead.
Related to #289 https://github.com/heptio/contour/issues/289 which some
websocket-related libraries need sticky-sessions. Moving outside 0.6.0 as
non-essential to closing out IngressRoute support.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/heptio/contour/issues/361#issuecomment-411814685, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA39_rX3k3aheE_zmFr4qpDEPNpcrks5uPGDSgaJpZM4TvYZ1
.

[alexbrand]

I did some initial research around this when looking into the websockets issue (#595). Found this which seems to indicate that it indeed is handled via a load balancing strategy, more specifically, a "hashing load balancer": https://groups.google.com/d/msg/envoy-users/POeaHFJI2W4/EaciXkZYBgAJ.

[kidlj]

Need this.

Envoy doc references:

https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/load_balancing#ring-hash

https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route.proto.html?highlight=cookie#route-routeaction-hashpolicy

[davecheney]

@kidlj thank you for the link. To make sure I understand what you are asking for

a. do you want ring-hash support?
b. do you want sticky session support?

From the link you provided, they don't appear to be the same thing.

[kidlj]

In my understanding, and as the ring hash document says,

A consistent hashing load balancer is only effective when protocol routing is used that specifies a value to hash on.

So ring hash(load_balancing) and route hash value(hash_policy) work together to achieve sticky session.

In short, I want sticky session.

Here's a example config that should work:

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 80
    filter_chains:
    - filters:
      - name: envoy.http_connection_manager
        config:
          codec_type: auto
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: backend
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/service/1"
                route:
                  cluster: service1
                  hash_policy:
                    connection_properties:
                      source_ip: true
              - match:
                  prefix: "/service/2"
                route:
                  cluster: service2
                  hash_policy:
                    cookie:
                      name: envoy-cookie
                      ttl: 60s
          http_filters:
          - name: envoy.router
            config: {}
  clusters:
  - name: service1
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: ring_hash
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service1
        port_value: 80
  - name: service2
    connect_timeout: 0.25s
    type: strict_dns
    lb_policy: ring_hash
    http2_protocol_options: {}
    hosts:
    - socket_address:
        address: service2
        port_value: 80
admin:
  access_log_path: "/dev/null"
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 8001

In the above configuration, service1 and service2 cluster both have lb_policy set as ring_hash. Additionally, service1 route's hash_policy is set to connection_properties which persists session by source_ip. And service2 route's hash_policy is set to cookie which achieves sticky session by inserting cookie.

[Starblade42]

I'd love some feedback on how to implement this feature, and want to make sure that I'm heading the right direction in how we'd specify this type of configuration.
(https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route.proto#route-routeaction)
(https://www.envoyproxy.io/docs/envoy/latest/api-v2/api/v2/route/route.proto.html?highlight=hash_policy#route-routeaction-hashpolicy)

It seems like specifying this on a per-k8s-service basis is correct since they map directly to a cluster, and we declare the lb-strategy on a per-k8s-service basis as well.

This is an example IngressRoute declaring the above Envoy configuration by @kidlj with sticky sessions for stickysessions.bar.com on envoy v1.8.0+

# stickysessions.ingressroute.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: stickysessions
  namespace: default
spec:
  virtualhost:
    fqdn: stickysessions.bar.com
  routes:
    - match: /service/1
      services:
        - name: service1
          port: 80
          strategy: RingHash
          hashPolicy:
            - connectionProperties:
                sourceIp: true
    - match: /service/2
      services:
        - name: service2
          port: 80
          strategy: RingHash
          hashPolicy:
            - cookie:
                name: envoy-cookie
                ttl: 60s

Since the different hash policies can all be specified for a given cluster as a list of policies for Envoy, this means that this is valid envoy and contour configuration:

# stickysessions.ingressroute.yaml
apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: stickysessions-all-hash-policies
  namespace: default
spec:
  virtualhost:
    fqdn: allhashpolicies.bar.com
  routes:
    - match: /
      services:
        - name: stickeysessionapp
          port: 80
          strategy: RingHash
          hashPolicy:
            - connectionProperties:
                sourceIp: true
              terminal: true
            - cookie:
                name: envoy-cookie
                ttl: 0s
                path: cookie/path
              terminal: true
            - header:
                headerName: userSessionHeader
              terminal: true

[kidlj]

@Starblade42 Me have the same draft design personally. And the IngressRoute level default load balancing strategy could also be ring_hash, then a hash_policy should be specified there.

[phylake]

The protobuf definition and docs already define the structure that should be mirrored here. Hash policy isn't a list of hash policies. The following is more appropriate

apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: stickysessions-all-hash-policies
  namespace: default
spec:
  virtualhost: 
    fqdn: allhashpolicies.bar.com
  routes:
  - match: /
    services:
    - name: stickeysessionapp
      port: 80
      strategy: RingHash
      hashPolicy:
        # Precisely one of header, cookie, connection_properties must be set.
        connectionProperties:
          sourceIp: true
        # Precisely one of header, cookie, connection_properties must be set.
        cookie:
          name: envoy-cookie
          path: cookie/path
          ttl: 0s
        # Precisely one of header, cookie, connection_properties must be set.
        header:
          headerName: userSessionHeader
        # look at the protobuf
        terminal: true

I'm working on this today. Please let me know if the above structure is acceptable.

[phylake]

Nope that won't work. HashPolicy is on the RouteAction, not the Cluster

Also I misread the RouteAction from my previous comment. HashPolicy _is_ a list.

I added a QUESTION inline below

apiVersion: contour.heptio.com/v1beta1
kind: IngressRoute
metadata:
  name: stickysessions-all-hash-policies
  namespace: default
spec:
  virtualhost: 
    fqdn: allhashpolicies.bar.com
  routes:
  - match: /
    # QUESTION: should this be nested under a "route" key as in https://www.envoyproxy.io/docs/envoy/v1.9.0/api-v2/api/v2/route/route.proto#route-route ?
    hashPolicy:
    - connectionProperties:
        sourceIp: true
      # Precisely one of header, cookie, connection_properties must be set.
      cookie:
        name: envoy-cookie
        path: cookie/path
        ttl: 0s
      # Precisely one of header, cookie, connection_properties must be set.
      header:
        headerName: userSessionHeader
      # look at the protobuf
      terminal: true
    services:
    - name: stickeysessionapp
      port: 80
      strategy: RingHash

[davecheney]

Thank you for #856. As I said in #856 I don't think this is the right approach.

From the user's point of view, if they have to use stick sessions, all they want to specify is the cookie to find the key in. For example, this is how nginx exposes this

https://nginx.org/en/docs/http/ngx_http_upstream_module.html#sticky_cookie

The underlying implementation using a hash policy is likely going to be the way we configure envoy do stick sessions, but we shouldn't expose any more of envoy's plumbing than is necessary.

[phylake]

all they want to specify is the cookie to find the key in

Actually we want consistent hashing on source IP. I know of a use case that would want header-based stickiness too.

[rohandvora]

I can work on this issue if no one else is working on it.

[davecheney]

You got it. Please be sure to review #856

[rohandvora]

@davecheney are we supporting all three use cases:

1. Cookie
2. Header
3. Source IP

or just cookie-based stickiness?

[davecheney]

I'd say design towards supporting all three, but if possible start by implementing something that is needed now.

[davecheney]

Due to the need to ship Contour 0.11 quickly to address the security issue in Envoy 1.9.0 this issue has been bumped to the 0.12 milestone.

@shabx please be aware that the milestone for this issue has change and currently 0.12 has no due date.

[davecheney]

Sadly looking at the work to be done on this ticket, specifically no agreed on design and limited bandwidth to complete these design and implementation, by the end of the 0.12 cycle I have decided to remove this issue from the 0.12 milestone. We'll continue to progress #937 in the background, but this won't be delivered fro 0.12.

[kidlj]

Regarding to exposing this feature as an annotation to IngressRoute object:

Annotations are flat, they don't express structures. Say we want different paths in an IngressRoute having their own sticky session methods, how can we achieve that with annotations?

Or should we add this annotation to k8s services?

[davecheney]

Closing as duplicate of #1006 which will implement this feature for Ingressroute