Ingress-nginx: Session affinity doesn't work for canaries

[magicyuli]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): canary sticky session affinity

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

NGINX Ingress controller version:

0.22.0

Kubernetes version (use kubectl version):

v1.5.4

Environment:

• Cloud provider or hardware configuration: No cloud provider
• OS (e.g. from /etc/os-release): Ubuntu 14.04.5 LTS, Trusty Tahr
• Kernel (e.g. uname -a): Linux 4.9.16-040916-generic x86_64
• Install tools:
• Others:

What happened:
According to the document, all non-canary annotations will be inherited from the main ingress, which I assume include the annotations for session affinity. My main ingress has following annotations:

kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/affinity: cookie
nginx.ingress.kubernetes.io/session-cookie-name: route
nginx.ingress.kubernetes.io/session-cookie-hash: sha1

and my canary ingress has following annotations:

kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: X-Canary

What's happening is the route cookie is only added to the requests sent to the non-canary pool, not to the those sent to the canary pool, which causes session affinity to not work for the canary requests.

What you expected to happen:
Session affinity works for canary ingresses.

How to reproduce it (as minimally and precisely as possible):
Deploy a non-canary service with session affinity annotations, and its canary counterpart and set nginx.ingress.kubernetes.io/canary-weight: "100", send requests to the service, and you can see the session affinity cookie is not set.

Anything else we need to know:

[mau21mau]

I'm facing the exact same issue. Any updates on this?

[kidlj]

Me facing the same issue.

[ElvinEfendi]

Have you tried setting nginx.ingress.kubernetes.io/affinity: cookie in the canary ingress itself?

[kidlj]

Have you tried setting nginx.ingress.kubernetes.io/affinity: cookie in the canary ingress itself?

I've tried setting the same set nginx.ingress.kubernetes.io/affinity: cookie annotations both on the main ingress and the canary ingress, the main ingress behaves as expected, but the canary ingress seems to ignore the affinity annotations.

$ while sleep 0.5; do curl --cookie "route=hello" "test.192.168.2.20.iv.com";echo; done
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-hs5fn"}
{"v2.2 hostname":"app-new-5fb7b7b7d5-tgjgf"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}
{"v1 hostname":"app-old-945c649fb-j6qsl"}

[magicyuli]

Just FYI, we settled on using nginx.ingress.kubernetes.io/upstream-hash-by with cookie. Achieves the same thing, but suboptimal since the canary service would be slightly different from the main service.

[myLoverLiesUnder]

Just FYI, we settled on using nginx.ingress.kubernetes.io/upstream-hash-by with cookie. Achieves the same thing, but suboptimal since the canary service would be slightly different from the main service.

Hi, now i also want to use canary release with sticky session , can you share how to using nginx.ingress.kubernetes.io/upstream-hash-by with cookie to figure it out ?
Thank you ~

[chris-ng-scmp]

Have the same issue, and still not working in version 0.25.0

[rmuehlbauer]

I'm also trying to setup canary testing with sticky sessions - any news on this??

[zelenoff]

Have the same issue(

[gaius-qi]

Have the same issue (_)

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[chris-ng-scmp]

This is sad the ingress doesn't support the feature

[benpettman]

Will this be fixed?

/reopen