Ingress-nginx: Dynamic configuration causes caching problems due to default proxy_cache_key

[ostkrok]

Is this a request for help?
No

What keywords did you search in NGINX Ingress controller issues before filing this one?
cache, proxy_cache_key

---

Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT

NGINX Ingress controller version:
0.20.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.7-gke.6", GitCommit:"9b635efce81582e1da13b35a7aa539c0ccb32987", GitTreeState:"clean", BuildDate:"2018-08-16T21:33:47Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: GKE
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

What happened:

We upgraded our nginx from 0.15.0 to 0.20.0 and ran into some caching problems due to the new default dynamic configuration.

We started to see some really weird caching behaviour where a page from one host was being served on a different one, seemingly at random.

What you expected to happen:

I expected the our cache to continue working normally.

How to reproduce it (as minimally and precisely as possible):

I'm guessing this should be easily reproducable by creating two hosts and enabling the following settings in nginx:

• proxy_cache
• proxy_cache_valid

Anything else we need to know:

I've tried to figure out why this happened and I think I've found the problem to be that the default proxy_cache_key ($scheme$proxy_host$request_uri) is incompatible with the new dynamic configuration.

The variable $proxy_host will always be set to http://upstream_balancer regardless what site is being proxied, which causes responses from different hosts to be saved to the cache with the same key.

We've solved this on our end by setting proxy_cache_key $scheme$proxy_upstream_name$request_uri;, which should work since $proxy_upstream_name always seem to get set by the ingress-controller.

So the "bug" here seems to be that this ingress controller is now incompatible with the default value of proxy_cache_key, so I would suggest that we explicitly set the value of proxy_cache_key to something that work, perhaps $scheme$proxy_upstream_name$request_uri.

[ElvinEfendi]

@ostkrok ingress-nginx does not configure those directives at the moment. And I'm guessing you're enabling proxy caching using custom snippet. I'm not sure how to make this better for users other than introducing another configmap/annotation to configure proxy caching automatically.

[ostkrok]

@ElvinEfendi That's correct, we're enabling the caching using the custom http snippet.

I guess you're right. Simply setting proxy_cache_key in the template would cause problems for people trying to override it using a custom snippet.

Maybe adding some documentation on how to enable caching would keep others from running in to the same problems?

[ElvinEfendi]

Maybe adding some documentation on how to enable caching would keep others from running in to the same problems?

This would help indeed!

[ostkrok]

I'll try to find some time to create a pull request with some example on how to enable caching correctly then.

[chrono2002]

I'll try to find some time to create a pull request with some example on how to enable caching correctly then.

paste here, please

[ostkrok]

I'll try to find some time to create a pull request with some example on how to enable caching correctly then.

paste here, please

Hi,

We got our cache working correctly by adding the following to under http-snippet:

      proxy_cache_path /tmp/nginx_cache levels=1:2 keys_zone=nginx_cache:10m max_size=500M inactive=5h use_temp_path=off;
      proxy_cache nginx_cache;
      proxy_cache_valid 200 1m;
      proxy_cache_key $scheme$proxy_upstream_name$request_uri;