Ingress-nginx: /etc/nginx/nginx.conf don't apply changes when ingress's annotations update

[dengqingpei1990]

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):
No
What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.):

annotations change

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: local virtualbox vm with 1 master 2 workers
• OS (e.g. from /etc/os-release): CentOS 7.4
• Kernel (e.g. uname -a): Linux master1 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: kubeadm 1.9
• Others:

What happened:
I have wrote a nginx-ingress.yaml as follows:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* /aaaa {
        proxy_pass  http://nginx-service.default:80;
      }
spec:
  rules:
  - host: registry.example.com
````
`kubectl apply -f nginx-ingress.yaml`. And I see `/etc/nginx/nginx.conf` in ingress-controller pod being update.  The `location{}` server-snippet is added to the server configuration block. It works as expect.
Then I update `nginx-ingress.yaml`:
```yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      location ~* /bbbb {
        proxy_pass  http://nginx-service.default:80;
      }
spec:
  rules:
  - host: registry.example.com

location ~* /aaaa changes to location ~* /bbbb. Run kubectl apply -f nginx-ingress.yaml. We can see the ingress object being changed by kubectl get ingress -o yaml. But the configuration (/etc/nginx/nginx.conf) don't apply this change. It still shows location ~* /aaaa
What you expected to happen:
nginx configuration file(/etc/nginx/nginx.conf) in ingress controller apply my changes on ingress annotations.

How to reproduce it (as minimally and precisely as possible):

1. create a simple deployment (simple nginx)
2. create a svc point to that deployment (nginx-service)
3. create an ingress with annotations as above.
4. update ingress annotations and watch if nginx configuration file(/etc/nginx/nginx.conf) in ingress controller changed.

Anything else we need to know:
Every step as described above do not cause any failure. Ingress controller's log shows that
Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"nginx-ingress", UID:"73a54956-6d89-11e8-a186-080027e6c129", APIVersion:"extensions", ResourceVersion:"363720", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/nginx-ingress

[antoineco]

You're right, ServerSnippet is not compared for equality inside https://github.com/kubernetes/ingress-nginx/blob/master/internal/ingress/types_equals.go.

/kind bug

[ObviousDWest]

Version 0.29

Seems pretty serious. I just added a body-size change in an Ingress spec that was never reflected into the controller pod (it wasn't restarted, nginx config wasn't updated):

metadata:
  name: "a28-ingress-service"
  annotations:
    # Allow 50MB file uploads to pass through the Ingress (e.g. during testing)
    nginx.org/client-max-body-size: "50m"

I'm assuming that if I made as similar change using the Ingress ConfigMap, it would work better? Assuming those changes are periodically looked for, and we don't have to restart the controller pod to get that read. I'm a bit new to this. Is the nginx-configuration ConfigMap in the ingress-nginx namespace a probable workaround, and are edits there picked up?

[antoineco]

@ObviousDWest it seems that you were reading the wrong documentation, because this annotation is not supported: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/

[ObviousDWest]

Thanks, yes, I've switched to using

    nginx.ingress.kubernetes.io/proxy-body-size: "48m"

(from here: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#custom-max-body-size) and had more luck.

What confused me were these links:
https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/
https://docs.nginx.com/nginx-ingress-controller/configuration/ingress-resources/advanced-configuration-with-annotations/

I must have scroll-blindness, because I found it very hard to see any difference between what was being referred to by nginx.com (ingress controller) vs kubernetes/ingress-nginx. Are there two competing controllers with similar names?

[antoineco]

@ObviousDWest yes, those are two unrelated projects. The one you were referring to in your original question is developed by NGINX inc., while the project we are currently discussing is maintained by the Kubernetes community.