Ingress-nginx: Rewrite-target annotation does not get picked up

[boazj]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

NGINX Ingress controller version:
0.10.0

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa46
3b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"windows/amd64"
}                                                                                                                        
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa46
3b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} 

Environment:
Baremetal

• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release): Ubuntu server 16.04.3
• Kernel (e.g. uname -a):
• Install tools: kubeadm
• Others: Weave

What happened:
Applied the following ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress1
  namespace: ns1
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: false
    nginx.ingress.kubernetes.io/force-ssl-redirect: false
spec:
  rules:
  - host: k8s.local
    http:
      paths:
      - path: /path
        backend:
          serviceName: service1
          servicePort: port1

The route works, but checking the access log on the server behind service1 - the rewrite never happened.
I've checked nginx.conf and indeed the location was created as /path instead of the pattern based location.
I've raised the log level using --v=5 (per ingress-nginx/internal/ingress/annotations/annotations.go it does some logging of the parsed values)
The log value is
I0123 22:07:23.498566 7 annotations.go:134] annotation Rewrite in Ingress ns1/ingress1: &{ false true false }

Per the Config type in ingress-nginx/internal/ingress/annotations/rewrite/main.go it means that the Target field is empty though the Ingress has "/".

Unfortunately I'm not sure how to progress to the next debugging step and decipher what's going on
Any assistance is appriciated

P.S, it worked once, with the same manifest, same controller, and then stopped.

What you expected to happen:
The location in nginx.conf should be pattern based per ingress-nginx/internal/ingress/controller/template/template.go buildLocation
And should have a rewrite instruction.

How to reproduce it (as minimally and precisely as possible):
See description above

Anything else we need to know:
Thanks!

[aledbf]

@boazj please use quay.io/aledbf/nginx-ingress-controller:0.317

[boazj]

I0124 17:40:05.485271 7 annotations.go:134] annotation Rewrite in Ingress home/home-ingress: &{ false true false }
Still
I've removed the ingress and re-added it, same result.

This is the pod description

Containers:
  nginx-ingress-controller:
    Container ID:  docker://8919e34da0904c34823cd2820ac94e7f8c7d04af521c816e0abddbb6f21a0f7d
    Image:         quay.io/aledbf/nginx-ingress-controller:0.317
    Image ID:      docker-pullable://quay.io/aledbf/nginx-ingress-controller@sha256:87698328ab013fea0d812223c73f4589280a8588a23b9458a2eb0ade72b43916
    Ports:         80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --annotations-prefix=nginx.ingress.kubernetes.io
      --v=5

I also see the following:
I0124 17:42:40.975375 7 main.go:47] annotation kubernetes.io/ingress.class is not present in ingress ns1/ingress1
even though I have kubernetes.io/ingress.class: "nginx" defined on the ingress

[boazj]

Found the issue, it's unrelated to ingress-nginx
I noticed that when I do kubectl get ingress -o json I don't see the annotation defined in the manifest so I dug around a bit.
I've added the ingress using kubectl apply -f <mf> which apparently does less validations then kubectl create -f <mf> - not sure why.
Apparently I needed to add quotes around all the annotation values

    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: "/"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"

This contradicts every documentation I've seen, but at least now we know :)

@aledbf thanks for all your help!