Ingress-nginx: ssl certif cannot be find by nginx ?

[mmaryo]

Hello

I would like to install Nginx Ingress Controller on a new Kubernates 1.13 cluster on EKS

First, I install Kong Ingress Controller for applications. And it' works
Then, I install Nginx Ingress Controller to redirect to Keycloak (the only application behind)
Then I remove Nginx Ingress Controller
And reinstall

I follow instructions :

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/service-l4.yaml

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/patch-configmap-l4.yaml

Then I notice nginx pod not start
in logs :

W0805 11:03:55.445795       6 queue.go:130] requeuing default/keycloak, err 
-------------------------------------------------------------------------------
Error: exit status 1
2019/08/05 11:03:55 [emerg] 107#107: unknown "proxy_alternative_upstream_name" variable
nginx: [emerg] unknown "proxy_alternative_upstream_name" variable
nginx: configuration file /tmp/nginx-cfg954580780 test failed

-------------------------------------------------------------------------------
E0805 11:03:55.560411       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
E0805 11:03:58.354992       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
I0805 11:03:58.368576       6 main.go:154] Received SIGTERM, shutting down
I0805 11:03:58.368615       6 nginx.go:402] Shutting down controller queues
I0805 11:03:58.368635       6 status.go:117] updating status of Ingress rules (remove)
I0805 11:03:58.389674       6 status.go:136] removing address from ingress status ([a4ffc19efb77011e9a82d0ae7a18edce-72790691.ap-northeast-2.elb.amazonaws.com])
I0805 11:03:58.395317       6 status.go:309] updating Ingress default/ingress-keycloak status from [{ a4ffc19efb77011e9a82d0ae7a18edce-72790691.ap-northeast-2.elb.amazonaws.com}] to []
I0805 11:03:58.401469       6 nginx.go:418] Stopping NGINX process
2019/08/05 11:03:58 [notice] 108#108: signal process started
I0805 11:04:01.427543       6 nginx.go:431] NGINX process has stopped
I0805 11:04:01.431543       6 main.go:162] Handled quit, awaiting Pod deletion
E0805 11:04:05.560211       6 checker.go:41] healthcheck error: Get http+unix://nginx-status/healthz: dial unix /tmp/nginx-status-server.sock: connect: no such file or directory
^C

I try to change Nginx version from 0.25.0 to 0.24.1

-> The pod start and work. But there is an error in logs :

W0805 11:13:30.344035       7 queue.go:130] requeuing &ObjectMeta{Name:sync status,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,ManagedFields:[],}, err services "ingress-nginx" not found

Have you an idea ?

Regards

[aledbf]

nginx: [emerg] unknown "proxy_alternative_upstream_name" variable

Are you using a custom template? Please check the template in 0.25.0 includes a new variable proxy_alternative_upstream_name

[mmaryo]

I just run the 3 commands

Maybe the problem is because I deleted nginx ingress and run again the 3 commands ?

[mmaryo]

Unfortunatly I reinstall nginx ingress on new cluster and I have same error

kubectl get pod

NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-748cd7b559-5qkhs   1/1     Running   0          30m       

kubectl logs -f nginx-ingress-controller-748cd7b559-5qkhs

-------------------------------------------------------------------------------
W0807 11:53:09.228778       7 queue.go:130] requeuing default/ingress-keycloak, err 
-------------------------------------------------------------------------------
Error: exit status 1
2019/08/07 11:53:09 [emerg] 1470#1470: unknown "proxy_alternative_upstream_name" variable
nginx: [emerg] unknown "proxy_alternative_upstream_name" variable
nginx: configuration file /tmp/nginx-cfg559534501 test failed

Do you know what can I do ?

[mmaryo]

I have all stacktrace

"Error obtaining X.509 certificate: no object matching key "keycloak/mbq-certs" in local store"

Maybe because of ssl certificat ?

My tls secret

kubectl get secret -n default

NAME                  TYPE                                  DATA   AGE
default-token-nr4kh   kubernetes.io/service-account-token   3      24h
mbq-certs             kubernetes.io/tls                     2      25m

My keycloak ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-keycloak
  namespace: keycloak
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: mbq-certs
spec:
  tls:
    - hosts:
        - auth-staging.mbq.app
      secretName: mbq-certs
  rules:
    - host:
      http:
        paths:
          - path: "/"
            backend:
              serviceName: keycloak
              servicePort: http

I tried the tls secret as "mbq-certs" and "default/mbq-certs"

And the Nginx log

➜ k logs -f nginx-ingress-controller-748cd7b559-7g9l4

W0807 12:30:45.729615       6 flags.go:221] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
-------------------------------------------------------------------------------
NGINX Ingress controller
  Release:    0.25.0
  Build:      git-1387f7b7e
  Repository: https://github.com/kubernetes/ingress-nginx
-------------------------------------------------------------------------------

nginx version: openresty/1.15.8.1
W0807 12:30:45.734086       6 client_config.go:541] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0807 12:30:45.734312       6 main.go:183] Creating API client for https://10.100.0.1:443
I0807 12:30:45.743688       6 main.go:227] Running in Kubernetes cluster version v1.13+ (v1.13.8-eks-a977ba) - git (clean) commit a977bab148535ec195f12edc8720913c7b943f9c - platform linux/amd64
I0807 12:30:46.000388       6 main.go:102] Created fake certificate with PemFileName: /etc/ingress-controller/ssl/default-fake-certificate.pem
E0807 12:30:46.001846       6 main.go:131] v1.13.8-eks-a977ba
W0807 12:30:46.001991       6 main.go:106] Using deprecated "k8s.io/api/extensions/v1beta1" package because Kubernetes version is < v1.14.0
I0807 12:30:46.024135       6 nginx.go:275] Starting NGINX Ingress controller
I0807 12:30:46.047401       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"udp-services", UID:"b27e26bf-b90b-11e9-aaa8-0a9f1134a3e8", APIVersion:"v1", ResourceVersion:"147567", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/udp-services
I0807 12:30:46.047442       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"tcp-services", UID:"b2741b20-b90b-11e9-aaa8-0a9f1134a3e8", APIVersion:"v1", ResourceVersion:"147566", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/tcp-services
I0807 12:30:46.048093       6 event.go:258] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"nginx-configuration", UID:"b2695fea-b90b-11e9-aaa8-0a9f1134a3e8", APIVersion:"v1", ResourceVersion:"147955", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/nginx-configuration
I0807 12:30:47.128020       6 store.go:350] ignoring add for ingress vor based on annotation kubernetes.io/ingress.class with value 
I0807 12:30:47.128665       6 backend_ssl.go:66] Adding Secret "default/mbq-certs" to the local store
W0807 12:30:47.129303       6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "ingress-nginx/default/mbq-certs" in local store
E0807 12:30:47.129613       6 annotations.go:191] error reading CertificateAuth annotation in Ingress keycloak/ingress-keycloak: Location denied, reason: invalid format (namespace/name) found in 'mbq-certs'
W0807 12:30:47.129775       6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "keycloak/mbq-certs" in local store
E0807 12:30:47.129922       6 annotations.go:191] error reading CertificateAuth annotation in Ingress default/ingress-keycloak: Location denied, reason: invalid format (namespace/name) found in 'mbq-certs'
I0807 12:30:47.130811       6 store.go:350] ignoring add for ingress messaging-app based on annotation kubernetes.io/ingress.class with value 
I0807 12:30:47.130854       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"ingress-keycloak", UID:"6985737a-b908-11e9-b46d-06ca9fe17fde", APIVersion:"extensions/v1beta1", ResourceVersion:"150674", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/ingress-keycloak
I0807 12:30:47.130881       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"ingress-nginx", Name:"ingress-keycloak", UID:"44783309-b90d-11e9-aaa8-0a9f1134a3e8", APIVersion:"extensions/v1beta1", ResourceVersion:"150675", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress ingress-nginx/ingress-keycloak
I0807 12:30:47.130908       6 event.go:258] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"keycloak", Name:"ingress-keycloak", UID:"c622b978-b90d-11e9-b46d-06ca9fe17fde", APIVersion:"extensions/v1beta1", ResourceVersion:"150739", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress keycloak/ingress-keycloak
I0807 12:30:47.224814       6 nginx.go:319] Starting NGINX process
I0807 12:30:47.224860       6 leaderelection.go:235] attempting to acquire leader lease  ingress-nginx/ingress-controller-leader-nginx...
W0807 12:30:47.225770       6 controller.go:878] Service "default/keycloak" does not have any active Endpoint.
W0807 12:30:47.225795       6 controller.go:878] Service "ingress-nginx/keycloak" does not have any active Endpoint.
I0807 12:30:47.225916       6 controller.go:133] Configuration changes detected, backend reload required.
I0807 12:30:47.230135       6 status.go:86] new leader elected: nginx-ingress-controller-748cd7b559-tsmzk
E0807 12:30:47.262947       6 controller.go:145] Unexpected failure reloading the backend:

-------------------------------------------------------------------------------
Error: exit status 1
2019/08/07 12:30:47 [emerg] 36#36: unknown "proxy_alternative_upstream_name" variable
nginx: [emerg] unknown "proxy_alternative_upstream_name" variable
nginx: configuration file /tmp/nginx-cfg092283618 test failed

[mmaryo]

I rename the title because the bug seems to be because of nginx not found ssl certificat

[aledbf]

Closing. Fixed in master. Please use quay.io/kubernetes-ingress-controller/nginx-ingress-controller:dev to test it.

[mmaryo]

I unfortunately switch to Ambassador
thanks for fix