Ingress-nginx: ca bundle corrupted

Created on 28 Sep 2018  路  4Comments  路  Source: kubernetes/ingress-nginx

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/.):

What keywords did you search in NGINX Ingress controller issues before filing this one? (If you have found any duplicates, you should instead reply there.): ca bundle

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug report

NGINX Ingress controller version:
chart: 0.28.2
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
tag: 0.17.1

Kubernetes version (use kubectl version):
Server Version: v1.9.3

Environment:

  • Cloud provider or hardware configuration: AWS
  • OS (e.g. from /etc/os-release): CoreOS
  • Kernel (e.g. uname -a): Linux ip-10-30-36-21.ec2.internal 4.14.59-coreos-r2 #1 SMP Sat Aug 4 02:49:25 UTC 2018 x86_64 Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz GenuineIntel GNU/Linux
  • Install tools: kube-aws and helm
  • Others:

What happened:
I created a tls secret and a pod, service, and ingress. I attempted to curl the ingress and received an error (curl: (60) SSL certificate problem: unable to get local issuer certificate). When I copied the CA bundle cert and checked the subject of each of the certs, I found that one of the Go Daddy CA certs had been replaced with my wildcard cert.

What you expected to happen:
I expect the CA bundle to match what I put into my secret and I expect the curl to succeed, without using the --insecure|-k option.

How to reproduce it (as minimally and precisely as possible):

old environment

I had some incorrectly formatted secrets. I deleted them, along with the nginx ingress and my foo-svc chart.

  1. kubectl delete secret example-tls
  2. helm delete --purge nginx
  3. helm delete --purge foo

current environment

  1. Download ca bundle and cert from go daddy in apache format
  2. Base64 encoded key and cert, create secret

    • kubectl apply -f tls.yaml

  3. Create ingress, service, and deployment which use the tls secret (I used a helm chart)
  4. Attempt to curl the ingress, an error is returned

    • curl: (60) SSL certificate problem: unable to get local issuer certificate

  5. Copy the bundle from the pod

    • kubectl cp POD:/etc/ingress-controller/ssl/default-example-tls-full-chain.pem /tmp

  6. Break up the bundle into individual files, check subjects of each cert, I find that one of the go daddy root certs has been swapped out with my wildcard cert

    • subject= /OU=Domain Control Validated/CN=*.example.com

    • subject= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2

    • subject= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./CN=Go Daddy Root Certificate Authority - G2

Anything else we need to know:

tls.yaml

apiVersion: v1
kind: Secret
metadata:
  name: example-tls
type: Opaque
data:
  tls.crt: BASE_64_CRT
  tls.key: BASE_64_KEY

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/whitelist-source-range: |
      10.10.0.0/16
  creationTimestamp: 2018-09-26T21:34:17Z
  generation: 1
  labels:
    app: svc
    chart: svc-0.1.0
    heritage: Tiller
    release: foo
  name: foo-svc
  namespace: default
  resourceVersion: "37226655"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/foo-svc
  uid: ec1198cb-c1d3-11e8-89dc-0ef587d6ea0e
spec:
  rules:
  - host: foo.example.com
    http:
      paths:
      - backend:
          serviceName: foo-svc
          servicePort: http
  tls:
  - hosts:
    - foo.example.com
    secretName: example-tls
status:
  loadBalancer:
    ingress:
    - {}

service.yaml

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: 2018-09-26T21:34:17Z
  labels:
    app: svc
    chart: svc-0.1.0
    heritage: Tiller
    release: foo
  name: foo-svc
  namespace: default
  resourceVersion: "37021867"
  selfLink: /api/v1/namespaces/default/services/foo-svc
  uid: ec0ed168-c1d3-11e8-89dc-0ef587d6ea0e
spec:
  clusterIP: 10.31.0.164
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: svc
    release: foo
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
  creationTimestamp: 2018-09-26T21:34:17Z
  generation: 1
  labels:
    app: svc
    chart: svc-0.1.0
    heritage: Tiller
    release: example
  name: example-svc
  namespace: default
  resourceVersion: "37021957"
  selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/example-svc
  uid: ec1006da-c1d3-11e8-89dc-0ef587d6ea0e
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: svc
      release: example
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: svc
        release: example
    spec:
      containers:
      - command:
        - ./entrypoint.sh
        image: quay.io/example/foo:latest
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: svc
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 54321
          name: jmx
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: http
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: pull-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: 2018-09-26T21:34:44Z
    lastUpdateTime: 2018-09-26T21:34:44Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: 2018-09-26T21:34:17Z
    lastUpdateTime: 2018-09-26T21:34:44Z
    message: ReplicaSet "example-svc-677fb7b4bf" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1
picture flah00

Most helpful comment

@aledbf
That fixed it, thanks very much for you help!
Sorry, I assumed this was not a support issue. Are there docs on ca bundles, which I overlooked?

picture flah00 on 28 Sep 2018
👍5

All 4 comments

@flah00 please post the ingress controller pod logs

aledbf picture aledbf on 28 Sep 2018 
NGINX Ingress controller
  Release:    0.17.1
  Build:      git-12f7966
  Repository: https://github.com/kubernetes/ingress-nginx.git
-------------------------------------------------------------------------------

I0928 00:06:47.337641       7 flags.go:180] Watching for Ingress class: nginx
nginx version: nginx/1.13.12
W0928 00:06:47.340959       7 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0928 00:06:47.341143       7 main.go:191] Creating API client for https://10.31.0.1:443
I0928 00:06:47.353299       7 main.go:235] Running in Kubernetes cluster version v1.9 (v1.9.3) - git (clean) commit d2835416544f298c919e2ead3be3d0864b52323b - platform linux/amd64
I0928 00:06:47.356547       7 main.go:100] Validated ingress/ingress-nginx-ingress-default-backend as the default backend.
I0928 00:06:48.985328       7 nginx.go:255] Starting NGINX Ingress controller
I0928 00:06:49.062450       7 event.go:221] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"ingress-nginx-ingress-controller", UID:"6343f0b2-c2b2-11e8-89dc-0ef587d6ea0e", APIVersion:"v1", ResourceVersion:"37231367", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/ingress-nginx-ingress-controller
I0928 00:06:50.087633       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"grafana", UID:"41791d41-c026-11e8-89dc-0ef587d6ea0e", APIVersion:"extensions/v1beta1", ResourceVersion:"37231031", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress monitoring/grafana
I0928 00:06:50.088413       7 backend_ssl.go:68] Adding Secret "monitoring/example-tls" to the local store
I0928 00:06:50.088557       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"spinnaker-spinnaker-char-gate", UID:"f1e7097b-7335-11e8-b0a6-0ef468f8fc50", APIVersion:"extensions/v1beta1", ResourceVersion:"37231034", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress default/spinnaker-spinnaker-char-gate
I0928 00:06:50.089084       7 backend_ssl.go:68] Adding Secret "default/example-tls" to the local store
I0928 00:06:50.089271       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"prometheus-server", UID:"66efb171-c023-11e8-89dc-0ef587d6ea0e", APIVersion:"extensions/v1beta1", ResourceVersion:"37231035", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress monitoring/prometheus-server
I0928 00:06:50.089891       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"kibana-kibana-chart", UID:"0fc4aa3b-a6e2-11e8-9601-0e57c10fd788", APIVersion:"extensions/v1beta1", ResourceVersion:"37231033", FieldPath:""}): type: 'Normal' reason: 'CREATE' Ingress monitoring/kibana-kibana-chart
I0928 00:06:50.185665       7 nginx.go:276] Starting NGINX process
I0928 00:06:50.185752       7 leaderelection.go:185] attempting to acquire leader lease  ingress/ingress-controller-leader-nginx...
W0928 00:06:50.186121       7 controller.go:359] Service "ingress/ingress-nginx-ingress-default-backend" does not have any active Endpoint
W0928 00:06:50.186166       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:06:50.186217       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:06:50.186317       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:06:50.189852       7 status.go:197] new leader elected: ingress-nginx-ingress-controller-8446f46968-xstm5
I0928 00:06:50.493281       7 controller.go:185] Backend successfully reloaded.
W0928 00:06:50.493667       7 nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused
W0928 00:06:50.553720       7 nginx_status.go:207] unexpected error obtaining nginx status info: unexpected error scraping nginx status page: unexpected error scraping nginx : Get http://0.0.0.0:18080/nginx_status: dial tcp 0.0.0.0:18080: connect: connection refused
I0928 00:06:50.555546       7 backend_ssl.go:182] Updating local copy of SSL certificate "default/example-tls" with missing intermediate CA certs
I0928 00:06:50.665983       7 backend_ssl.go:182] Updating local copy of SSL certificate "monitoring/example-tls" with missing intermediate CA certs
W0928 00:06:53.519560       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:06:53.519613       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:06:53.519691       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:06:53.756551       7 controller.go:185] Backend successfully reloaded.
W0928 00:06:56.852924       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:06:56.853011       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
W0928 00:07:00.186235       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:07:00.186288       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:07:27.572591       7 leaderelection.go:194] successfully acquired lease ingress/ingress-controller-leader-nginx
I0928 00:07:27.572619       7 status.go:197] new leader elected: ingress-nginx-ingress-controller-8446f46968-ftlth
I0928 00:08:27.606789       7 status.go:362] updating Ingress monitoring/grafana status to [{ }]
I0928 00:08:27.607949       7 status.go:362] updating Ingress default/spinnaker-spinnaker-char-gate status to [{ }]
I0928 00:08:27.607955       7 status.go:362] updating Ingress monitoring/prometheus-server status to [{ }]
I0928 00:08:27.608196       7 status.go:362] updating Ingress monitoring/kibana-kibana-chart status to [{ }]
I0928 00:08:27.653281       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"grafana", UID:"41791d41-c026-11e8-89dc-0ef587d6ea0e", APIVersion:"extensions/v1beta1", ResourceVersion:"37231659", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress monitoring/grafana
I0928 00:08:27.654470       7 backend_ssl.go:60] Updating Secret "monitoring/example-tls" in the local store
W0928 00:08:27.654583       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:08:27.654616       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:08:27.654750       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:08:27.655008       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"spinnaker-spinnaker-char-gate", UID:"f1e7097b-7335-11e8-b0a6-0ef468f8fc50", APIVersion:"extensions/v1beta1", ResourceVersion:"37231660", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress default/spinnaker-spinnaker-char-gate
I0928 00:08:27.655523       7 backend_ssl.go:60] Updating Secret "default/example-tls" in the local store
I0928 00:08:27.655678       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"prometheus-server", UID:"66efb171-c023-11e8-89dc-0ef587d6ea0e", APIVersion:"extensions/v1beta1", ResourceVersion:"37231661", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress monitoring/prometheus-server
I0928 00:08:27.657689       7 event.go:221] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"monitoring", Name:"kibana-kibana-chart", UID:"0fc4aa3b-a6e2-11e8-9601-0e57c10fd788", APIVersion:"extensions/v1beta1", ResourceVersion:"37231662", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress monitoring/kibana-kibana-chart
I0928 00:08:27.897501       7 controller.go:185] Backend successfully reloaded.
W0928 00:08:30.988031       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:08:30.988065       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:08:30.988179       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:08:31.187849       7 controller.go:185] Backend successfully reloaded.
I0928 00:08:50.744412       7 backend_ssl.go:182] Updating local copy of SSL certificate "monitoring/example-tls" with missing intermediate CA certs
W0928 00:08:50.744591       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:08:50.744619       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:08:50.744721       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:08:50.902611       7 controller.go:185] Backend successfully reloaded.
I0928 00:08:50.982752       7 backend_ssl.go:182] Updating local copy of SSL certificate "default/example-tls" with missing intermediate CA certs
W0928 00:08:54.078042       7 controller.go:797] Service "monitoring/grafana" does not have any active Endpoint.
W0928 00:08:54.078078       7 controller.go:722] Error obtaining Endpoints for Service "default/spinnaker-spinnaker-char-gate": no object matching key "default/spinnaker-spinnaker-char-gate" in local store
I0928 00:08:54.078190       7 controller.go:169] Configuration changes detected, backend reload required.
I0928 00:08:54.298826       7 controller.go:185] Backend successfully reloaded.
flah00 picture flah00 on 28 Sep 2018

@flah00 from the output

curl: (60) SSL certificate problem: unable to get local issuer certificate

and 

I0928 00:08:50.982752       7 backend_ssl.go:182] Updating local copy of SSL certificate "default/example-tls" with missing intermediate CA certs

you need to update the secret (tls.cert)with the GoDaddy intermediate certificate bundle
Like cat <your cerr>.crt <go daddy intermediate>.crt >> bundle.crt using bundle.crt for the tls.crt

aledbf picture aledbf on 28 Sep 2018
👍2

@aledbf
That fixed it, thanks very much for you help!
Sorry, I assumed this was not a support issue. Are there docs on ca bundles, which I overlooked?

flah00 picture flah00 on 28 Sep 2018
👍5
Was this page helpful?
0 / 5 - 0 ratings

Related issues

bashofmann picture bashofmann  路  3Comments
sophaskins picture sophaskins  路  3Comments
smeruelo picture smeruelo  路  3Comments
cabrinoob picture cabrinoob  路  3Comments
whereisaaron picture whereisaaron  路  3Comments