Ingress-nginx: [nginx] Metrics of the defaultbackend

[amalucelli]

I'm working with nginx controller and using for the default-backend-service the gcr.io/google_containers/defaultbackend:1.4 image, and I verified that when I make a request to defaultbackend/metrics by my external load balancer or directly into the container, I get a several metrics that I don't think that should be visible to the internet.

» kubectl port-forward -n ingress-nginx nginx-default-backend-702016294-63mrp 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
Handling connection for 8080

» curl localhost:8080/metrics
# HELP default_http_backend_http_request_count_total Counter of HTTP requests made.
# TYPE default_http_backend_http_request_count_total counter
default_http_backend_http_request_count_total{proto="1.1"} 5
# HELP default_http_backend_http_request_duration_milliseconds Histogram of the time (in milliseconds) each request took.
# TYPE default_http_backend_http_request_duration_milliseconds histogram
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.001"} 0
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.003"} 0
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.005"} 3
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.01"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.025"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.05"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.1"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.25"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="0.5"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="1"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="2.5"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="5"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="10"} 5
default_http_backend_http_request_duration_milliseconds_bucket{proto="1.1",le="+Inf"} 5
default_http_backend_http_request_duration_milliseconds_sum{proto="1.1"} 0.024166
default_http_backend_http_request_duration_milliseconds_count{proto="1.1"} 5
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary

I found that this is hardcoded in the defaultbackend image, and can't be disabled by a configuration.

https://github.com/kubernetes/ingress-nginx/blob/adfbc8cc843c110825076e5377f4ea660341b455/images/404-server/server.go#L62

Is this the default and expected behavior?

It would be nice if in the defaultbackend image the /healthz and /metrics paths could be also be exposed in a different port, like the nginx-ingress-controller image.

[aledbf]

@amalucelli thank you for the report. We will add two flags to enable this features.

[lblackstone]

As a workaround for now, you can mask the /metrics endpoint with an Ingress rule so that metrics aren't exposed publicly.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: example
spec:
  rules:
  - http:
      paths:
      - path: /metrics
        backend:
          serviceName: default-http-backend
          servicePort: 80

[noqcks]

@aledbf what was your plan for supporting this? I have some free time.

I think hiding it behind port 10254 is a nice solution.

[aledbf]

I think hiding it behind port 10254 is a nice solution.

Exactly that :)
Please submit a PR if you have time to change this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[jonpulsifer]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@jonpulsifer: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[aledbf]

Closing. For those interested in such metrics, the PR https://github.com/kubernetes/ingress-nginx/pull/3125 contains the required changes. This just requires a custom image.

[m4xx101]

What could be the security impacte of this??