Containers-roadmap: [EKS] [request]: allow to configure metricsBindAddress for kube-proxy

[chancez]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I want to be able to monitor kube-proxy with Prometheus, but cannot because, by default the metricsBindAddress is set to 127.0.0.1:10249 meaning it isn't accessible outside of the pod.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
I want to monitor kube-proxy. I cannot do this unless I can reconfigure kube-proxy in some way.

Are you currently working around this issue?
I am not.

Additional context
You can see the kube-proxy config created by EKS in the kube-system namespace via k get cm -n kube-system kube-proxy-config -o yaml.

[TBeijen]

Additional context illustrating the problem: https://github.com/helm/charts/tree/master/stable/prometheus-operator#kubeproxy

[dawidmalina]

Quick hotfix:

kubectl --kubeconfig kubeconfig* -n kube-system get cm kube-proxy-config -o yaml |sed 's/metricsBindAddress: 127.0.0.1:10249/metricsBindAddress: 0.0.0.0:10249/' | kubectl --kubeconfig kubeconfig* apply -f -
kubectl --kubeconfig kubeconfig* -n kube-system patch ds kube-proxy -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"updateTime\":\"`date +'%s'`\"}}}}}"

[devkid]

I cannot do this unless I can reconfigure kube-proxy in some way.

You can see the kube-proxy config created by EKS in the kube-system namespace via k get cm -n kube-system kube-proxy-config -o yaml.

So what exactly prevents you from changing this configmap?

[chancez]

Nothing, except that it's entirely possible that'll get overwritten at some point in the future. EKS has no documentation on what's supported and unsupported when it comes to this kind of modification, so I'd like to know what kind of guarantees there would be that it won't be stomped on in the future.

[devkid]

The EKS cluster upgrade documentation explicitly instructs to manually update kube-proxy, coredns and aws-node after a control plane upgrade. I highly doubt that AWS will touch these components in an automated fashion. We have been using custom configuration for all three (kube-proxy, coredns and aws-node) since we started using EKS and never had a problem with this.

[chancez]

That's the daemonset, not the configuration. While I agree it's unlikely to be an issue, having previously worked on commercial Kubernetes distributions, making assumptions on what's going to be safe to modify isn't always the best choice. I suppose I can always open a ticket.

[kogeler]

I really need this feature!

[Cre-eD]

+

[adamrbennett]

Manually editing configurations is a work around, not a solution. All configs should be coded and recorded in source control. Unless I'm missing something, it appears EKS users have no control over kube-proxy configuration, except for manual edits.

[damscott]

I think @devkid is correct, but we could use confirmation from the EKS team that automated changes will not be made to the kube-system namespace in a running cluster.

Here is the problem I encountered when setting metricsBindAddress:

When I tried to edit metricsBindAddress I found that I did not have a kube-proxy-config configmap. I have two EKS clusters: one is 565 day old, the other is 325 days old.
The newer cluster has the kube-proxy-config configmap, the older cluster does not. I have never touched the configuration of those resources beyond patching the DaemonSet image tag as documented in Updating an Amazon EKS cluster Kubernetes version.

Here is my proposed solution:

Have a git repo with all of the baseline EKS manifests (including kube-proxy) and document which EKS Platform Version corresponds with each revision

• This already exists to some degree for the aws-node DaemonSet as part of the CNI patching process
• It would provide guidance during upgrades while leaving full control in the hands of the user
• It would help recover from accidental misconfigurations made to core EKS components
• It would have helped me dealing with the undocumented drift in the kube-proxy manifests

Update:

Copying the DaemonSet and ConfigMaps from the newer cluster to the old one works. It was necessary to change one line in the kube-proxy configmap:

        server: https://<foo>.<bar>.<region>.eks.amazonaws.com

[abangser]

Just to add, I have a cluster that was last upgraded in late Jan to version 1.14 but initially created in October at 1.10. It does not seem to have the kube-proxy-config configmap. We are also running prometheus-operator and unable to use the kubeProxy exporter without an edit to the metricsBindAddress referenced in the docs here: https://github.com/aws/containers-roadmap/issues/657#issuecomment-567804666.

[kskewes]

100% agree with @damscott. Good write up.
kube-proxy-config ConfigMap and kube-proxy DaemonSet should be in a versioned manifest like other components (cni, coredns, calico).

There's a second ConfigMap called kube-proxy that is a cluster specific k8s kubeconfig from the looks of it that I guess we wouldn't touch.

[mbarrien]

Note that in my investigation of the clusters in my org (originally made in a wide range of EKS versions from 1.11 to 1.15), any EKS clusters created from 1.12 onwards use the kube-proxy-config configmap. There is still the issue of migrating any EKS clusters created before 1.12 to using a configmap; this might be possible to do manually.

Any EKS clusters created up to 1.12 also have a slightly different kube-proxy configmap; the configmap 1.12+ contains the cluster URL, while 1.11 and below have the cluster URL (along with oom-scor-0adj and proxy-mode) passed as a command line argument to kubeproxy (within the podspec inside the DaemonSet spec).

If you created your cluster pre-EKS 1.12, to be able pass metricsBindAddress you either need to "upgrade" your daemonset to take in the kube-proxy-config configmap (which you'll need top create manually), or pass it in as a command line argument. Unfortunately this "upgrade" of kube-proxy to the new way is not documented.

[mbarrien]

@damscott https://github.com/aws/eks-charts/tree/master/stable/aws-vpc-cni has some helm charts for keeping aws-vpc-cni up to date. Would be great if they added similar charts of kube-proxy and coredns.

[TBBle]

I haven't verified this, but CoreDNS already has a Helm chart, so it should just be a matter of getting the right parameters for that to match the default-installed version, then any changes you need.

[OffCode]

I also have the same issue with most recent EKS version (Kubernetes 1.16) and prometheus-operator stable helm chart. The root cause of the problem is that EKS likely uses an out-dated config value metricsBindAddress.
The Kubernetes documentation describes:

--metrics-bind-address ipport     Default: 127.0.0.1:10249

  | The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable.

See also
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/

[boatmisser]

This is still an issue with EKS 1.17.

The kube-proxy-config has 0.0.0.0 for bindAddress and healthzBindAddress.
However it still uses 127.0.0.1 for metricsBindAddress!!!

Would it be much of an issue to move metricsBindAddress to 0.0.0.0 to bring it in line with the other addresses, and allow us to monitor kube-proxy???

[chancez]

Original creator of the issue here. I thought I'd chime in on how I handle this these days.

At this point I just maintain a kustomize base for kube-proxy that I apply to all clusters immediately after deploying and it's managed via ArgoCD. You could do the same with a helm chart or similar.

At this point I think AWS has made it quite clear, it's unlikely they'll manage components inside the cluster, at least in the short to medium term. If anything, I'd bet that if they were to begin managing "core" components, like kube-proxy, aws-node/cni, etc, that you would opt-in to EKS managing these components for you, and even then, it's likely to only be in new EKS clusters. This is just a guess.

Their documentation provides instructions on managing the versions of various components in the cluster, so it makes me think that it's unlikely there will be any automated management of this stuff for quite some time, so there's little risk to managing the versions, and configuration of these resources yourself.

As others suggested, I think a more likely avenue for supporting this feature is https://github.com/aws/eks-charts/issues/156, https://github.com/aws/containers-roadmap/issues/923, and the other issues where you would fully manage kube-proxy yourself.