Windows_exporter: Best Practices to deploy windows_exporter for Kubernetes

[Gerthum]

Hi Guys

Since Prometheus is recommending windows_exporter as an alternative to node_exporter when running Windows Node Pools in Kubernetes. Does anyone have any best practices for running it in this environment.

Things I can think of:

1. Do you need to run it on the Host or can you run it as a Container?
2. Example Dockerfile/pre-built container?
3. Example configuration file?
4. Example Dashboard?

[AMoghrabi]

Any information on this is appreciated. I also have the same question. Thanks!

[ksmuthuus]

I hope https://github.com/prometheus-community/windows_exporter/pull/581 might help to some extent meet your requirement.

[carlpett]

Just to keep the things in as few places as possible, here's a summary of information gathered in other issues and PRs:

Currently it is not possible to run the windows_exporter inside a container and get meaningful results (you can start it, and you'll get some metrics, but they do not reflect the host machine). Windows does not support privileged containers yet, which is a requirement for this to work. So, at this time, you'll need to install the exporter on the host itself.

[Gerthum]

Hi Guys - We've had some success with the following exporter:
https://github.com/cloudworkz/kube-eagle
It only runs on the Linux nodes but uses kube state metrics to pull in the windows nodes and pods metrics. No need to run it on the windows nodes.

[keikhara]

@Gerthum , when you pull the metrics using kube state metrics, it can show the node and pod metrics but it chokes after certain amount of pod. I would not recommend it. I think it was 30pods or so.

[keikhara]

@carlpett Windows provides named pipe so you can pull information about the pods. https://dille.name/blog/2017/11/29/using-the-docker-named-pipe-as-a-non-admin-for-windowscontainers/
Can you elaborate why you need the privilege container on Windows and what you are trying to do?

[carlpett]

@keikhara The windows_exporter has a much larger scope than just container information. And the rest of the exporter primarily uses the WMI and Performance Counter subsystems in Windows, which are not available from unprivileged containers.

edit: As I see that you work within Microsoft, if you have any nice ideas for how this could be accomplished, I'd be super happy about any tips!

[DerrickMartinez]

For now, you need to run it through wins. This is config for the old version but it still works. Proxy is a simple proxy I wrote in go. Prometheus scrapes the proxy, proxy hits the host's IP. Works very well

total 67680
drwxr-xr-x   8 dmartinez  staff       256 Nov  4 16:44 .
drwxr-xr-x@ 30 dmartinez  staff       960 Nov  5 10:56 ..
-rw-r--r--@  1 dmartinez  staff       426 May 14 14:54 Dockerfile
-rw-r--r--   1 dmartinez  staff       192 Apr 27  2020 entry.ps1
-rwxr-xr-x   1 dmartinez  staff   6361088 Apr 28  2020 proxy.exe
-rw-r--r--   1 dmartinez  staff       131 Apr 27  2020 proxy_entry.ps1
-rw-r--r--@  1 dmartinez  staff  12459520 Apr 24  2020 wins.exe
-rw-r--r--@  1 dmartinez  staff  15818240 Apr 24  2020 wmi_exporter-0.11.1-amd64.exe

proxy_entry.ps1

$j = C:\wins\wins.exe cli net get | ConvertFrom-Json
C:\wins\proxy.exe --remote-port 9182 --remote-ip $j.AddressCIDR.split("/")[0]

Dockerfile

# escape=`
FROM mcr.microsoft.com/powershell:nanoserver-1909
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

COPY wins.exe C:\\wins\\
COPY proxy.exe C:\\wins\\
COPY wmi_exporter-0.11.1-amd64.exe C:\\wins\\wmi-exporter.exe
COPY entry.ps1 entry.ps1
COPY proxy_entry.ps1 proxy_entry.ps1

EXPOSE 9182 8080
USER ContainerAdministrator
CMD ["pwsh", "-f", "entry.ps1"]

entry.ps1

copy c:\wins\wmi-exporter.exe c:\host
c:\wins\wins.exe cli prc run --path c:\host\wmi-exporter.exe --args "--collectors.enabled container,cpu,cs,logical_disk,net,os,system" --exposes TCP:9182

[casey-robertson]

Also for those of us running the prometheus operator, it would be nice to have this integrated.