Kube-prometheus: use `awscni_ip_max` to alert on available IPs in EKS node

[alam0rt]

Hey - quick one hopefully...

tl;dr - awscni_ip_max is how many IPs a node can support based on instance type whereas awscni_total_ip_addresses is how many IPs are available based on currently attached ENIs (which can change at any time). The latter is probably not what we want to base this alert off.

As we can see below we are alerting on the fact that when a node has less than the desired amount of IPs to dish out there is a problem:
https://github.com/prometheus-operator/kube-prometheus/blob/52827617108b222c3fed65192121b73dab0ffcaf/jsonnet/kube-prometheus/kube-prometheus-eks.libsonnet#L75

I think that instead of awscni_total_ip_addresses we should use awscni_max_ip. Take this scenario for example

node-a is an r5.large with a maximum of 3 ENIs to attach. The ENIs can support 9 pods each (1 is used for the host). In our scenario we have 9 pods allocated to this node.

awscni_assigned_ip_addresses: 9

The CNI aws-node keeps a single ENI worth of IPs available on standby, so..

awscni_total_ip_addresses: 18

And finally - let's say we want to alert on when we have less than 10 IPs available. So the alert below would fire advising that we have less than 10 IPs available to allocated when in reality we have 18

sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10

The above isn't really correct - we have 18 IPs available, not 9.

[brancz]

A fix would be very welcome if you want to contribute it! (I and I believe all other kube-prometheus maintainers don't use kube-prometheus on EKS so it's difficult to verify for us, but we'd be more than happy to accept a contribution!)

[alam0rt]

Alright, I'll whip something up soon.

On Mon, 30 Nov 2020, 11:03 pm Frederic Branczyk, [e-mail]
wrote:

A fix would be very welcome if you want to contribute it! (I and I believe
all other kube-prometheus maintainers don't use kube-prometheus on EKS so
it's difficult to verify for us, but we'd be more than happy to accept a
contribution!)

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/prometheus-operator/kube-prometheus/issues/817#issuecomment-735744602,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AILMHNRX657ZCIYMYFIDX5TSSOCX7ANCNFSM4UHDWL4Q
.

[alam0rt]

https://github.com/prometheus-operator/kube-prometheus/pull/819

[alam0rt]

I am happy to think of extra alerts and contribute/verify them also.

[paulfantom]

I am closing this as the main issue was fixed, but @alam0rt don't hesitate to add new alerts :)