Kube-prometheus: Switch to jsonnet-libs/k8s-alpha instead of ksonnet

[ajp-lqx]

What is missing?

Ksonnet was deprecated over a year ago. It seems one of kube-prometheus' maintainers kept a fork of it updated for newer k8s api versions here: https://github.com/kube-jsonnet/k

However this has been replaced with (hopefully) a final resting place, here: https://github.com/jsonnet-libs/k8s-alpha

Why do we need it?

The old ksonnet (which is still used in docs) does not use modern API versions, and this is really starting to show on k8s cluster's v1.16+ where, for example, extensions/v1beta1 is no longer recognized (I ran into this when trying to add ingress support).

Environment

• kube-prometheus version: 2c1fc1cc11547ca06a143fce6e430e4f7e0be294

Anything else we need to know?:
I've migrated the example jsonnet file to the latest version of k8s-alpha jsonnet lib. I've copied this below. 90% of it is identical, main changes are in the imports, and the location of the various mixins has changed.

local k = import "github.com/jsonnet-libs/k8s-alpha/1.18/main.libsonnet";
local pvc = k.core.v1.persistentVolumeClaim;
local networking = k.networking.v1beta1;
local ingress = networking.ingress;
local ingressTLS = networking.ingressTLS;
local ingressRule = networking.ingressRule;
local httpIngressPath = networking.httpIngressPath;

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +
           (import 'kube-prometheus/kube-prometheus-eks.libsonnet') + {
  _config+:: {
    namespace: 'monitoring',
  },
  prometheusRules+:: {
    groups+: [
      {
        name: 'example-group',
        rules: [
          {
            record: 'aws_eks_available_ip',
            expr: 'sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10',
          },
        ],
      },
    ],
  },
  prometheus+:: {
    prometheus+: {
      spec+: {
        retention: '30d',
        storage: {
          volumeClaimTemplate:
            pvc.new('') +
            pvc.mixin.spec.withAccessModes('ReadWriteOnce') +
            pvc.mixin.spec.resources.withRequests({ storage: '100Gi' }) +
            pvc.mixin.spec.withStorageClassName('gp2'),
        },
      },
    },
  },
  ingress+:: {
    'prometheus':
      ingress.new('prometheus') +
      ingress.mixin.metadata.withNamespace($.prometheus.prometheus.metadata.namespace) +
      ingress.mixin.metadata.withAnnotations({
        'nginx.ingress.kubernetes.io/force-ssl-redirect': 'true',
        'kubernetes.io/ingress.class': 'nginx-internal',
      }) +
      ingress.mixin.spec.withRules(
        ingressRule.withHost('prometheus.aws-uswest2-hldltdev.inblock.com') +
        ingressRule.mixin.http.withPaths(
          httpIngressPath.mixin.backend.withServiceName($.prometheus.service.metadata.name) +
          httpIngressPath.mixin.backend.withServicePort('web')
        ),
      ) +
      ingress.mixin.spec.withTls(
        ingressTLS.withHosts('prometheus.aws-uswest2-hldltdev.inblock.com') +
        ingressTLS.withSecretName('tls-prometheus')
      ),
    'alertmanager':
      ingress.new('alertmanager') +
      ingress.mixin.metadata.withNamespace($.prometheus.prometheus.metadata.namespace) +
      ingress.mixin.metadata.withAnnotations({
        'nginx.ingress.kubernetes.io/force-ssl-redirect': 'true',
        'kubernetes.io/ingress.class': 'nginx-internal',
      }) +
      ingress.mixin.spec.withRules(
        ingressRule.withHost('alertmanager.aws-uswest2-hldltdev.inblock.com') +
        ingressRule.mixin.http.withPaths(
          httpIngressPath.mixin.backend.withServiceName($.alertmanager.service.metadata.name) +
          httpIngressPath.mixin.backend.withServicePort('web')
        ),
      ) +
      ingress.mixin.spec.withTls(
        ingressTLS.withHosts('alertmanager.aws-uswest2-hldltdev.inblock.com') +
        ingressTLS.withSecretName('tls-alertmanager')
      ),
    'grafana':
      ingress.new('grafana') +
      ingress.mixin.metadata.withNamespace($.prometheus.prometheus.metadata.namespace) +
      ingress.mixin.metadata.withAnnotations({
        'nginx.ingress.kubernetes.io/force-ssl-redirect': 'true',
        'kubernetes.io/ingress.class': 'nginx-internal',
      }) +
      ingress.mixin.spec.withRules(
        ingressRule.withHost('grafana.aws-uswest2-hldltdev.inblock.com') +
        ingressRule.mixin.http.withPaths(
          httpIngressPath.mixin.backend.withServiceName($.grafana.service.metadata.name) +
          httpIngressPath.mixin.backend.withServicePort('web')
        ),
      ) +
      ingress.mixin.spec.withTls(
        ingressTLS.withHosts('grafana.aws-uswest2-hldltdev.inblock.com') +
        ingressTLS.withSecretName('tls-grafana')
      ),
  },
};

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ [name + '-ingress']: kp.ingress[name] for name in std.objectFields(kp.ingress) }

[brancz]

I'm happy with moving if it's becoming a problem for people. Would you like to contribute this change?

[metalmatze]

I'm happy to move things gradually. :blush: We should also make sure to only start using absolute imports going forward.

FWIW I started trying this in some other project and encountered some blockers I want to let @sh0rez know about.

[sh0rez]

Hey all, happy to part of the migration.

@metalmatze I'd be surprised if there were not blockers, given that this code is very young. Feel free to let me know here / on jsonnet-libs/k8s what's missing so we can work on that :)

[ajp-lqx]

@brancz yes planning on it, I've been looking at the kube-prometheus jsonnet files and familiarizing with the codebase.

@metalmatze @sh0rez any recommendations on best way to do this piece-meal, or perhaps best starting point? Totally on-board with avoiding some massive whole-repo PR which would be impossible to review.

[metalmatze]

I think https://github.com/brancz/kubernetes-grafana/blob/master/grafana/grafana.libsonnet would actually be a really good candidate. The scope is super clear and it's a direct dependency of kube-prometheus, but not yet part of the big project. :)

[brancz]

Happy to cut a release of kubernetes-grafana to allow it to be pinned to until we figure out the rest of the pieces.

[ajp-lqx]

Sorry for the late replies, been a busy couple of days, but I've gotten through the initial steps of getting my env setup and a consistent build process, and have started working on the grafana.libsonnet file.

I'm brand-new to jsonnet so might take a little bit to get up to speed :) Is there an existent unit-testing infra or none for now?

[brancz]

Jsonnet has the assert function built in, which we haven't made use of (yet!), if you want to start, feel free to! The grafana package is probably a great one to experiment with that.

[olegmayko]

You may try this one https://github.com/yugui/jsonnetunit.

[brancz]

Awesome! I had not known about that one before! Thanks for sharing!

[kakkoyun]

How about removing ksonnet completely? If you are ok with this approach, I have some spare time I can work on it today.

cc @brancz @metalmatze @lilic

[lilic]

@kakkoyun sounds great to me! Feel free to do it in stages if that's easier, as we have a lot of places that need changing, maybe per libsonnet? But if you manage to get to all of it's great as well of course!

[kakkoyun]

@lilic Per libsonnet makes sense 👍 It'd be easier on the eyes for the reviewers.

[metalmatze]

Closing this then.

[lilic]

We have not migrated all the ksonnet yet, so might make sense to have this open until we do. :)