Kube-prometheus: Operators, Upgrade Paths, Backwards Compatibility and Documentation

[arianitu]

We are looking at prometheus-operator and kube-prometheus for our prometheus manifests instead of writing them ourselves, but we are a bit worried when we look at the documentation when there is no clear way to upgrade between versions.

For example, our cluster is on GKE 1.14.x since that is what is covered by the GKE SLA for a production environment. When I look at Prometheus-operator, it recommends GKE 1.18.x and upgrading before starting.

For us, that's a giant red flag for a project and we are wondering if the maintainers have clear upgrade paths with maintainability of older versions, especially stable versions of Kubernetes covered by SLA's.

Right now, I ran jb install github.com/coreos/kube-prometheus/jsonnet/[email protected] since that's what is compatibile with Kubernetes 1.14.x. I tried to find documentation on what to do when we go to release-0.4, or release-0.5 as Kubernetes stable channel is updated on GKE, but I found nothing in this repo.

So at this point, we are leaning more towards creating our own manifests instead of using the operator or this project at all since vanilla manifests have been always backwards compatible for us and especially predictable.

Can anyone shine some light on this since right now we're looking at operators and we see them as a disadvantage, not backwards compatible and unstable at the very least. Although, they do seem very convenient, since defining Prometheus and ServiceMonitor as manifests seem great, how do we know it's going to be supported 1 year from now? From the looks of it, if you don't upgrade, you're laying in the dust in 1 year with no clear migration path.

[brancz]

We attempt to support the latest 4 releases of Kubernetes. Keep in mind that fact that a particular vendor fails to support the latest versions that doesn’t mean another upstream project needs to follow that. That said we know of no compatibility issues other than upgrading Kubernetes with the same versions. In fact we ship all release versions as part of OpenShift releases so it’s definitely possible to upgrade. It also means we still accept and support older release branches as we have to as we happen to support OpenShift commercially, however much like Kubernetes 1.14 we are not adding new features to those branches.

Regarding “do it in plain manifests” if you study this repository you will find out that the overwhelming part of this repo is in fact plain manifests (granted generated, but there is nothing special about the manifests), only the prometheus operator itself is an operator and not “plain manifests”. I strongly encourage you not to go down the do it yourself road as you’re going to run into all the problems we’ve already solved over the years.

[arianitu]

@brancz thanks for following up.

I guess our main contention when we look at the repo is we don't see a clear upgrade path from 0.3 to 0.4 for example? Can you perhaps point us in the right direction? We want a clear upgrade path when upgrading our stable version to the next stable version.

Do we just run kubectl apply -f on all the new manifests after generating them for 0.4? I notice there is jb update but does that update to the latest? How do we do jb update 0.4? Will the operator, etc all work on the upgrade without doing anything else?

The other issue is what if we want to not include certain things like Grafana or Alertmanager since we want to have a setup similar to the Reddit infrastructure team. Just prometheus scraping data from node-exporter and that data is shipped to a central store using remote_write. (https://www.reddit.com/r/kubernetes/comments/ebxrkp/we_are_the_reddit_infrastructure_team_ama_about/fbb659m/)

What we want is:

1. Prometheus scraping data from node-exporters (using a small 2 hr retention period.) and shipping the data to a separate monitoring cluster that is hosting VictoriaMetrics on 1 node.
2. The monitoring cluster has Grafana and Alertmanager also living on the cluster.

Is there any way to modify the configuration files to not include alertmanager / grafana / etc. I think that's why we're seeing a custom manifest that is simpler:

1. DaemonSet for node-exporter
2. StatefulSet for prometheus with a configuration to export using remote_writer
3. The monitoring cluster would run VictoriaMetrics + Grafana + Alertmanager also all using vanilla manifests.

We feel like the above is going to be stable, scalable and easy to upgrade. When we look at kube-prometheus, we are getting a bit overwhelmed with the amount of things it uses by default.

We use to use Helm (ripped it out entirely.) for example and we really do not want to go back to any system that pulls us away from the core of Kubernetes which is yaml files fed to the API server. We find it extremely simple and easy to maintain. Right now I still know how to upgrade any component on our cluster, if we were still using Helm, I would have to go and see if Helm has any new versions, if the charts are still stable, etc. It's a nightmare.

But we're definitely open and that's why we're investigating kube-prometheus and prometheus-operator as a first case on the operator pattern since we do see value in using CRDs to simplify complex operations on harder to maintain systems, especially like Elasticsearch.

Sorry for the bit of the rant above, to summarize:

1. How do we generate manifests without including grafana, alertmanager, etc.
2. How do we upgrade from 0.3, to 0.4, etc. Is there guides, or none are needed since it's simple to just re-apply manifests?
3. Why the operator pattern? Is there anything out there that supplies vanilla manifests for prometheus?

Cheers.

[arianitu]

Hmm, when we try to customize this project, our build script hangs infinitely:

+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests prometheus.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'

Edit:

I have no idea what's wrong with my local jsonnet (installed from homebrew) and/or gojsontoyaml, but they hang infinitely.

I can build the project using the docker container tho:

docker run --rm -v $(pwd):$(pwd) --workdir $(pwd) quay.io/coreos/jsonnet-ci ./build.sh prometheus.jsonnet

No errors reported anywhere.

[arianitu]

I've removed AlertManager and Grafana from our jsonnet, that seemed to remove the generated manifests. Is this the recommended way of not including parts of prometheus?

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  // Uncomment the following imports to enable its patches
  // (import 'kube-prometheus/kube-prometheus-anti-affinity.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-managed-cluster.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-node-ports.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-thanos-sidecar.libsonnet') +
  // (import 'kube-prometheus/kube-prometheus-custom-metrics.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }

[brancz]

Yeah you can definitely do that! Kube-prometheus is first and foremost a library so you can build the monitoring experience you want. Since there are a number of default configurations to make the out of the box experience nicely integrated you might find yourself having to tweak one thing or another though.

FWIW I can sympathize with your feelings towards helm, it’s one of the reasons we choose not to maintain a chart.

[arianitu]

@brancz that's great, we've removed prometheus-adapter as well since we're on GKE and GKE includes metrics-server.

The only thing we're held up on is upgrade paths. We would love to start with Kubernetes 1.18.x, but our clients expect stability and the only way we're covered by SLA on GKE is by being on the stable branch 1.14.x.

Can you help me understand how we would upgrade everything when upgrading Kubernetes when using kube-prometheus? We have a staging cluster that we can test things on, but would be useful to know how to upgrade. I think as you upgrade to 1.16.x the CRDs change, but we're unsure if we need to delete the CRDs or apply them again.

Take a look at https://cert-manager.io/docs/installation/upgrading/. It would be amazing if kube-prometheus had this or prometheus-operator.

[brancz]

Prometheus Operator already performs upgrade tests so we always know that we can upgrade from one prometheus operator version to another and we also test a compatibility matrix of Prometheus versions. Only Prometheus and Alertmanager are stateful, so they are the only ones that really need care in regards to upgrades. From there on the kube-prometheus stack is entirely stateless and it's just a matter of running kubectl apply on the manifests.

That said kube-prometheus is first and foremost a library of jsonnet code that is intended to be building blocks to build your very own monitoring experience with the best default out of the box configuration that tries to cater to as many people as possible while making best use of the components that ship with it. Since you are intended to use jsonnet to customize the stack to your desire there is only so much testing we can do and mileage will vary greatly.

Can you help me understand how we would upgrade everything when upgrading Kubernetes when using kube-prometheus? We have a staging cluster that we can test things on, but would be useful to know how to upgrade. I think as you upgrade to 1.16.x the CRDs change, but we're unsure if we need to delete the CRDs or apply them again.

We've upgraded prometheus-operator on clusters that have existed since Kubernetes 1.6, ever since the APIs have been v1, we have not required users to ever recreate CRDs. The only thing that goes in a direction like this is that we have added stronger validations at creation time, which prevents bad CRs from being created in the first place instead of failing to reconcile, which overall results in the same thing, just earlier and more obvious failure.

Especially for GKE and other providers that we don't use (we integrate this stack as part of OpenShift) we rely on the community to maintain those integrations and let us know and document if things need special care. That generally doesn't happen too often, but if you would like to take ownership of the GKE integration and maybe add additional testing for this, we'd be more than happy to work with you! :)

[tonchev-ivan]

Hmm, when we try to customize this project, our build script hangs infinitely:

+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests prometheus.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml' -- '{}'

Edit:

I have no idea what's wrong with my local jsonnet (installed from homebrew) and/or gojsontoyaml, but they hang infinitely.

I can build the project using the docker container tho:

docker run --rm -v $(pwd):$(pwd) --workdir $(pwd) quay.io/coreos/jsonnet-ci ./build.sh prometheus.jsonnet

No errors reported anywhere.

I had the same issue with the script, the problem is in jsonnet from brew. This is how I resolved it:

go get github.com/google/go-jsonnet/cmd/jsonnet
cp $(go env GOPATH)/bin/jsonnet /usr/local/bin/