Kube-prometheus: Question: kube-prometheus "as a library"

[bfleming-ciena]

Hi. I think this is great! But in the docs it says "Although this project is intended to be used as a library". What's wrong with just deploying kubectl create -f manifests/? Is there something wrong with that deployment? It all seemed to work pretty well, and was usable.

Thanks

[Vince-Cercury]

yeah I'm a little bit confused now as how to use the latest release 1.9.

I figured the Readme version I should use is this one https://github.com/coreos/prometheus-operator/tree/release-0.19/contrib/kube-prometheus

Since master is not released yet.

But if/how to use it as a library. I used to alter the yaml files before running the hack/cluster-monitoring/deploy command.

How do I:

• tell Operator where to find the etcd secrets to monitor Etcd
• Add my custom Grafana dashboards to the mix ( I can't find the folder for raw Grafana dashboards)
• Best way to add an ingress to Grafana and Prometheus? (I used to just create them separately but not sure if there is a better way now)
• Get Prometheus to store data in a Persistent Volume ( I used to edit prometheus-k8s.yaml and add the storage section in the spec)

In the Readme it says a HELM chart will be made available. How does that compare to Operator. I'm used to HELM chart so I wouldn't mind one

[brancz]

The rendered manifests in contrib/kube-prometheus/ are only there because it's an easy way to get started. The new readme in contrib/kube-prometheus/ describes in detail, how these manifests can be generated with any customization you may want.

My personal opinion is that helm is fundamentally the wrong solution, because every possible customization that could be possible needs to be in the "upstream" helm chart. That said, the community seems to like them, which is why we have some in this repository, however, the official endorsed solution is use the methods outlined in contrib/kube-prometheus/README.md to generate the manifests with exactly the customizations you want.

Regarding all the questions about customizations, I think they are great examples of documentation that we should write. I'll make sure we add more documentation next week!

[Vince-Cercury]

@brancz I'm with you. If HELM is not the preference of the main contributor, I'll stick to the normal setup procedure.

I'm going to revert to 0.17 or 0.18 until docs are written.

I found it hard to understand the new readme in contrib/kube-prometheus/ but with your new docs next week, I have hope :)
I think we need some sort of design/concept document to understand how it's intended to be used.

Happy to help with docs and testing the docs. I have dedicated time available next week to make it easier to setup Prometheus at my company. So want to do it for the community and kube-aws

[brancz]

Awesome! I'm starting to write some more documentation, I'll let you know when I have something to go through! :slightly_smiling_face: Thanks a lot for offering your help, I highly highly appreciate it!

[brancz]

I feel my first comment wasn't clear enough on the initial question. The way it is intended to be consumed is that you use the jsonnet files as a library in your internal infrastructure repository. Documentation on how we intend you to do this can be found here: https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus#usage

The result of that is that your very own manifests are rendered into your repository/directory, instead of having to modify manifests in place. But the best part is that all you need to update the entire stack is run jb install in your repository and re-render the manifests.

More concretely about the docs you asked for @VinceMD, we have just merged new documentation for Prometheus rules and Grafana dashboards: https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/developing-prometheus-rules-and-grafana-dashboards.md

I'll be working on adding further ones in the upcoming days.

[Vince-Cercury]

@brancz thanks

Some feedback:

• "provided example rule." -> link is broken in developing-prometheus-rules-and-grafana-dashboards.md
• " using the grafonnet library for jsonnet" I'm not sure this is the right link neither. Goes to a directory
• " a provided example dashboard." broken link

[Vince-Cercury]

I think the Usage section of https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus can be improved further:

• You may wish to not use ksonnet and simply render the generated manifests to files on disk -> as opposed to what? If we don't do that, then what is the set of commands to install prometheus operator in a Kubernetes cluster
• this can be done with: -> for the snippet that follows, we could give more specific instructions. Example
  a) create a file name X.jsonnet and place it in your Git repository or wherever you manage and version configuration files
  b) create a file build.sh with this content
  c) give permission to this file to be executed chmod +x build.sh
  d) execute the build.sh ./build.sh
  e) the manifest files were generated in manifest/. Now what is the next step? Does one simply run kubectl apply -f manifests/ . Would that work?

And then we can add the customisation section with my example questions above. And how it's done with jsonnet step by step or example folder. And maybe a kube-aws example too :)

[Vince-Cercury]

Another issue after running build.sh and apply -f manifests/

The ConfigMap "grafana-dashboard-definitions" is invalid: metadata.annotations: Too long: must have at most 262144 characters

What is the tool we have to use to shared the dashboards in multiple configmaps? or is it down automatically with jsonnet
Can this be added to the doc https://github.com/coreos/prometheus-operator/blob/master/contrib/kube-prometheus/docs/developing-prometheus-rules-and-grafana-dashboards.md?

In my case I was able to create the map with
kubectl create -f manifests/grafana-dashboardDefinitions.yaml
instead of apply and this issue https://github.com/coreos/prometheus-operator/issues/535 seems to indicate apply has a limitation of 256 KB when using apply.

So what is in your opinion the preferred way of creating the resources from generated manifest?
is create ok? kubectl create -f manifests like that?

[Vince-Cercury]

About the other questions (storage, etcd secret, ingress)

EDIT: updated the examples after I managed to get some of it working

etcd secret

I created a jsonnet like this:
```local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
{
prometheus+: {
prometheus+:
{
spec+: {
secrets: ["etcd-certs"]
},
},
},
}

It did add the secret to the spec of the Prometheus in the manifest file.
I'm not sure if this is the correct way of doing it. Can someone confirm so we can add to the docs?

# Ingress
I looked at kube-prometheus-node-ports.libsonnet and tried to adapt it.
The first step is to expose a couple more ports because my ingress works only for port 80 so far:

local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local service = k.core.v1.service;
local servicePort = k.core.v1.service.mixin.spec.portsType;

{
prometheus+: {
service+: {
spec+: {
ports: [
{
name: 'web',
port: 9090,
# You don't need to specify the targetPort if it is the same as the port.
# Using the name of the port, meaning it uses the same inboubnd port as target
targetPort: 'web',
},
{
name: 'ingress',
port: 80,
# the target is the same port as the web port
targetPort:'web',
},
],
type: 'NodePort',
},
},
},
alertmanager+: {
service+:
service.mixin.spec.withPorts(servicePort.newNamed('web', 9093, 'web') + servicePort.withNodePort(30903)) +
service.mixin.spec.withType('NodePort'),
},
grafana+: {
service+: {
#service.mixin.spec.withPorts(servicePort.newNamed('http', 3000, 'http') + servicePort.withNodePort(30902)) +
#service.mixin.spec.withType('NodePort'),
spec+: {
ports: [
{
name: 'http',
port: 9090,
# You don't need to specify the targetPort if it is the same as the port.
# Using the name of the port, meaning it uses the same inboubnd port as target
targetPort: 'http',
},
{
name: 'ingress',
port: 80,
# the target is the same port as the web port
targetPort:'http',
},
],
type: 'NodePort',
},
},
},
}

The next step is to create the actual ingress which will map to the two services (Grafana and Prometheus) on port 80:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: prometheus
namespace: monitoring
spec:
rules:

• host: prometheus.monitoring.MYDOMAIN
  http:
  paths:
  
  
  
  • backend:
    
    serviceName: prometheus-k8s
  
  

servicePort: 80

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: grafana
namespace: monitoring
spec:
rules:

• host: grafana.monitoring.MYDOMAIN
  http:
  paths:
  
  
  
  • backend:
    
    serviceName: grafana
    
    servicePort: 80
  
  

That I could not figure how to do it. I would also like `MYDOMAIN` to be some form of parameter so I can use it for multiple clusters that have different domains for Ingress.

# Storage

I've created the below jsonnet, which seems to work. Is it the right way of doing this?

local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';

{
prometheus+: {
prometheus+: {
spec+: {
storage: {
volumeClaimTemplate: {
metadata: {
name: 'prometheus-storage',
namespace: 'monitoring',
},
spec: {
accessModes: ['ReadWriteOnce'],
resources: {
requests: {
storage: '20Gi',
},
},
},
},
},
},
},
},
}
```

It's a lot to learn: jsonnet, ksonnet. I like the idea of the library very much but Prometheus Operator had lowered the barrier to entry before making it hard again :(

I think I will for now generate the manifests with build.sh and then manually edit/fix. Which prevents easy upgrades, and so forth.

Happy to keep providing feedback and testing. I think we need a few more examples.

Shouldn't persistence volume be default by the way?
Does anyone know what other customisations we need for kube-aws?

[brancz]

Sorry for not answering, I was on vacation for the past two days (and Monday was a public holiday in Germany).

Thanks a lot for all your feedback, it's extremely valuable. I have already started writing docs for all those topics. I will get back to this within the next couple of days, hopefully with more docs to test! :slightly_smiling_face:

I think I will for now generate the manifests with build.sh and then manually edit/fix. Which prevents easy upgrades, and so forth.

This is generally fine, it just means you will ultimately have more work. It's not any worse than before, we're just allowing to make it more easy for people who want to put in the effort, but you are completely right that the docs are not sufficient at all today.

Shouldn't persistence volume be default by the way?

The problem is that when we do this Prometheus Pods may not end up being scheduled on peoples local clusters, and especially the rendered manifests are intended to be tried out super quickly in order for people to see why this is worth it.

Does anyone know what other customisations we need for kube-aws?

I'm writing/converting instructions for kops right now, would love to add kube-aws to the list eventually! :slightly_smiling_face:

[Vince-Cercury]

cool, looking forward to the doc updates.

Here is another thing I did: allow Prometheus to monitor any namespace, which could be added to examples of customisation. This is to go around the issue:

Failed to list *v1.Service: services is forbidden: User \"system:serviceaccount:monitoring:prometheus-k8s\" cannot list services at the cluster scope"

```local k = import 'ksonnet/ksonnet.beta.3/k.libsonnet';
local clusterRole = k.rbac.v1.clusterRole;
local policyRule = clusterRole.rulesType;

{
prometheus+: {
clusterRole+: {
rules:
[
policyRule.new() + policyRule.withApiGroups(['']) +
policyRule.withResources(['nodes/metrics', 'services', 'endpoints', 'pods']) +
policyRule.withVerbs(['get', 'list', 'watch'])
,policyRule.new() +
policyRule.withNonResourceUrls('/metrics') +
policyRule.withVerbs(['get']),
]
},
},
}
```

As usual, if the syntax is not right or there is a better way to do it, let us know

[ArvinDevel]

I'm confused with the customization way of kube-prometheus using jsonnet. For example, When I want to change alertmanager config, Where to put those lines below, how to execute it?

((import 'kube-prometheus/kube-prometheus.libsonnet') + {
   _config+:: {
     alertmanager+: {
       config: |||
         global:
           resolve_timeout: 10m
         route:
           group_by: ['job']
           group_wait: 30s
           group_interval: 5m
           repeat_interval: 12h
           receiver: 'null'
           routes:
           - match:
               alertname: DeadMansSwitch
             receiver: 'null'
         receivers:
         - name: 'null'
       |||,
     },
   },
 }).alertmanager.secret

I put it into one single jsonnet file alertmanager.jsonnet, and create dir alertmanger, then execute using

jsonnet -J vendor -m alertmanager ${1-alertmanager.jsonnet} | xargs -I{} sh -c 'cat $1 | gojsontoyaml > $1.yaml; rm -f $1' -- {}

Then I apply them using

kubectl apply -f alertmanager/

The output is

error validating "alertmanager/apiVersion.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false
error validating "alertmanager/data.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "alertmanager/kind.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false
error validating "alertmanager/metadata.yaml": error validating data: [apiVersion not set, kind not set]; if you choose to ignore these errors, turn validation off with --validate=false
error validating "alertmanager/type.yaml": error validating data: invalid object to validate; if you choose to ignore these errors, turn validation off with --validate=false

When I put it as the last line into example.jsonnet file, jsonnet compile will fail.

[ArvinDevel]

@brancz , I want to delete some of the default dashboards, is there some quick way to do it?
Essentially, I want to modify some of the default dashboards, but found the exported json file can't be loaded to the grafana using tuitorial, so I want to delete original first, and then import them.

[ArvinDevel]

@brancz I realized how to override default config of grafanadashbords, using

  // without `+` to override
grafanaDashboards:: {
}

But that would be better if there exists some doc explaining the fields of every object, for example, the fields of grafanaDashboards, so that we can do set favourite/star dashboard and other stuff easily.

[brancz]

@ArvinDevel as mentioned in the beginning, any of the examples depicted should be run in your own infrastructure repository, not within this repository and dependency management is handled with jsonnet-bundler.

The example you picked just renders the Alertmanager secret, you can remove the .alertmanager.secret at the end to generate the entire stack.

The examples always show the minimal code to demonstrate the respective configuration, so it's meant to be a demonstration and then for you to pick the examples into your setup and make all the modifications to your need.

But that would be better if there exists some doc explaining the fields of every object, for example, the fields of grafanaDashboards, so that we can do set favourite/star dashboard and other stuff easily.

Please do let us know where we can improve this, the jsonnet implementation is still relatively young, so yes there are tons of opportunities to improve docs. As you look at this with a fresh pair of eyes, we would highly appreciate if you could even drop us a pull request to improve what we have! :slightly_smiling_face:

Essentially, I want to modify some of the default dashboards, but found the exported json file can't be loaded to the grafana using tuitorial, so I want to delete original first, and then import them.

This is strange, we regularly test all of these and it all works for us. Can you elaborate what doesn't work and what you want to modify? Maybe we can help. :slightly_smiling_face:

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in last 60d. Thank you for your contributions.

[brancz]

Closing due to inactivity.