Kube-prometheus: Manifests compilation failing out of the box (possibly due to grafana.libsonnet)

Created on 23 Dec 2020  ·  11Comments  ·  Source: prometheus-operator/kube-prometheus

What happened?
Fresh install starting from scratch following docs does not compile (fails on grafana.libsonnet)

in my test I installed the release-0.4 branch because my kubernetes version is 1.17.x but I also tried with master just to test.

This is the main manifest (basic.jsonnet)


local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-all-namespaces.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',

      // given we include the all-namespaces patch above, we need to remove the builtin namespaces
      // otherwise additional (useless) rbacs will be created
      prometheus+:: {
        namespaces: [],
      },
    }, 
  };

local manifests =
  { ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
  {
    ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
    for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
  } +
  // serviceMonitor is separated so that it can be created after the CRDs are ready
  { 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
  { ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
  { ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
  { ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
  { ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
  { ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
  { ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) };

local kustomizationResourceFile(name) = './manifests/' + name + '.yaml';
local kustomization = {
  apiVersion: 'kustomize.config.k8s.io/v1beta1',
  kind: 'Kustomization',
  resources: std.map(kustomizationResourceFile, std.objectFields(manifests)),
};

manifests {
  'kustomization': kustomization,
}

Did you expect to see some different?
yes, to compile the manifests.

How to reproduce it (as minimally and precisely as possible):
Create a new directory and place the above basic.jsonnet files alongside the following ones:
jsonnetfile.json:

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/kube-prometheus.git",
          "subdir": "jsonnet/kube-prometheus"
        }
      },
      "version": "release-0.4"
    }
  ],
  "legacyImports": true
}

jsonnetfile.lock.json:

{
  "version": 1,
  "dependencies": [
    {
      "source": {
        "git": {
          "remote": "https://github.com/brancz/kubernetes-grafana.git",
          "subdir": "grafana"
        }
      },
      "version": "4204279da8d3d6317116ee161ac706fadbba9193",
      "sum": "VUavLhri7lTnJ2V7F9lDlL+K96NwIhqqlxMtasYBs3Q="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/coreos/etcd.git",
          "subdir": "Documentation/etcd-mixin"
        }
      },
      "version": "a3174d0f8ec6ec58827d7d86448bb4df08ae69e4",
      "sum": "bLqTqEr0jky9zz5MV/7ucn6H5mph2NlXas0TVnGNB1Y="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/coreos/prometheus-operator.git",
          "subdir": "jsonnet/prometheus-operator"
        }
      },
      "version": "8b9d024467383d84b55d7e5e0d4f7a33eb5007b3",
      "sum": "exdWZ9NLaBId9tUmkHnAx2zECdVvhfy3RUQsSqI96Fk="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/grafonnet-lib.git",
          "subdir": "grafonnet"
        }
      },
      "version": "4cd00bb4e372bd3ea460c05cece770f9f66831e8",
      "sum": "uJqGLrC9O1zy/Of6IjsveIUbS6I8MCbLMa1e3h3EwKs="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/grafana/jsonnet-libs.git",
          "subdir": "grafana-builder"
        }
      },
      "version": "b662248e4dce100139afb050244503c83bd095c1",
      "sum": "9/eJqljTTtJeq9QRjabdKWL6yD8a7VzLmGKBK3ir77k="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/ksonnet/ksonnet-lib.git",
          "subdir": ""
        }
      },
      "version": "0d2f82676817bbf9e4acf6495b2090205f323b9f",
      "sum": "h28BXZ7+vczxYJ2sCt8JuR9+yznRtU/iA6DCpQUrtEg=",
      "name": "ksonnet"
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes-monitoring/kubernetes-mixin.git",
          "subdir": ""
        }
      },
      "version": "f0a5099c8214241842b827a08eba6d7b515550ea",
      "sum": "n3D/4k8wAW5NDjmwyvca/GS18ofBoS7Z5F9d3rWiqfw="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/kubernetes-monitoring/kubernetes-mixin.git",
          "subdir": "lib/promgrafonnet"
        }
      },
      "version": "243110f771b791e00b52b85c2eb9d20f085c93bd",
      "sum": "zv7hXGui6BfHzE9wPatHI/AGZa4A2WKo6pq7ZdqBsps="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/metalmatze/slo-libsonnet.git",
          "subdir": "slo-libsonnet"
        }
      },
      "version": "e238df4fac957357d78a405966c51523ef151cbc",
      "sum": "5XrxbGgMFMHnKGeE++fiduDlDEiDunM2ZMTYCmPbWgA="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus-operator/kube-prometheus.git",
          "subdir": "jsonnet/kube-prometheus"
        }
      },
      "version": "68692f7aec038090bfba951bdf821dd8f1c08e75",
      "sum": "3K86dkG/R9Ejw0SAD84K7quqYsoCLVUA3bdCxeRcMHk="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus/node_exporter.git",
          "subdir": "docs/node-mixin"
        }
      },
      "version": "8a3cc9db052cb12beae8565a363d77d08dce8a24",
      "sum": "rvyiD/yCB4BeYAWqYF53bP8c+aCUt2ipLHW2Ea8ELO8="
    },
    {
      "source": {
        "git": {
          "remote": "https://github.com/prometheus/prometheus.git",
          "subdir": "documentation/prometheus-mixin"
        }
      },
      "version": "e5478983c43cb839a77c144f953b9ea25783b995",
      "sum": "6kUzElCBWZ5U/3cxEpHNMmoKKPubG45QxpmLu8PGg08=",
      "name": "prometheus"
    }
  ],
  "legacyImports": false
}

Then install the dependencies and try to generate the manifests:

❯ jb install
[...]
❯ jsonnet -J vendor -m manifests basic.jsonnet
RUNTIME ERROR: Field does not exist: config
    vendor/grafana/grafana.libsonnet:52:20-44   thunk from <object <anonymous>>
    vendor/grafana/grafana.libsonnet:52:9-45    object <anonymous>
    basic.jsonnet:30:72-82  thunk from <thunk <manifests> from <$>>
    <std>:1293:24-25    thunk from <function <anonymous>>
    <std>:1293:5-33 function <anonymous>
    basic.jsonnet:30:55-83  thunk <manifests> from <$>
    basic.jsonnet:30:3-85
    basic.jsonnet:30:3-85
    basic.jsonnet:39:1-10   $
    During evaluation

Environment

  • Kubernetes cluster kind:

    kind, version 1.17.x.

  • Manifests:

    see above

Anything else we need to know?:
Before opening an issue I reached out over slack. For more background info and what has been tried, you can refer to this thread.

kinbug
Source
picture elisiano

All 11 comments

Experiencing the same issue

RUNTIME ERROR: Field does not exist: config
        vendor/grafana/grafana.libsonnet:52:20-44       thunk from <object <anonymous>>
        vendor/grafana/grafana.libsonnet:52:9-45        object <anonymous>
        grafana-dashboards.libsonnet:41:70-80   thunk from <$>
        <std>:1278:24-25        thunk from <function <anonymous>>
        <std>:1278:5-33 function <anonymous>
        grafana-dashboards.libsonnet:41:53-81   $


        During evaluation
mmn01-sky picture mmn01-sky on 23 Dec 2020

also seeing the same thing:

√ jcorley@j-corley-md6m:~/tmp % ~/go/bin/jb install
GET https://github.com/prometheus-operator/kube-prometheus/archive/68692f7aec038090bfba951bdf821dd8f1c08e75.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/8b9d024467383d84b55d7e5e0d4f7a33eb5007b3.tar.gz 200
GET https://github.com/coreos/etcd/archive/a3174d0f8ec6ec58827d7d86448bb4df08ae69e4.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/e5478983c43cb839a77c144f953b9ea25783b995.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/8a3cc9db052cb12beae8565a363d77d08dce8a24.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/f0a5099c8214241842b827a08eba6d7b515550ea.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/4204279da8d3d6317116ee161ac706fadbba9193.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/4cd00bb4e372bd3ea460c05cece770f9f66831e8.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/b662248e4dce100139afb050244503c83bd095c1.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/243110f771b791e00b52b85c2eb9d20f085c93bd.tar.gz 200
GET https://github.com/metalmatze/slo-libsonnet/archive/e238df4fac957357d78a405966c51523ef151cbc.tar.gz 200
√ jcorley@j-corley-md6m:~/tmp % jsonnet -J vendor -m manifests basic.jsonnet
RUNTIME ERROR: Field does not exist: config
    vendor/grafana/grafana.libsonnet:52:20-44   thunk from <object <anonymous>>
    vendor/grafana/grafana.libsonnet:52:9-45    object <anonymous>
    basic.jsonnet:29:72-82  thunk from <thunk <manifests> from <$>>
    <std>:1293:24-25    thunk from <function <anonymous>>
    <std>:1293:5-33 function <anonymous>
    basic.jsonnet:29:55-83  thunk <manifests> from <$>
    basic.jsonnet:29:3-85   
    basic.jsonnet:29:3-85   
    basic.jsonnet:38:1-10   $
    During evaluation   

?1 jcorley@j-corley-md6m:~/tmp %
jpackagejasonc picture jpackagejasonc on 23 Dec 2020

I'm not the best with jsonnet but after some digging around and seeing it complain about 'vendor/grafana/grafana.libsonnet' which I think it points towards this repo - https://github.com/brancz/kubernetes-grafana/tree/master/grafana

_Checking that file out it does not have a config section._ - Forget this... my terminal was messed up.... it does have a config section.

Adam2Marsh picture Adam2Marsh on 23 Dec 2020

I have same your issue when I follow document.

jb install https://github.com/prometheus-operator/kube-prometheus/jsonnet/[email protected]

./build.sh example.jsonnet 
+ set -o pipefail
+ rm -rf manifests
+ mkdir -p manifests/setup
+ jsonnet -J vendor -m manifests example.jsonnet
+ xargs '-I{}' sh -c 'cat {} | gojsontoyaml > {}.yaml; rm -f {}' -- '{}'
RUNTIME ERROR: Field does not exist: config
        vendor/grafana/grafana.libsonnet:52:20-43
        vendor/grafana/grafana.libsonnet:52:9-44        object <anonymous>
        vendor/kube-prometheus/kube-prometheus.libsonnet:(16:36)-(44:1) object <anonymous>
        example.jsonnet:27:70-79        thunk <o>
        std.jsonnet:957:28
        std.jsonnet:957:9-36    function <anonymous>
        example.jsonnet:27:53-80        thunk <a>
        example.jsonnet:27:1-82 function <anonymous>
        example.jsonnet:27:1-82
thanaphat1991 picture thanaphat1991 on 24 Dec 2020

Also ran into this..

Not positive.. but it looks like this change in the main prometheus repository (upstream) is overwriting the grafana config provided by the grafana lib: https://github.com/prometheus/prometheus/pull/8287

Swapping out line 41 in the vendor/prometheus/config.libsonnet seems to correct.
Old
grafana: {

New
grafana+: {

Edit:

This was my workaround when using release-0.3 of kube-prometheus which is using prometheus master under the hood to pull prometheus-mixin (see vendor/kube-prometheus/jsonnetfile.json), I have not looked at other versions.

toscott picture toscott on 28 Dec 2020

@toscott is right. I experience the same thing when updating to the latest dependencies.

ath88 picture ath88 on 30 Dec 2020

@brancz: I feel this is a better place to discuss your comment ( https://github.com/prometheus/prometheus/pull/8287#issuecomment-752681120 ).

As you mentioned there, there is already a plan to refactor kube-prometheus to avoid a global config ( https://github.com/prometheus-operator/kube-prometheus/issues/566 ).

That doesn't help users facing this issue right now though. Is there any known workaround that users can implement until #566 is done?

elisiano picture elisiano on 30 Dec 2020

Since https://github.com/prometheus/prometheus/pull/8287 breaks current user behavior, I think we should just add the +: in prometheus/prometheus. I'll open a PR shortly.

brancz picture brancz on 30 Dec 2020
👍1

Since prometheus/prometheus#8287 breaks current user behavior, I think we should just add the +: in prometheus/prometheus. I'll open a PR shortly.

Perfect, thank you. Will this be back-ported to older branches as well?

elisiano picture elisiano on 30 Dec 2020

I believe all versions of kube-prometheus use latest of the prometheus mixin, so this should happen automatically for every kube-prometheus release after running jb update once it's merged.

brancz picture brancz on 30 Dec 2020
👍1

Thanks @brancz, https://github.com/prometheus/prometheus/pull/8332 fixed this issue, closing.

elisiano picture elisiano on 31 Dec 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

KlavsKlavsen picture KlavsKlavsen  ·  3Comments
brancz picture brancz  ·  5Comments
arpita-hub picture arpita-hub  ·  3Comments
saraAlizadeh picture saraAlizadeh  ·  5Comments
Aracki picture Aracki  ·  6Comments