Kube-prometheus: Jsonnet Bundler fails to install kube-prometheus

[DimitrijeManic]

What happened?
jsonnet bundler fails to install this library, most likely due to a versioning issue
https://github.com/jsonnet-bundler/jsonnet-bundler/issues/81

Did you expect to see some different?
kube-prometheus & dependencies to be installed

How to reproduce it (as minimally and precisely as possible):
Taken from the current RADME.

mkdir /tmp/my-kube
cd /tmp/my-kube
go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
jb init
jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]

this fails with

Cloning into 'vendor/.tmp/jsonnetpkg-kube-prometheus-release-0.1225842681'...
remote: Enumerating objects: 32, done.
remote: Counting objects: 100% (32/32), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 7639 (delta 11), reused 15 (delta 0), pack-reused 7607
Receiving objects: 100% (7639/7639), 4.36 MiB | 5.68 MiB/s, done.
Resolving deltas: 100% (4598/4598), done.
Branch 'release-0.1' set up to track remote branch 'release-0.1' from 'origin'.
Switched to a new branch 'release-0.1'
>>> Installed kube-prometheus version release-0.1
Cloning into 'vendor/.tmp/jsonnetpkg-ksonnet-master703480580'...
remote: Enumerating objects: 5, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 2505 (delta 0), reused 3 (delta 0), pack-reused 2500
Receiving objects: 100% (2505/2505), 2.48 MiB | 3.95 MiB/s, done.
Resolving deltas: 100% (933/933), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
>>> Installed ksonnet version master
Cloning into 'vendor/.tmp/jsonnetpkg-kubernetes-mixin-release-0.1157117075'...
remote: Enumerating objects: 95, done.
remote: Counting objects: 100% (95/95), done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 2207 (delta 53), reused 33 (delta 17), pack-reused 2112
Receiving objects: 100% (2207/2207), 5.37 MiB | 8.13 MiB/s, done.
Resolving deltas: 100% (1430/1430), done.
Branch 'release-0.1' set up to track remote branch 'release-0.1' from 'origin'.
Switched to a new branch 'release-0.1'
>>> Installed kubernetes-mixin version release-0.1
Cloning into 'vendor/.tmp/jsonnetpkg-grafonnet-master294048470'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 1774 (delta 0), reused 8 (delta 0), pack-reused 1765
Receiving objects: 100% (1774/1774), 1023.68 KiB | 2.95 MiB/s, done.
Resolving deltas: 100% (1060/1060), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
>>> Installed grafonnet version master
Cloning into 'vendor/.tmp/jsonnetpkg-grafana-builder-master073880637'...
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 5648 (delta 23), reused 21 (delta 7), pack-reused 5598
Receiving objects: 100% (5648/5648), 15.01 MiB | 13.00 MiB/s, done.
Resolving deltas: 100% (1932/1932), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
>>> Installed grafana-builder version master
Cloning into 'vendor/.tmp/jsonnetpkg-grafana-master591154808'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 531 (delta 0), reused 5 (delta 0), pack-reused 524
Receiving objects: 100% (531/531), 144.19 KiB | 1.04 MiB/s, done.
Resolving deltas: 100% (240/240), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
>>> Installed grafana version master
Cloning into 'vendor/.tmp/jsonnetpkg--master078274935'...
remote: Enumerating objects: 9, done.
remote: Counting objects: 100% (9/9), done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 1774 (delta 0), reused 8 (delta 0), pack-reused 1765
Receiving objects: 100% (1774/1774), 1023.68 KiB | 2.55 MiB/s, done.
Resolving deltas: 100% (1060/1060), done.
Already on 'master'
Your branch is up to date with 'origin/master'.
jb: error: failed to install: failed to install package: failed to move package: rename vendor/.tmp/jsonnetpkg--master078274935/grafonnet vendor: no such file or directory

[metalmatze]

For now try using jsonnet bundle version v0.2. Master branch of jb has a few unreleased changes that might be the problem here.

[DimitrijeManic]

@metalmatze how can I install this version?

[metalmatze]

There are pre compiled binaries for Linux and Mac on
https://github.com/jsonnet-bundler/jsonnet-bundler/releases/tag/v0.2.0

[DimitrijeManic]

Thanks, I tried as you suggested but it seems to be still failing

$ jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]
GET https://github.com/coreos/kube-prometheus/archive/a4a143a9ab770d72eb0f7dd44bf8e72184347944.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/0ee0da08a9691201a118d30f6b396381c2c8d017.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/23c20eb8df5047c4735d2d341d4a7bab6f87870b.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/18fbf558ab7f8809fd610a3dc50bf483508dc1bb.tar.gz 200
GET https://github.com/coreos/etcd/archive/d6a3c995cf86b479cb5a44b48d000feb33e3d8f8.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/c459106d2d2b583dd3a83f6c75eb52abee3af764.tar.gz 200
GET https://github.com/kausalco/public/archive/7ac7da1a0fe165b68cdb718b2521b560d51bd1f4.tar.gz 200
jb: error: failed to install packages: downloading: failed to create tmp dir: stat vendor/.tmp: no such file or directory

[gerrit8143]

i have the problem with jb v0.2.0, but master works without problems

[fentas]

I was using v0.1 and the error occurred - was the reason I upgraded to v0.2 but same error.

> jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]

GET https://github.com/coreos/kube-prometheus/archive/989c6813aac21790bb38dd55057fd505d972d274.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/7f3e0130ccd3e39400d1dc36e690cab16f8d4881.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/23c20eb8df5047c4735d2d341d4a7bab6f87870b.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/1aa773ddbbbd9b05405e9785e0190b975b1faadc.tar.gz 200
GET https://github.com/coreos/etcd/archive/d6a3c995cf86b479cb5a44b48d000feb33e3d8f8.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/87971fff907386ff2ab9f3c92411b7190e5d809c.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/3e1b0f1bee40af140c1349ab401392882ed658de.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/7ac7da1a0fe165b68cdb718b2521b560d51bd1f4.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/c459106d2d2b583dd3a83f6c75eb52abee3af764.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/7f3e0130ccd3e39400d1dc36e690cab16f8d4881.tar.gz 200
GET https://github.com/metalmatze/slo-libsonnet/archive/437c402c5f3ad86c3c16db8471f1649284fef0ee.tar.gz 200
jb: error: failed to install packages: downloading: failed to create tmp dir: stat vendor/.tmp: no such file or directory

and vendor folder vanishes.
So I do this while install :smile_cat:

while true; do [ -d vendor/.tmp ] || mkdir vendor/.tmp; done

Then it goes through

> jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]

GET https://github.com/coreos/kube-prometheus/archive/989c6813aac21790bb38dd55057fd505d972d274.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200
GET https://github.com/kubernetes-monitoring/kubernetes-mixin/archive/7f3e0130ccd3e39400d1dc36e690cab16f8d4881.tar.gz 200
GET https://github.com/brancz/kubernetes-grafana/archive/23c20eb8df5047c4735d2d341d4a7bab6f87870b.tar.gz 200
GET https://github.com/coreos/prometheus-operator/archive/1aa773ddbbbd9b05405e9785e0190b975b1faadc.tar.gz 200
GET https://github.com/coreos/etcd/archive/d6a3c995cf86b479cb5a44b48d000feb33e3d8f8.tar.gz 200
GET https://github.com/prometheus/prometheus/archive/87971fff907386ff2ab9f3c92411b7190e5d809c.tar.gz 200
GET https://github.com/prometheus/node_exporter/archive/3e1b0f1bee40af140c1349ab401392882ed658de.tar.gz 200
GET https://github.com/grafana/jsonnet-libs/archive/7ac7da1a0fe165b68cdb718b2521b560d51bd1f4.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/c459106d2d2b583dd3a83f6c75eb52abee3af764.tar.gz 200
GET https://github.com/metalmatze/slo-libsonnet/archive/437c402c5f3ad86c3c16db8471f1649284fef0ee.tar.gz 200
GET https://github.com/grafana/grafonnet-lib/archive/c459106d2d2b583dd3a83f6c75eb52abee3af764.tar.gz 200
GET https://github.com/ksonnet/ksonnet-lib/archive/0d2f82676817bbf9e4acf6495b2090205f323b9f.tar.gz 200

While it fetches all I can see the different projects. But in the end, it removes all - only ksonnet remains :thinking:

> ls -la vendor/
total 16
drwxr-xr-x  4 user user 4096 19. Feb 18:45 ./
drwxr-xr-x  5 user user 4096 19. Feb 18:45 ../
drwx------ 12 user user 4096 19. Feb 18:45 ksonnet/
drwxr-xr-x  2 user user 4096 19. Feb 18:45 .tmp/

Tried also different releases 0.2 0.3 0.4 all the same

[metalmatze]

Can you try cloning kube-prometheus checking out the version and then running jb install, that one works for me...

git clone [email protected]:coreos/kube-prometheus.git -b release-0.3
jb install

[s-urbaniak]

@metalmatze the above works for me too, but not when vendoring from a different repository (i.e. cluster-monitoring-operator)

[gerhard]

I was hitting exactly the same issue when running jb update (v0.2.0). Running jb install in the master branch worked.

What is the difference between different release branches? Should I be using master, release-0.3 or release-0.4? I'm running K8S v1.16.2.

[zpoley]

This took me a while to figure out as well, so I thought I would share my steps for anyone else who might be confused. On Mac OS X for me:

1. Install jb v0.2.0 as suggested by @metalmatze above. cp jb-darwin-amd64 ~/go/bin/jb, chmod +x ~/go/bin/jb.
2. Create a new directory for your build as described in the Installing instructions. mkdir my-kube-prometheus; cd my-kube-prometheus.
3. In my-kube-prometheus, run jb init.
4. Run jb install github.com/coreos/kube-prometheus/jsonnet/[email protected], release-0.4 worked for me with jb v0.2.0.
5. Before compiling, install gojsontoyaml and jsonnet as detailed in the Compiling Instructions. go get github.com/brancz/gojsontoyaml, and brew install jsonnet for me.
6. Copy the build.sh script into the my-kube-prometheus directory.
7. Copy the example.jsonnet script into the my-kube-prometheus directory.
8. Run build.sh

[brancz]

@zpoley I think this would make for a great tl;dr give me commands to run to get started section in the readme. What do you think?

[m1o1]

Ran into this also - using jb version 0.3.1 release. Downgrading to 0.2.0 worked though.

May be relevant for reproducing: I ran it with a Git Bash shell (mingw) on Windows from a tmp directory (made with mktemp -d) that I git cloned the repo into.

[carminalazu]

We deploy [email protected] onto K8S 1.17 from LKE with a single command:

make lke-metrics

This is what that command translated into on macOS:

mkdir -p tmp/kube-prometheus
mkdir -p /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& cd /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& /usr/bin/curl --progress-bar --fail --location --output /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 "https://github.com/jsonnet-bundler/jsonnet-bundler/releases/download/v0.3.1/jb-darwin-amd64" \
&& touch /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 \
&& chmod +x /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 --version 2>&1 \
   | grep "0.3.1" \
&& ln -sf /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb
printf "\nInitialize Jsonnet bundle ...\n"
cd tmp/kube-prometheus \
&& rm -fr * \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 init
printf "\nInstall [email protected] ...\n"
cd tmp/kube-prometheus \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/jb-0.3.1-darwin-amd64 install github.com/coreos/kube-prometheus/jsonnet/[email protected]
brew install go-jsonnet
mkdir -p /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& cd /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& /usr/bin/curl --progress-bar --fail --location --output /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 "https://github.com/mikefarah/yq/releases/download/3.3.0/yq_darwin_amd64" \
&& touch /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 \
&& chmod +x /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 --version 2>&1 \
   | grep "3.3.0" \
&& ln -sf /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq
printf "\nGenerate kube-prometheus manifests ...\n"
cd tmp/kube-prometheus \
&& rm -fr manifests \
&& mkdir -p manifests/setup \
&& /usr/local/bin/jsonnet \
      --jpath vendor \
      --multi manifests \
      --ext-str NAMESPACE="metrics" \
      --ext-str GITHUB_OAUTH_APP_CLIENT_ID="$(/usr/local/bin/lpass show --notes 6262503396338294422)" \
      --ext-str GITHUB_OAUTH_APP_CLIENT_SECRET="$(/usr/local/bin/lpass show --notes 6396745408918286971)" \
      --ext-str GITHUB_ORG=thechangelog \
      --ext-str GITHUB_TEAM_ID=3796119 \
      --ext-str ROOT_URL=https://metrics.changelog.com \
      /Users/carminalazu/github.com/thechangelog/changelog.com/k8s/kube-prometheus.jsonnet \
   | while read file; do /Users/carminalazu/github.com/thechangelog/changelog.com/bin/yq_darwin_amd64 read --prettyPrint $file > $file.yml && rm $file; done
mkdir -p /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& cd /Users/carminalazu/github.com/thechangelog/changelog.com/bin \
&& /usr/bin/curl --progress-bar --fail --location --output /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 "https://github.com/k14s/ytt/releases/download/v0.27.1/ytt-darwin-amd64" \
&& touch /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 \
&& chmod +x /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 version \
   | grep 0.27.1 \
&& ln -sf /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt
/usr/local/Cellar/kubernetes-cli/1.18.2/bin/kubectl apply --filename tmp/kube-prometheus/manifests/setup \
&& /usr/local/Cellar/kubernetes-cli/1.18.2/bin/kubectl apply --filename tmp/kube-prometheus/manifests \
&& /Users/carminalazu/github.com/thechangelog/changelog.com/bin/ytt-0.27.1-darwin-amd64 \
    --data-value namespace=metrics \
    --file /Users/carminalazu/github.com/thechangelog/changelog.com/k8s/metrics-changelog > /Users/carminalazu/github.com/thechangelog/changelog.com/k8s/metrics-changelog.yml \
&& /usr/local/Cellar/kubernetes-cli/1.18.2/bin/kubectl apply --filename k8s/metrics-changelog.yml

I know, it's a lot of stuff, but it's comprehensive and idempotent, with decent solutions that will save time and complexity further down the road:

• fast go-jsonnet instead of slow jsonnet
• simple JSON to YAML conversion via yq
• YAML templating via ytt

If you take a look at https://github.com/thechangelog/changelog.com, you will find everything that makes the above possible, as well pvc mixins for both Prometheus & Grafana + a few other important add-ons such as nginx-ingress & external-dns. The Grafana pvc chown was a pain-in-the-neck to deal with (I'm suspecting it's a K8S <1.18 issue) and the nginx-ingress integration was simpler via ytt'ed YAML instead of jsonnet. YMMV

Hope this helps others 💚

cc @jerodsanto @Qtrain

[m1o1]

Actually, release-0.5 fails with this issue:

+ jb install github.com/coreos/kube-prometheus/jsonnet/[email protected]
GET https://github.com/coreos/kube-prometheus/archive/486fa0797bc1556f2540fda4ce4f4d6101a560a7.tar.gz 200
jb.exe: error: failed to install packages: downloading: failed to create tmp dir: CreateFile vendor\.tmp: The system cannot find the path specified.

Master also fails. 0.4 and 0.3 both work though. Using jb 0.2, though the latest version of jb seems to fix it up until it gets to grafonnet:

jb.exe: error: failed to install packages: downloading: failed to create tmp dir: mkdir C:\Users\{MY_NAME}\prom-helm-test\vendor\jsonnet\vendor\.tmp\jsonnetpkg-github.com-grafana-grafonnet-lib\grafonnet-master055835561: The system cannot find the path specified.

Edit: All versions work if I do it in a linux docker container, so maybe the issue is Windows only?

[brancz]

There were recently some bug fixes for windows I believe, could you try master?

[m1o1]

That's interesting... when I installed the latest jb with:

$ GO111MODULE=on go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@master
go: downloading github.com/jsonnet-bundler/jsonnet-bundler v0.4.1-0.20200608221931-15c0828a2f87
go: found github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb in github.com/jsonnet-bundler/jsonnet-bundler v0.4.1-0.20200608221931-15c0828a2f87

All versions of kube-prometheus are installing okay for me now. I could have sworn I tried jb@master - must have been a very recent change. Maybe I didn't include the @ master and it just got the latest tagged version that wasn't new enough. Edit* Yep looks like that's probably what happened.

I don't see 0.4.1 in the jsonnet-bundler repo - 0.4.0 is the last tag, so I'm not sure where Go is getting that from (maybe it auto increments the patch number when using master?), but it works.

[brancz]

Yeah this a go mod weirdness, in any case, happy that worked! :)

This issue has gotten a bit fragmented, so if anyone still has issues, please feel free to open a new issue, I'm going to close this one for now.