Skaffold: Jib example doesn't build
Actual behavior
The jib example doesn't build. The basic example (examples/getting-started) works well. My console logs :
➜ jib git:(master) skaffold dev --default-repo localhost:5000
Starting build...
Found [minikube] context, using local docker daemon.
Building [localhost:5000/gcr_io_k8s-skaffold_skaffold-jib]...
[INFO] Scanning for projects...
[INFO]
[INFO] -------------------< org.skaffold:hello-spring-boot >-------------------
[INFO] Building hello-spring-boot 0.1.0
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:resources (default-resources) @ hello-spring-boot ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/vmaubert/GIT/skaffold/examples/jib/src/main/resources
[INFO] skip non existing resourceDirectory /home/vmaubert/GIT/skaffold/examples/jib/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) @ hello-spring-boot ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:3.0.2:testResources (default-testResources) @ hello-spring-boot ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/vmaubert/GIT/skaffold/examples/jib/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) @ hello-spring-boot ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hello-spring-boot ---
[INFO] No tests to run.
[INFO]
[INFO] --- jib-maven-plugin:0.9.11:dockerBuild (default-cli) @ hello-spring-boot ---
[WARNING] Base image 'gcr.io/distroless/java' does not use a specific image digest - build may not be reproducible
[INFO]
[INFO] Containerizing application to Docker daemon as jib__087e74d83f207d1cd326fc12cd7e0110bb02da61...
[INFO]
[INFO] Getting base image gcr.io/distroless/java...
[INFO] Building dependencies layer...
[INFO] Building classes layer...
[INFO] Finalizing...
[INFO] Loading to Docker daemon...
[INFO]
[INFO] Container entrypoint set to [java, -Djava.security.egd=file:/dev/./urandom, -XX:+UnlockExperimentalVMOptions, -XX:+UseCGroupMemoryLimitForHeap, -cp, /app/resources:/app/classes:/app/libs/*, hello.Application]
[INFO]
[INFO] Built image to Docker daemon as jib__087e74d83f207d1cd326fc12cd7e0110bb02da61
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 9.040 s
[INFO] Finished at: 2018-11-09T10:56:35+01:00
[INFO] ------------------------------------------------------------------------
Cleaning up...
Cleanup complete in 66.502926ms
FATA[0012] exiting dev mode because the first build failed: building [localhost:5000/gcr_io_k8s-skaffold_skaffold-jib]: digest not found
Information
- Skaffold version: v0.18.0
- Operating system: Fedora 29
- I have a local registry (localhost:5000)
Thank you :-)
vmaubert
All 12 comments
Thank you for filing this issue!
It's odd that the jib build finishes successfully but then we fail - if I have to guess, it looks like something around the push with jib is not working with the default-repo setup
/cc @loosebazooka @coollog @briandealwis
balopat
on 9 Nov 2018
I wonder if it has something to do with communicating with the local registry (since we ask it for the digest). Can you tell us some more about the local registry?
loosebazooka
on 9 Nov 2018
@loosebazooka It's a simple docker registry launched with docker run -d -p 5000:5000 registry:2. I pushed another image inside it without any problem.
Maybe this could help you :
➜ jib git:(master) docker images --digests
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
registry 2 sha256:5a156ff125e5a12ac7fdec2b90b7e2ae5120fa249cf62248337b6d04abc574c8 2e2f252f3c88 8 weeks ago 33.3MB
jib__087e74d83f207d1cd326fc12cd7e0110bb02da61 latest <none> 17a575dc8325 48 years ago 134MB
I don't understand the name jib__087e74d83f207d1cd326fc12cd7e0110bb02da61, I expected something like localhost:5000/gcr_io_k8s-skaffold_skaffold-jib :-/
Thank you for your help :-)
vmaubert
on 9 Nov 2018
Hi @vmaubert , could you share with us your skaffold.yaml?
Since it looks like you're using minikube, Skaffold tells Jib to actually build to the Docker daemon, so it isn't building directly to your local registry as you might have wanted it to. The jib__087e74d83f207d1cd326fc12cd7e0110bb02da61 is a unique image tag that Jib generates when it builds to the Docker daemon so that it can retag it with the tag the tagger configured for Skaffold. It is just a temporary id that can be ignored or deleted afterwards.
coollog
on 9 Nov 2018
Hi @coollog , my skaffold.yaml is this one. I just cloned the skaffold repository, I didn't change anything.
vmaubert
on 9 Nov 2018
Did you happen to change the image to your own (like localhost:5000/skaffold-jib) in both the skaffold.yaml artifact image and the k8s yaml (web.yaml)?
Regardless, it looks to me like the issue here is that Jib built to the minikube docker daemon but Skaffold is trying to resolve the image on your local registry and thus fails. Nevermind, Skaffold turns the image (gcr.io/k8s-skaffold/skaffold-jib) into a repo under localhost:5000 (localhost:5000/gcr_io_k8s-skaffold_skaffold-jib).
coollog
on 9 Nov 2018
Did you happen to change the image to your own (like localhost:5000/skaffold-jib) in both the skaffold.yaml artifact image and the k8s yaml (web.yaml)?
Yes, exactly the same error :-/
vmaubert
on 9 Nov 2018
@vmaubert it works for me on my machine with minikube. Could you try running with -v debug?
This is what I see:
```
[...]
[INFO] Built image to Docker daemon as jib__087e74d83f207d1cd326fc12cd7e0110bb02da61
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.186 s
[INFO] Finished at: 2018-11-09T16:22:05-05:00
[INFO] ------------------------------------------------------------------------
DEBU[0006] Running command: [git rev-parse --short HEAD]
DEBU[0006] Command output: stdout b7bf0cff
, stderr:
DEBU[0006] Running command: [git status . --porcelain]
DEBU[0006] Command output: stdout , stderr:
DEBU[0006] Running command: [git describe --tags --exact-match]
Build complete in 4.084562969s
Starting test...
Test complete in 5.366µs
Starting deploy...
DEBU[0006] Running command: [kubectl version --client -ojson]
DEBU[0006] Command output: stdout {
"clientVersion": {
"major": "1",
"minor": "11",
"gitVersion": "v1.11.2",
"gitCommit": "bb9ffb1654d4a729bb4cec18ff088eacc153c239",
"gitTreeState": "clean",
"buildDate": "2018-08-07T23:17:28Z",
"goVersion": "go1.10.3",
"compiler": "gc",
"platform": "darwin/amd64"
}
}
, stderr:
kubectl client version: 1.11
kubectl version 1.12.0 or greater is recommended for use with skaffold
DEBU[0006] manifests apiVersion: v1
kind: Service
metadata:
name: web
spec:
ports:
- port: 8080
name: http
type: LoadBalancer
selector:
app: web
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: gcr.io/k8s-skaffold/skaffold-jib
DEBU[0006] manifests with tagged images apiVersion: v1
kind: Service
metadata:
name: web
spec:
ports:
- name: http
port: 8080
selector:
app: web
type: LoadBalancer
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- image: localhost:5000/gcr_io_k8s-skaffold_skaffold-jib:b7bf0cff
name: web
DEBU[0006] 2 manifests to deploy. 2 are updated or new
DEBU[0006] Running command: [kubectl --context minikube apply --force -f -]
service/web created
deployment.apps/web created
DEBU[0006] Labels are not applied to service [web] because of issue: https://github.com/GoogleContainerTools/skaffold/issues/887
DEBU[0006] Patching web in namespace default
Deploy complete in 268.936032ms
Watching for changes every 1s...
INFO[0007] Stream logs from pod: web-78b6c95865-xqnw4 container: web
DEBU[0007] Running command: [mvn --quiet --non-recursive jib:_skaffold-files]
[web-78b6c95865-xqnw4 web]
[web-78b6c95865-xqnw4 web] . ____ _ __ _ _
[web-78b6c95865-xqnw4 web] /\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
[web-78b6c95865-xqnw4 web] ( ( )___ | '_ | '_| | '_ \/ _` | \ \ \ \
[web-78b6c95865-xqnw4 web] \/ ___)| |_)| | | | | || (_| | ) ) ) )
[web-78b6c95865-xqnw4 web] ' |____| .__|_| |_|_| |___, | / / / /
[web-78b6c95865-xqnw4 web] =========|_|==============|___/=/_/_/_/
[...]
briandealwis
on 9 Nov 2018
@coollog: Regardless, it looks to me like the issue here is that Jib built to the minikube docker daemon but Skaffold is trying to resolve the image on your local registry and thus fails.
I tried that (eval $(minikube docker-env); skaffold dev -v debug --default-repo localhost:5000) and it worked too.
briandealwis
on 9 Nov 2018
Ah right, Skaffold automatically makes the image as a repo under localhost:5000.
coollog
on 9 Nov 2018
Hi @briandealwis ,
I see a lot less stuff than you with -v debug:
skaffold dev --default-repo localhost:5000 -v debug
INFO[0000] Skaffold &{Version:v0.18.0 ConfigVersion:skaffold/v1alpha5 GitVersion: GitCommit:34651689be78b2c6bcfbace5072b00b93661f895 GitTreeState:clean BuildDate:2018-11-08T22:32:00Z GoVersion:go1.10.5 Compiler:gc Platform:linux/amd64}
DEBU[0000] Defaulting build type to local build
DEBU[0000] Defaulting deploy type to kubectl
INFO[0000] Using kubectl context: minikube
DEBU[0000] Using builder: local
DEBU[0000] Running command: [minikube docker-env --shell none]
DEBU[0000] Command output: stdout DOCKER_TLS_VERIFY=1
DOCKER_HOST=tcp://192.168.39.131:2376
DOCKER_CERT_PATH=/home/vmaubert/.minikube/certs
DOCKER_API_VERSION=1.35
, stderr:
DEBU[0000] push value not present, defaulting to false because localCluster is true
DEBU[0000] Running command: [mvn --quiet --non-recursive jib:_skaffold-files]
DEBU[0002] Command output: stdout /home/vmaubert/GIT/skaffold/examples/jib/pom.xml
[...]
[INFO] Built image to Docker daemon as jib__087e74d83f207d1cd326fc12cd7e0110bb02da61
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.702 s
[INFO] Finished at: 2018-11-10T11:18:36+01:00
[INFO] ------------------------------------------------------------------------
DEBU[0008] Defaulting build type to local build
DEBU[0008] Defaulting deploy type to kubectl
INFO[0008] Using kubectl context: minikube
DEBU[0008] Using builder: local
DEBU[0008] push value not present, defaulting to false because localCluster is true
Cleaning up...
DEBU[0008] manifests apiVersion: v1
kind: Service
metadata:
name: web
spec:
ports:
- port: 8080
name: http
type: LoadBalancer
selector:
app: web
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
containers:
- name: web
image: gcr.io/k8s-skaffold/skaffold-jib
DEBU[0008] Running command: [kubectl --context minikube delete --ignore-not-found=true -f -]
Cleanup complete in 58.669392ms
FATA[0008] exiting dev mode because the first build failed: building [localhost:5000/gcr_io_k8s-skaffold_skaffold-jib]: digest not found
I also tried eval $(minikube docker-env); skaffold dev -v debug --default-repo localhost:5000and it works !!! So, Skaffold doesn't automatically use the good docker deamon, is it the expected behavior?
vmaubert
on 10 Nov 2018
@vmaubert I just tried repeating the example and it's now failing on my side in the same way. I must have made a mistake in configuring my tests previously.
This actually makes sense: Skaffold is (currently) not passing any environment to the Jib build requests, and so Jib just picks up the surrounding environment. A workaround for now is to explicitly set the DOCKER_* variables before starting Skaffold, like:
(eval $(minikube docker-env); skaffold dev --default-repo localhost:5000 -v debug)
This all works with the Bazel builder as it builds to an image tarball and then loads the tarball into the provided Docker APIClient, which is configured from the minikube docker-env call seen earlier in the build log. Unfortunately the environment that is received is tossed and we can't recreate an environment from a given docker APIClient. But I think we could add the environment, if applicable, to the APIClient and then use it when calling out to the Jib build commands.
briandealwis
on 12 Nov 2018
Related issues
Hudsonzp
·
4Comments
r2d4
·
3Comments
ianlewis
·
4Comments
yurchenkosv
·
3Comments
kdevu
·
3Comments