Kubevirt: make cluster-sync not working.
Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
What happened:
after make cluster-up
make cluster-sync fails .
The images are not build and hence while pushing images to local registry it fails.
What you expected to happen:
Images should be build and visible in host machine docker images output
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Log
$ make cluster-sync
PUSH_LOG_FILE= ./hack/cluster-build.sh
Building ...
[+] Building 0.2s (9/9) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load metadata for docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [1/4] FROM docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> => resolve docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 33B 0.0s
=> CACHED [2/4] RUN mkdir -p /go && source /etc/profile.d/gimme.sh && go get github.com/mattn/goveralls && go get -u github.com/golang/mock/gomock && go get -u git 0.0s
=> CACHED [3/4] RUN pip install j2cli && pip3 install operator-courier==1.3.0 0.0s
=> CACHED [4/4] COPY rsyncd.conf /etc/rsyncd.conf 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:6e512c03eb7b69fd0cca43aa2266799c72fead3ffa535c0390cfd7ae493d0ac8 0.0s
=> => naming to docker.io/library/kubevirt 0.0s
go version go1.11.5 linux/amd64
go version go1.11.5 linux/amd64
INFO: Build option --define has changed, discarding analysis cache.
INFO: Analysed 54 targets (10 packages loaded, 13248 targets configured).
INFO: Found 54 targets...
INFO: Elapsed time: 2.877s, Critical Path: 1.27s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8939 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.936s, Critical Path: 0.10s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8939 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.854s, Critical Path: 0.09s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8941 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.906s, Critical Path: 0.11s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
INFO: Build option --platforms has changed, discarding analysis cache.
INFO: Analysed target //:build-virtctl (0 packages loaded, 8307 targets configured).
INFO: Found 1 target...
Target //:build-virtctl up-to-date:
bazel-genfiles/virtctl-copier
INFO: Elapsed time: 0.723s, Critical Path: 0.11s
INFO: 1 process: 1 processwrapper-sandbox.
INFO: Build completed successfully, 2 total actions
INFO: Build completed successfully, 2 total actions
[+] Building 0.1s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B
=> [internal] load metadata for docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [1/4] FROM docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> => resolve docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 33B 0.0s
=> CACHED [2/4] RUN mkdir -p /go && source /etc/profile.d/gimme.sh && go get github.com/mattn/goveralls && go get -u github.com/golang/mock/gomock && go get -u git 0.0s
=> CACHED [3/4] RUN pip install j2cli && pip3 install operator-courier==1.3.0 0.0s
=> CACHED [4/4] COPY rsyncd.conf /etc/rsyncd.conf 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:6e512c03eb7b69fd0cca43aa2266799c72fead3ffa535c0390cfd7ae493d0ac8 0.0s
=> => naming to docker.io/library/kubevirt 0.0s
go version go1.11.5 linux/amd64
go version go1.11.5 linux/amd64
devel devel_alt
devel
INFO: Build options --define and --platforms have changed, discarding analysis cache.
INFO: Analysed target //:push-images (0 packages loaded, 13125 targets configured).
INFO: Found 1 target...
Target //:push-images up-to-date:
bazel-bin/push-images
INFO: Elapsed time: 1.678s, Critical Path: 0.32s
INFO: 0 processes.
INFO: Build completed successfully, 20 total actions
INFO: Build completed successfully, 20 total actions
F0726 08:20:35.171421 24317 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-handler:devel: (503, 'Service Unavailable')
F0726 08:20:35.173954 24319 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-operator:devel: (503, 'Service Unavailable')
F0726 08:20:35.174829 24315 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-launcher:devel: (503, 'Service Unavailable')
F0726 08:20:35.226880 24312 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/example-cloudinit-hook-sidecar:devel: (503, 'Service Unavailable')
F0726 08:20:35.227720 24304 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/example-hook-sidecar:devel: (503, 'Service Unavailable')
F0726 08:20:35.228996 24308 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/subresource-access-test:devel: (503, 'Service Unavailable')
F0726 08:20:35.230473 24324 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/vm-killer:devel: (503, 'Service Unavailable')
F0726 08:20:35.234380 24306 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/container-disk-v1alpha:devel: (503, 'Service Unavailable')
F0726 08:20:35.246773 24321 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/cirros-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.264621 24318 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-api:devel: (503, 'Service Unavailable')
F0726 08:20:35.286813 24320 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/alpine-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.289851 24325 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/winrmcli:devel: (503, 'Service Unavailable')
F0726 08:20:35.299104 24330 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/fedora30-cloud-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.305036 24323 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/fedora-cloud-container-disk-demo:devel: (503, 'Service Unavailable')
F0726 08:20:35.305849 24329 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/cdi-http-import-server:devel: (503, 'Service Unavailable')
.
.
. <snip>
.
Environment:
- KubeVirt version (use
virtctl version): - Kubernetes version (use
kubectl version): - VM or VMI specifications:
- Cloud provider or hardware configuration:
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools:
- Others:
Master git commit: f680f4122bfa8da5bfb27cba8a51d5692e2df8b6
$ go version
go version go1.12.4 linux/amd64
updates
OS : Linux HP-workstation 4.15.0-54-generic #58-Ubuntu
Docker version: 19.03.1
KUBEVIRT_PROVIDER = k8s-1.13.3
kunalkushwaha
All 24 comments
Hi @kunalkushwaha, thanks for reporting this issue. Please, which OS are you running? What docker version are you using? Also do you have any nod default setup on you machine - eg virtualization solution.
Thanks a lot.
petrkotas
on 26 Jul 2019
also, which KUBEVIRT_PROVIDER do you use? I just tried it with the k8s-1.13.3 default provider and it worked (on current master branch, commit 292310f82a96e727e69c7cb7b5e692a46dd7f446)
slintes
on 26 Jul 2019
OS : Linux HP-workstation 4.15.0-54-generic #58-Ubuntu
Docker version: 19.03.1
KUBEVIRT_PROVIDER = k8s-1.13.3
@slintes @petrkotas I just updated the required information.
Also, I tried to do with make clean, but still images are not visible in docker images output.
kunalkushwaha
on 26 Jul 2019
$ make bazel-build-images
.
.
7978 blocks
Verifying packages...
Preparing packages...
qemu-img-2:2.11.2-5.fc28.x86_64
Verifying packages...
Preparing packages...
qemu-guest-agent-2:2.11.2-5.fc28.x86_64
Verifying packages...
Preparing packages...
stress-1.0.4-20.fc28.x86_64
Verifying packages...
Preparing packages...
libstdc++-8.3.1-2.fc28.x86_64
Verifying packages...
Preparing packages...
capstone-3.0.5-1.fc28.x86_64
Verifying packages...
Preparing packages...
libaio-0.3.110-11.fc28.x86_64
Verifying packages...
Preparing packages...
e2fsprogs-1.44.2-0.fc28.x86_64
Target //:build-images up-to-date (nothing to build) <-- Is this expected?
INFO: Elapsed time: 271.148s, Critical Path: 48.95s
INFO: 1653 processes: 1653 processwrapper-sandbox.
INFO: Build completed successfully, 1739 total actions
even after make clean, make bazel-build-images skips building images and push to local registry.
I see this line
Target //:build-images up-to-date (nothing to build)
Is this correct behaviour?
Is there any way, I can skip the cache check, and bazel forcefully build without cache?
kunalkushwaha
on 29 Jul 2019
@kunalkushwaha I am honestly not sure what is happening. I have not encountered this error before. Maybe @cynepco3hahue may help, since he build the bazel tooling.
petrkotas
on 29 Jul 2019
even after make clean, make bazel-build-images skips building images and push to local registry.
That the images aren't rebuild after a clean is strange indeed. That no push happens is expected, because bazel-build-images does not push at all. That's done by bazel-push-images
The bazel server is running in docker, and all cached files are in a docker volume. You can try do delete the latter and retry:
$ docker kill kubevirt-bazel-server && docker rm kubevirt-bazel-server
$ docker volume rm kubevirt
I tried deleting the kubevirt-bazel-server container , but never deleted volume. Maybe this will fix.
I am trying to rebuild again after deleting volume.
kunalkushwaha
on 29 Jul 2019
@kunalkushwaha Can you please provide the output of docker ps command on the environment when do you run make cluster-up, probably F0726 08:20:35.171421 24317 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-handler:devel: (503, 'Service Unavailable') happens because it refers wrong registry port.
cynepco3hahue
on 29 Jul 2019
The ports have changed, as I created cluster again.
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
80f21b0ec665 kubevirtci/k8s-1.13.3 "/bin/bash -c '/vm.s…" 55 seconds ago Up 54 seconds k8s-1.13.3-node01
e48f5b847a0c registry:2.7.1 "/entrypoint.sh /etc…" 55 seconds ago Up 54 seconds k8s-1.13.3-registry
2080ec374838 kubevirtci/k8s-1.13.3 "/bin/bash -c /dnsma…" 59 seconds ago Up 58 seconds 127.0.0.1:8443->8443/tcp, 0.0.0.0:32843->2201/tcp, 0.0.0.0:32842->5000/tcp, 0.0.0.0:32841->5901/tcp, 0.0.0.0:32840->6443/tcp k8s-1.13.3-dnsmasq
3a45e8c0c776 kubevirt "/entrypoint.sh hack…" 19 minutes ago Up 19 minutes kubevirt-bazel-server
22eb02e05003 kubevirt "/entrypoint.sh /usr…" 19 minutes ago Up 19 minutes 0.0.0.0:32839->873/tcp suspicious_heisenberg
But I had checked that and I was able to push other images into registry on same port and pull it in kubevirt k8s VM from registry.
The problem is images e.g (localhost:32803/kubevirt/virt-handler:devel) should be visible after build images on host with docker images. Its not visible.
even after I removed the docker volume kubvevirt and build again.
kunalkushwaha
on 29 Jul 2019
nope it is not, currently we are building via bazel, so all images exist under the bazel cache and not under the local registry
cynepco3hahue
on 29 Jul 2019
The error has changed now.
Target //:build-virtctl up-to-date:
bazel-genfiles/virtctl-copier
INFO: Elapsed time: 26.367s, Critical Path: 22.15s
INFO: 279 processes: 279 processwrapper-sandbox.
INFO: Build completed successfully, 280 total actions
INFO: Build completed successfully, 280 total actions
[+] Building 0.2s (9/9) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 38B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [1/4] FROM docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> => resolve docker.io/kubevirt/builder@sha256:08b4db91e5f6272004892c6174129172c1bef5c6906bde9f8414cb945a588e12 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 33B 0.0s
=> CACHED [2/4] RUN mkdir -p /go && source /etc/profile.d/gimme.sh && go get github.com/mattn/goveralls && go get -u github.com/golang/mock/gomock && go get -u git 0.0s
=> CACHED [3/4] RUN pip install j2cli && pip3 install operator-courier==1.3.0 0.0s
=> CACHED [4/4] COPY rsyncd.conf /etc/rsyncd.conf 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:6e512c03eb7b69fd0cca43aa2266799c72fead3ffa535c0390cfd7ae493d0ac8 0.0s
=> => naming to docker.io/library/kubevirt 0.0s
go version go1.11.5 linux/amd64
go version go1.11.5 linux/amd64
devel devel_alt
devel
[bazel FATAL src/main/cpp/blaze.cc:1325] Unknown startup option: '-dfs'.
For more info, run 'bazel help startup_options'.
devel_alt
[bazel FATAL src/main/cpp/blaze.cc:1325] Unknown startup option: '-dfs'.
For more info, run 'bazel help startup_options'.
[+] Building 0.1s (9/9) FINISHED
- I updated the code to master ( earlier code was of last week)
- removed the docker volume
kubevirt
complete log https://gist.github.com/kunalkushwaha/b85f45ccc086023afaac72120ff425cb
kunalkushwaha
on 29 Jul 2019
@cynepco3hahue got it. Please check here https://github.com/kubevirt/kubevirt/issues/2532#issuecomment-515902657 error is changed now.
kunalkushwaha
on 29 Jul 2019
@kunalkushwaha hm, what exactly command do you run?
cynepco3hahue
on 29 Jul 2019
$ make bazel-build-images
$ make cluster-sync
error happens on make cluster-sync
kunalkushwaha
on 29 Jul 2019
@kunalkushwaha Give me some time will check it locally.
cynepco3hahue
on 29 Jul 2019
@kunalkushwaha hm I check it locally after the rebase, and all works as expected :worried:
cynepco3hahue
on 30 Jul 2019
Can proxy be issues?
Though I am able to push image to registry form host and pull it from
KubeVirt VM.
Can you give me pointer to debug the issue? Like check in cache if images
are build, and place where I need to look for pushing images
On Tue, Jul 30, 2019, 7:34 PM Artyom Lukianov notifications@github.com
wrote:
@kunalkushwaha https://github.com/kunalkushwaha hm I check it locally
after the rebase, and all works as expected 😟—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubevirt/kubevirt/issues/2532?email_source=notifications&email_token=AAVBOIOP4MN2HU2P3RZHP7TQCAKMRA5CNFSM4IHBVRE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3DRMXY#issuecomment-516363871,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAVBOIJDSI4RSTRB5TACKJ3QCAKMRANCNFSM4IHBVREQ
.
kunalkushwaha
on 30 Jul 2019
Mh, the push is trying to push to localhost:32803
F0726 08:20:35.171421 24317 fast_pusher_.py:194] Error publishing localhost:32803/kubevirt/virt-handler:devel: (503, 'Service Unavailable')
But I don't see that port open:
2080ec374838 kubevirtci/k8s-1.13.3 "/bin/bash -c /dnsma…" 59 seconds ago Up 58 seconds 127.0.0.1:8443->8443/tcp, 0.0.0.0:32843->2201/tcp, 0.0.0.0:32842->5000/tcp, 0.0.0.0:32841->5901/tcp, 0.0.0.0:32840->6443/tcp k8s-1.13.3-dnsmasq
The correct port should be 32842, the one which is being forwared to 5000.
What value do you have for docker_prefix in _ci-configs/k8s-1.13.3/config-provider-k8s-1.13.3.sh? Can you delete _ci-configs and retry? That should recreate that file with the correct value hopefully :thinking:
slintes
on 30 Jul 2019
I just came into a similar situation and noticed that make push ignores KUBEVIRT_PROVIDER and always uses settings of k8s-1.13.3, which does not work if you started another cluster. make cluster-sync should work though.
slintes
on 30 Jul 2019
the make push issue mentioned in the last comment will be fixed by https://github.com/kubevirt/kubevirt/pull/2549
slintes
on 30 Jul 2019
The correct port should be 32842, the one which is being forwared to 5000
The output is from different run, so ports changed.
I feel this error is due to proxy settings
I have one question:
F0731 00:55:18.670257 4551 fast_pusher_.py:194] Error publishing localhost:32849/kubevirt/virt-handler:devel: (503, 'Service Unavailable')
this line i.e. push logic executes on host or within container?
kunalkushwaha
on 31 Jul 2019
@slintes @cynepco3hahue @petrkotas It got resolved :)
Thanks for all help :)
----- edit ---
bazel push script understand proxy with
HTTP_PROXY&HTTPS_PROXYsetting not in small cases i.e.http(s)_proxy
bazel running inside container kubevirt-bazel-server do not need proxy setting to push images.
- so setting
no_proxy=localhostalong with proxy settings in Dockerfile resolves the problem.
kunalkushwaha
on 31 Jul 2019
glad that you sorted this out! :slightly_smiling_face:
slintes
on 31 Jul 2019
@kunalkushwaha thanks for sharing the how-to solve this issue. And I am glad that this issue is sorted out.
petrkotas
on 31 Jul 2019
Related issues
MarkSpencerTan
·
7Comments
dhiller
·
4Comments
vatsalparekh
·
11Comments
enp0s3
·
6Comments
lxs137
·
11Comments