inspection-report-20200211_171636.tar.gz
- Error Message:
$ microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Enabling juju...
Kubeflow could not be enabled:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
22 116M 22 25.8M 0 0 51231 0 0:39:39 0:08:48 0:30:51 71301
curl: (18) transfer closed with 94828385 bytes remaining to read
Command '('microk8s-enable.wrapper', 'juju')' returned non-zero exit status 1
Failed to enable kubeflow
- microk8s version:
$ sudo snap find microk8s
Name Version Publisher Notes Summary
microk8s v1.17.2 canonical✓ classic Kubernetes for workstations and appliances
Maybe this issue is same as #943 but I'm a microk8s newbie and cannot make a
correct judgement about it.
titsuki
All 72 comments
I'm having a similar problem enabling kubeflow, slightly different error
microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Enabling juju...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (77) Problem with the SSL CA cert (path? access rights?)
Command '('microk8s-enable.wrapper', 'juju')' returned non-zero exit status 1
snap find microk8s
Name Version Publisher Notes Summary
microk8s v1.17.2 canonical✓ classic Kubernetes for workstations and appliances
mattk08
on 11 Feb 2020
https://github.com/ubuntu/microk8s/issues/943#issuecomment-581424578
suggested
microk8s.disable kubeflow
microk8s.enable kubeflow
but that did not help
mattk08
on 11 Feb 2020
My error message looks like this:
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 2
Failed to enable kubeflow
andrewcheny
on 11 Feb 2020
I think this is the same as #943
andrewcheny
on 11 Feb 2020
@titsuki I think the problem you are facing is slow internet connection. If I read the curl output correctly you are downloading with 71k and the juju client is 116M so curl at some point gave up.
@mattk08 are you behind a firewall/proxy? For some reason curl failed to initiate the ssl handshake and download the juju client.
@andrewcheny in your case the juju client is available but the bootstrap failed. Could you share the microk8s.inspect tarball and the result of mirok8s.juju controllers. Thank you.
ktsakalozos
on 12 Feb 2020
are you behind a firewall/proxy? For some reason curl failed to initiate the ssl handshake and download the juju client.
@ktsakalozos I am behind a firewall (no proxy), however, this is the first issue I've had downloading any packages with any package manager.
I was able to find the URL curl was using in enable.juju.sh
run_with_sudo "${SNAP}/usr/bin/curl" -L https://launchpad.net/juju/$JUJU_SERIES/$JUJU_VERSION/+download/juju-$JUJU_VERSION-centos7.tar.gz -o "$SNAP_DATA/tmp/juju.tar.gz"
I was able to download the tar file fine outside of the script. I went ahead and just ran each command in the enable.juju.sh script by hand to enable juju and then tried running microk8s.enable kubeflow and that worked, I think, I might have another problem unrelated to this issue.
mattk08
on 12 Feb 2020
@ktsakalozos seems I see the same problem, and it only affects ubuntu desktop for whatever reason.
Clean installation on 18.04 server, as well as inside multipass machine on the desktop works fine.
On the desktop itself the container start dying immediately when trying to bootstrap juju, and the node turns into "NotReady".
Tried on 2 different machine with the same result.
$ microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
andreygx1 NotReady
$ microk8s.kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default test-db5495c67-gthfl 0/1 ContainerCreating 0 59m
ingress nginx-ingress-microk8s-controller-bmqq8 0/1 Unknown 0 85m
kube-system coredns-7b67f9f8c-88htl 0/1 Unknown 0 90m
kube-system dashboard-metrics-scraper-687667bb6c-nnvw5 0/1 Unknown 0 85m
kube-system heapster-v1.5.2-5c58f64f8b-hvl5t 0/4 Unknown 0 85m
kube-system hostpath-provisioner-7b9cb5cdb4-mt7fb 0/1 Unknown 0 85m
kube-system kubernetes-dashboard-5c848cc544-c5l9r 0/1 Unknown 0 85m
kube-system monitoring-influxdb-grafana-v4-6d599df6bf-65dkc 0/2 Unknown 0 85m
$ microk8s.inspect
[sudo] password for agrebennikov:
Sorry, try again.
[sudo] password for agrebennikov:
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
WARNING: Docker is installed.
File "/etc/docker/daemon.json" does not exist.
You should create it and add the following lines:
{
"insecure-registries" : ["localhost:32000"]
}
and then restart docker with: sudo systemctl restart docker
Building the report tarball
Report tarball is at /var/snap/microk8s/1173/inspection-report-20200213_220150.tar.gz
grebennikov
on 14 Feb 2020
grebennikov
on 14 Feb 2020
The issue starts happening when rbac is enabled. In this case I can't spin up pods with "mountvolume.setup failed for volume "default-token-XXX": failed to sync secret cache: timed out waiting for the condition" error.
Once the rbac is disabled pods can be run again.
grebennikov
on 14 Feb 2020
@grebennikov could you help me reproduce this? When you say "it only affects ubuntu desktop" what do you mean?
If you sudo snap remove microk8s and reinstall it with sudo snap install microk8s --classic does the problem persist? Does the microk8s.reset have any effect? Thanks.
ktsakalozos
on 14 Feb 2020
@ktsakalozos Thanks for your advice!
AFAIK, curl or wget are robust enough to download a large file under poor internet connection speed (but I think my internet connection speed isn't very lower than the average in Japan).
I tried curl resume option (i.e., -C -) and checked if retry-and-resume approach could work but it doesn't work. The juju package on lanuchpad ( https://launchpad.net/juju/ ) seems not capable to resume.
Isn't there any other better repository that hosts the juju package?
titsuki
on 16 Feb 2020
Isn't there any other better repository that hosts the juju package?
Would you be able to manualy run the commands found in the microk8s.enable juju script at https://github.com/ubuntu/microk8s/blob/8ec87c8de7258e5c8f13f22e0cccbfd427a096fa/microk8s-resources/actions/enable.juju.sh ?
ktsakalozos
on 17 Feb 2020
@ktsakalozos
I tried but I couldn't.
$ microk8s.enable juju
Installing Juju...
[sudo] itoyota のパスワード:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
14 116M 14 16.6M 0 0 80574 0 0:25:12 0:03:36 0:21:36 99k
curl: (18) transfer closed with 104486649 bytes remaining to read
Failed to enable juju
@ktsakalozos I could observe the same error as $ microk8s.enable juju caused.
$ curl -L "https://launchpad.net/juju/2.7/2.7.1/+download/juju-2.7.1-centos7.tar.gz" --output juju-2.7.1-centos7.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
13 74.0M 13 10.1M 0 0 31634 0 0:40:54 0:05:35 0:35:19 139k
curl: (18) transfer closed with 67051539 bytes remaining to read
You could snap install juju and then cp /snap/juju/current/bin/juju /var/snap/microk8s/current/bin/. Note this is a hack.
ktsakalozos
on 17 Feb 2020
@ktsakalozos Your idea works fine to me. snap seems 30~40 times faster than launchpad w.r.t. downloading speed.
titsuki
on 17 Feb 2020
@ktsakalozos This also works to enable juju. But when I then try to enable Kubeflow, it gives me the following message.
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Enabling juju...
Deploying Kubeflow...
Kubeflow could not be enabled:
ERROR cannot load ssh client keys: mkdir /var/snap/microk8s/1173/juju/share: permission denied
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 2
Failed to enable kubeflow
dgrahn
on 18 Feb 2020
I am also facing the "Problem with the SSL CA cert" issue. It seems curl from snapd has different behavior than curl in OS. I sniffed the connection and curl-snapd has a FYN,ACK after the TCP handshake, while curl-OS issues a TLS1.2 Client Hello. No idea why.
[root@kubeflow-lab ~]# /var/lib/snapd/snap/microk8s/1173/usr/bin/curl -L https://launchpad.net/juju/$JUJU_SERIES/$JUJU_VERSION/+download/juju-$JUJU_VERSION-centos7.tar.gz -o juju.tar.gz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (77) Problem with the SSL CA cert (path? access rights?)
[root@kubeflow-lab ~]# /var/lib/snapd/snap/microk8s/1173/usr/bin/curl --version
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP UnixSockets
[root@kubeflow-lab ~]# curl -L https://launchpad.net/juju/$JUJU_SERIES/$JUJU_VERSION/+download/juju-$JUJU_VERSION-centos7.tar.gz -o juju.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
100 116M 100 116M 0 0 1670k 0 0:01:11 0:01:11 --:--:-- 2816k
[root@kubeflow-lab ~]# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.44 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets
evertonberz
on 20 Feb 2020
@dgrahn I faced the same error but this error itself could solve by changing the permission.
There are some other problems:
- microk8s cannot recognize juju is already installed or not if you use the copied juju.
- microk8s doesn't seem to have a skip option that ignores already installed or enabled module.
titsuki
on 24 Feb 2020
@titsuki we have recently addressed these issues with the PR https://github.com/ubuntu/microk8s/pull/927 You could try the candidate channel to verify the fix: sudo snap install microk8s --classic --channel=candidate
ktsakalozos
on 24 Feb 2020
@ktsakalozos Thanks for your advice. Fortunately, I could make juju enabled (by trying $ microk8s.enable juju many time) but $ microk8s.enable kubeflow caused another error.
This error says https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-54/archive?channel=stable is unavailable but when I checked manually this link seems to work fine:
Deploying Kubeflow...
Kubeflow could not be enabled:
Located bundle "cs:bundle/kubeflow-170"
Resolving charm: cs:~kubeflow-charmers/ambassador-54
Resolving charm: cs:~kubeflow-charmers/argo-controller-53
Resolving charm: cs:~kubeflow-charmers/argo-ui-54
Resolving charm: cs:~kubeflow-charmers/jupyter-controller-54
Resolving charm: cs:~kubeflow-charmers/jupyter-web-56
Resolving charm: cs:~kubeflow-charmers/katib-controller-52
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/katib-manager-50
Resolving charm: cs:~kubeflow-charmers/katib-ui-46
Resolving charm: cs:~kubeflow-charmers/kubeflow-dashboard-11
Resolving charm: cs:~kubeflow-charmers/kubeflow-gatekeeper-14
Resolving charm: cs:~kubeflow-charmers/kubeflow-login-13
Resolving charm: cs:~kubeflow-charmers/kubeflow-profiles-16
Resolving charm: cs:~kubeflow-charmers/metacontroller-43
Resolving charm: cs:~kubeflow-charmers/metadata-api-6
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/metadata-ui-10
Resolving charm: cs:~kubeflow-charmers/minio-54
Resolving charm: cs:~kubeflow-charmers/modeldb-backend-49
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/modeldb-store-45
Resolving charm: cs:~kubeflow-charmers/modeldb-ui-44
Resolving charm: cs:~kubeflow-charmers/pipelines-api-58
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/pipelines-persistence-53
Resolving charm: cs:~kubeflow-charmers/pipelines-scheduledworkflow-55
Resolving charm: cs:~kubeflow-charmers/pipelines-ui-53
Resolving charm: cs:~kubeflow-charmers/pipelines-viewer-55
Resolving charm: cs:~kubeflow-charmers/pytorch-operator-55
Resolving charm: cs:~kubeflow-charmers/tf-job-dashboard-55
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-53
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-54": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-54": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-54/archive?channel=stable: dial tcp: i/o timeout
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow', '--channel', 'stable', '--overlay', '/tmp/tmp81lxmw43')' returned non-zero exit status 1
I follow the kubeflow document to install kubeflow with Microk8s:
sudo snap install microk8s --classic --channel=candidate
microk8s.status --wait-ready
microk8s.enable dns dashboard storage
microk8s.enable gpu
microk8s.enable kubeflow
But with this error messages:
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
[sudo] password for eli:
Enabling juju...
Deploying Kubeflow...
Kubeflow could not be enabled:
Creating Juju controller "uk8s" on microk8s/localhost
Creating k8s resources for controller "controller-uk8s"
ERROR failed to bootstrap model: creating controller stack for controller: creating statefulset for controller: timed out waiting for controller pod: pending: -
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
JIElite
on 25 Feb 2020
@dgrahn: Running these commands should fix the issue you're facing with the permission denied error:
https://github.com/ubuntu/microk8s/blob/050e98b/snap/hooks/configure#L380-L382
https://github.com/ubuntu/microk8s/blob/050e98b/snap/hooks/configure#L386
Where $SNAP_DATA is /var/snap/microk8s/current
Additionally, #989 will fix this issue generally, as it just includes the juju binary in the snap, instead of downloading it after snap install.
knkski
on 28 Feb 2020
@JIElite: I haven't seen that error before. Can you create a new issue with the output from KUBEFLOW_DEBUG=true microk8s.enable kubeflow?
knkski
on 28 Feb 2020
Same problem here on centos 7.7. Installed via snap:
sudo snap install microk8s --classic --channel=candidate
microk8s.enable dns dashboard storage
microk8s.enable kubeflow
Getting the error:
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Enabling juju...
Kubeflow could not be enabled:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (77) Problem with the SSL CA cert (path? access rights?)
Command '('microk8s-enable.wrapper', 'juju')' returned non-zero exit status 1
Failed to enable kubeflow
egkiastas
on 3 Mar 2020
@egkiastas: Can you download https://launchpad.net/juju/2.7/2.7.3/+download/juju-2.7.3-k8s.tar.xz, and place the unzipped juju binary at /var/snap/microk8s/current/bin/juju, then run these commands, and try again?
https://github.com/ubuntu/microk8s/blob/0ebbaed/snap/hooks/configure#L378-L385
989 will fix the issue you're running into by including the juju binary in the snap instead of downloading it. If you'd like to try that instead of the above, you should be able to by switching microk8s to the latest/edge channel (sudo snap switch microk8s --channel latest/edge && sudo snap refresh).
knkski
on 4 Mar 2020
@egkiastas: Can you download https://launchpad.net/juju/2.7/2.7.3/+download/juju-2.7.3-k8s.tar.xz, and place the unzipped
jujubinary at/var/snap/microk8s/current/bin/juju, then run these commands, and try again?https://github.com/ubuntu/microk8s/blob/0ebbaed/snap/hooks/configure#L378-L385
989 will fix the issue you're running into by including the juju binary in the snap instead of downloading it. If you'd like to try that instead of the above, you should be able to by switching microk8s to the
latest/edgechannel (sudo snap switch microk8s --channel latest/edge && sudo snap refresh).
I'm getting the same error as OP and I followed this, but it didn't help.
Here's my error:
`sudo microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Enabling juju...
Deploying Kubeflow...
Kubeflow could not be enabled:
ERROR microk8s:
running: false
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit
status 1
Failed to enable kubeflow`
amitmsingh
on 4 Mar 2020
Hi all,
I am also getting an error for enabling kubeflow on Ubuntu 18.04. I have also tried both options suggested by @knkski and did not succeed. My error is:
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-54": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-54": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-54/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow', '--channel', 'stable', '--overlay', '/tmp/tmpp902pdg2')' returned non-zero exit status 1
Failed to enable kubeflow
Thanks a lot.
SrdjanStevanetic
on 20 Mar 2020
Hi all,
I am getting the simillar error on Ubuntu 18.04.
after running:
sudo microk8s.enable kubeflow
I get the following error:
nabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling rbac...
Deploying Kubeflow...
Kubeflow could not be enabled:
Creating Juju controller "uk8s" on microk8s/localhost
Creating k8s resources for controller "controller-uk8s"
ERROR failed to bootstrap model: creating controller stack for controller: creating statefulset for controller: timed out waiting for controller pod: unschedulable: pvc storage-controller-0 pending due to ExternalProvisioning - waiting for a volume to be created, either by external provisioner "microk8s.io/hostpath" or manually created by system administrator
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow.
Any idea how to fix it ? Thank you very much!
sygin123
on 21 Mar 2020
@amitmsingh: If you switch to the edge version of microk8s, it includes a fix for this error:
# If you don't have it installed
sudo snap install microk8s --classic --edge
# If you have it installed
sudo snap switch microk8s --channel edge
sudo snap refresh microk8s
Note that with the edge version, you'll have to use the edge kubeflow bundle, so you'll need to enable microk8s like this:
KUBEFLOW_CHANNEL=edge microk8s.enable kubeflow
That requirement should disappear when microk8s 1.18 hits stable, which is targeted for this Thursday (March 26th).
knkski
on 23 Mar 2020
@SrdjanStevanetic: It looks like you're having network connectivity issues. Can you create a new issue in this repository, and post the output from KUBEFLOW_DEBUG=true microk8s.enable kubeflow? At a guess, you have a proxy or firewall in place that's causing issues with api.jujucharms.com.
knkski
on 23 Mar 2020
@sygin123: I haven't seen that error before. Can you create a new issue in this repository, and include the output from KUBEFLOW_DEBUG=true microk8s.enable kubeflow? And are you running microk8s inside of a VM or somewhere without a lot of disk space? It looks like you may be running into an issue with disk space.
knkski
on 23 Mar 2020
@sygin123: I haven't seen that error before. Can you create a new issue in this repository, and include the output from
KUBEFLOW_DEBUG=true microk8s.enable kubeflow? And are you running microk8s inside of a VM or somewhere without a lot of disk space? It looks like you may be running into an issue with disk space.
@knkski I am not running inside VM. I think you are right, my laptop does not have much space for /home. After extend the space, this issue is gone! Thanks!
sygin123
on 24 Mar 2020
@amitmsingh: If you switch to the edge version of microk8s, it includes a fix for this error:
# If you don't have it installed sudo snap install microk8s --classic --edge # If you have it installed sudo snap switch microk8s --channel edge sudo snap refresh microk8sNote that with the edge version, you'll have to use the edge kubeflow bundle, so you'll need to enable microk8s like this:
KUBEFLOW_CHANNEL=edge microk8s.enable kubeflowThat requirement should disappear when microk8s 1.18 hits stable, which is targeted for this Thursday (March 26th).
I am also facing the same issue. Even after running edge, I am getting following error.
Enabling dns...
Enabling storage...
Enabling rbac...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Deploying Kubeflow...
Kubeflow could not be enabled:
ERROR controller "uk8s" already exists
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
Regards,
Manish
guptam
on 24 Mar 2020
@guptam: Can you run microk8s.disable kubeflow && microk8s.enable kubeflow? If that doesn't work, microk8s.juju unregister -y uk8s should fix that error.
knkski
on 25 Mar 2020
Was switching to the edge channel supposed to help with the "misbehaving server" error or is that a separate issue? I'm still seeing this:
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-158
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-77": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-77": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-77/archive?channel=edge: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow', '--channel', 'edge', '--overlay', '/tmp/tmpyb4zgwqi')' returned non-zero exit status 1
Failed to enable kubeflow
atomic77
on 25 Mar 2020
@atomic77: That looks like the same issue as in #1050, can you post the output over there from KUBEFLOW_DEBUG=true microk8s.enable kubeflow, and attach the file generated by microk8s.inspect?
knkski
on 25 Mar 2020
Problem persists with v1.18.0/stable.
$ microk8s kubectl get nodes
NAME STATUS ROLES AGE VERSION
predator-2.60 Ready
$ microk8s kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.152.183.1
The Error returned is:
$ microk8s enable kubeflow
Enabling dns...
Enabling storage...
Enabling rbac...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Deploying Kubeflow...
Kubeflow could not be enabled:
Creating Juju controller "uk8s" on microk8s/localhost
Creating k8s resources for controller "controller-uk8s"
ERROR failed to bootstrap model: creating controller stack for controller: creating statefulset for controller: timed out waiting for controller pod: pending: -
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
SpudAldridge
on 27 Mar 2020
Got the same issue as above
17:22:16 ERROR juju.cmd.juju.commands bootstrap.go:778 failed to bootstrap model: creating controller stack for controller: creating statefulset for controller: timed out waiting for controller pod: unschedulable: pvc storage-controller-0 pending due to ExternalProvisioning - waiting for a volume to be created, either by external provisioner "microk8s.io/hostpath" or manually created by system administrator
17:22:16 DEBUG juju.cmd.juju.commands bootstrap.go:779 (error details: [{/workspace/_build/src/github.com/juju/juju/cmd/juju/commands/bootstrap.go:853: failed to bootstrap model} {/workspace/_build/src/github.com/juju/juju/environs/bootstrap/bootstrap.go:611: } {/workspace/_build/src/github.com/juju/juju/environs/bootstrap/bootstrap.go:284: } {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/k8s.go:411: creating controller stack for controller} {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/bootstrap.go:343: creating statefulset for controller} {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/bootstrap.go:603: } {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/bootstrap.go:737: timed out waiting for controller pod} {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/bootstrap.go:687: unschedulable} {/workspace/_build/src/github.com/juju/juju/caas/kubernetes/provider/bootstrap.go:667: pvc storage-controller-0 pending due to ExternalProvisioning - waiting for a volume to be created, either by external provisioner "microk8s.io/hostpath" or manually created by system administrator}])
17:22:16 DEBUG juju.cmd.juju.commands bootstrap.go:1424 cleaning up after failed bootstrap
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/controller" sets to "cba3474d-d6b7-47b6-86a5-c76dd37ac70c"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/is-controller" sets to "true"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/controller" sets to "cba3474d-d6b7-47b6-86a5-c76dd37ac70c"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/is-controller" sets to "true"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/model" sets to "c3949c23-37fb-4342-871a-eb1b58ba5f25"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/controller" sets to "cba3474d-d6b7-47b6-86a5-c76dd37ac70c"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/is-controller" sets to "true"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/model" sets to "c3949c23-37fb-4342-871a-eb1b58ba5f25"
17:22:16 DEBUG juju.kubernetes.provider k8swatcher.go:112 fire notify watcher for controller-uk8s
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/controller" sets to "cba3474d-d6b7-47b6-86a5-c76dd37ac70c"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/is-controller" sets to "true"
17:22:16 DEBUG core.annotations annotations.go:112 annotation "juju.io/model" sets to "c3949c23-37fb-4342-871a-eb1b58ba5f25"
17:22:16 DEBUG juju.kubernetes.provider k8s.go:484 namespace "controller-uk8s" is still been terminating
17:22:22 DEBUG juju.kubernetes.provider k8swatcher.go:112 fire notify watcher for controller-uk8s
17:22:22 DEBUG core.annotations annotations.go:112 annotation "juju.io/controller" sets to "cba3474d-d6b7-47b6-86a5-c76dd37ac70c"
17:22:22 DEBUG core.annotations annotations.go:112 annotation "juju.io/is-controller" sets to "true"
17:22:22 DEBUG core.annotations annotations.go:112 annotation "juju.io/model" sets to "c3949c23-37fb-4342-871a-eb1b58ba5f25"
17:22:22 DEBUG juju.kubernetes.provider k8s.go:484 namespace "controller-uk8s" is still been terminating
17:22:27 DEBUG juju.kubernetes.provider k8swatcher.go:112 fire notify watcher for controller-uk8s
17:22:27 INFO cmd supercommand.go:525 command finished
Command '('microk8s-juju.wrapper', '--debug', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
rickyking
on 31 Mar 2020
@SpudAldridge, @rickyking: I believe #1099 will fix the issue you're running into. That's available on edge now (sudo snap switch microk8s --channel edge && sudo snap refresh), or will be available on stable in a few days via #1104.
knkski
on 9 Apr 2020
I am not able to enable kubeflow. I am runnin it on ubuntu 18.4.4 machine.
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Deploying Kubeflow...
Kubeflow could not be enabled:
ERROR a controller called "uk8s" already exists on this k8s cluster.
Please bootstrap again and choose a different controller name.
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
nevil99
on 11 Apr 2020
Same error as above after updating to edge channel:
Kubeflow could not be enabled:
ERROR controller "uk8s" already exists
Command '('microk8s-juju.wrapper', 'bootstrap', 'microk8s', 'uk8s')' returned non-zero exit status 1
Failed to enable kubeflow
tonihurtado
on 12 Apr 2020
@tonihurtado could you try to microk8s disable kubeflow and reenable it?
ktsakalozos
on 12 Apr 2020
@ktsakalozos after debugging microk8s enable kubeflow i realized that it was a problem with not having enough memory, so everything working fine with the version from the edge channel. Thanks.
tonihurtado
on 12 Apr 2020
@nevil99: You'll need to disable kubeflow before re-enabling it:
microk8s.disable kubeflow
microk8s.enable kubeflow
@tonihurtado: Sorry to hear you ran into that, but #1110 will add a check for minimum memory before enabling kubeflow, which will prevent you from running into that in the future.
knkski
on 13 Apr 2020
Hi. I'm very new to microk8s and kubernetes, but I'm afraid I'm getting a similar problem when trying to enable kubeflow, except with a slightly different error printout at the bottom. This happens when I try to enable:
$ microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Deploying Kubeflow...
Kubeflow could not be enabled:
Located bundle "cs:bundle/kubeflow-185"
Resolving charm: cs:~kubeflow-charmers/ambassador-78
Resolving charm: cs:~kubeflow-charmers/argo-controller-162
Resolving charm: cs:~kubeflow-charmers/argo-ui-78
Resolving charm: cs:~kubeflow-charmers/cert-manager-controller-9
Resolving charm: cs:~kubeflow-charmers/cert-manager-webhook-9
Resolving charm: cs:~kubeflow-charmers/dex-auth-20
Resolving charm: cs:~kubeflow-charmers/jupyter-controller-176
Resolving charm: cs:~kubeflow-charmers/jupyter-web-81
Resolving charm: cs:~kubeflow-charmers/katib-controller-76
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/katib-manager-75
Resolving charm: cs:~kubeflow-charmers/katib-ui-71
Resolving charm: cs:~kubeflow-charmers/kubeflow-dashboard-36
Resolving charm: cs:~kubeflow-charmers/kubeflow-profiles-41
Resolving charm: cs:~kubeflow-charmers/metacontroller-68
Resolving charm: cs:~kubeflow-charmers/metadata-api-32
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/metadata-envoy-14
Resolving charm: cs:~kubeflow-charmers/metadata-grpc-13
Resolving charm: cs:~kubeflow-charmers/metadata-ui-34
Resolving charm: cs:~kubeflow-charmers/minio-78
Resolving charm: cs:~kubeflow-charmers/modeldb-backend-76
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/modeldb-store-69
Resolving charm: cs:~kubeflow-charmers/modeldb-ui-68
Resolving charm: cs:~kubeflow-charmers/oidc-gatekeeper-19
Resolving charm: cs:~kubeflow-charmers/pipelines-api-82
Resolving charm: cs:~charmed-osm/mariadb-k8s
Resolving charm: cs:~kubeflow-charmers/pipelines-persistence-167
Resolving charm: cs:~kubeflow-charmers/pipelines-scheduledworkflow-163
Resolving charm: cs:~kubeflow-charmers/pipelines-ui-78
Resolving charm: cs:~kubeflow-charmers/pipelines-viewer-102
Resolving charm: cs:~kubeflow-charmers/pipelines-visualization-13
Resolving charm: cs:~kubeflow-charmers/pytorch-operator-163
Resolving charm: cs:~kubeflow-charmers/seldon-core-15
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-159
added resource oci-image
added resource oci-image
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/argo-ui-78": cannot read entity archive: read tcp 10.1.71.99:36898->162.213.33.78:443: read: connection reset by peer
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow', '--channel', 'stable', '--overlay', '/tmp/tmpw2vspkr3')' returned non-zero exit status 1
Failed to enable kubeflow
I've made several attempts, and the charm that it could not add is different every time. For example, one time I tried to enable, the charm was "cs:~kubeflow-charmers/argo-controller-162". Furthermore, the 'Deploying Kubeflow' part takes about 10 minutes.
Is there a verbose mode while running a microk8s enable action, so I can see what 'Deploying Kubeflow...' gets stuck on? Are there any commands in the enable.kubeflow action file I can run from the terminal to help diagnose this problem? Thanks for any help you can give, I'll also attach my Inspector file, which I created after this failure.
inspection-report-20200413_163326.tar.gz
EDIT:
Well it appears this was fixed by assuring the following namespaces were in resolv.conf (using the resolvconf service):
nameserver 8.8.8.8
nameserver 8.8.4.4
Is this requirement mentioned in any microk8s documentation that you're aware of? Apologies if it was and I couldn't find it.
zshenkle
on 13 Apr 2020
@zshenkle: For future reference, you can enable more verbose logging like this:
KUBEFLOW_DEBUG=true microk8s.enable kubeflow
As far as the nameserver issue you ran into, I'm not sure why you would have to edit those in. Did you put those in /etc/resolv.conf on the system that you installed microk8s onto? I haven't encountered that issue, but we've got some relevant documentation here that might need to call that out if it's broadly necessary:
https://microk8s.io/docs/addon-dns
cc @ktsakalozos ^^
knkski
on 14 Apr 2020
@zshenkle: For future reference, you can enable more verbose logging like this:
KUBEFLOW_DEBUG=true microk8s.enable kubeflowAs far as the nameserver issue you ran into, I'm not sure why you would have to edit those in. Did you put those in
/etc/resolv.confon the system that you installed microk8s onto? I haven't encountered that issue, but we've got some relevant documentation here that might need to call that out if it's broadly necessary:https://microk8s.io/docs/addon-dns
cc @ktsakalozos ^^
Well if behind proxy, what should I set for DNS SERVER?
rickyking
on 14 Apr 2020
@zshenkle: For future reference, you can enable more verbose logging like this:
KUBEFLOW_DEBUG=true microk8s.enable kubeflowAs far as the nameserver issue you ran into, I'm not sure why you would have to edit those in. Did you put those in
/etc/resolv.confon the system that you installed microk8s onto? I haven't encountered that issue, but we've got some relevant documentation here that might need to call that out if it's broadly necessary:https://microk8s.io/docs/addon-dns
cc @ktsakalozos ^^
@knkski thanks so much for that verbose logging tip! The machine I'm setting up microk8s on is a relatively new machine, and we apparently hadn't gotten around to adding all the necessary nameservers (IE: 8.8.8.8 and 8.8.4.4). And yes we effectively put those in /etc/resolv.conf via the systemd service resolvconf.
zshenkle
on 14 Apr 2020
I also met this issue like this
(base) root@iZwz91n3l4i42m2cawbmccZ:~# sudo microk8s.enable kubeflow
Enabling dns...
Enabling storage...
Enabling dashboard...
Enabling ingress...
Enabling metallb:10.64.140.43-10.64.140.49...
Waiting for DNS and storage plugins to finish setting up
Kubeflow could not be enabled:
timed out waiting for the condition on deployments/coredns
timed out waiting for the condition on deployments/hostpath-provisioner
Command '('microk8s-kubectl.wrapper', 'wait', '--for=condition=available', '-nkube-system', 'deployment/coredns', 'deployment/hostpath-provisioner', '--timeout=10m')' returned non-zero exit status 1
Failed to enable kubeflow
Here my snap list
(base) root@iZwz91n3l4i42m2cawbmccZ:~# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.44.3 9066 latest/stable canonical✓ core
core18 20200311 1705 latest/stable canonical✓ base
juju 2.7.6 11454 latest/stable canonical✓ classic
lxd 4.0.1 14804 latest/stable canonical✓ -
microk8s v1.18.2 1357 latest/candidate canonical✓ classic
And the microk8s.inspect
(base) root@iZwz91n3l4i42m2cawbmccZ:~# sudo microk8s.inspect
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-flanneld is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-apiserver-kicker is running
Service snap.microk8s.daemon-proxy is running
Service snap.microk8s.daemon-kubelet is running
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy current linux distribution to the final report tarball
Copy openSSL information to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Building the report tarball
Report tarball is at /var/snap/microk8s/1357/inspection-report-20200426_183743.tar.gz
Any help would be greatly appreciated.
kawa23
on 26 Apr 2020
@kawa23, can you attach the /var/snap/microk8s/1357/inspection-report-20200426_183743.tar.gz file generated by microk8s.inspect?
knkski
on 4 May 2020
@kawa23 I have the exact same problem.
Could it be the case that my Kubernetes version is not supported yet? It is 1.18, while Kubeflow is on alpha status with Microk8s, and 1.18 kubernetes version is not even mentioned on https://www.kubeflow.org/docs/started/k8s/overview/#minimum-system-requirements.
MathewStylianidis
on 11 May 2020
@MathewStylianidis: It shouldn't be an issue with 1.18. Would you be able to run microk8s.inspect and attach the tarball that it generates here?
knkski
on 11 May 2020
MathewStylianidis
on 12 May 2020
@knkski @markshuttle Sorry for late reply, I have installed kubeflow based on k8s cluster with multipass, and removed the microk8s, so I can't get /var/snap/microk8s/1357/inspection-report-20200426_183743.tar.gz.
kawa23
on 5 Jun 2020
I tried to install kubeflow and got the same error:
snap install microk8s --classic
microk8s.status --wait-ready
microk8s.enable dns dashboard storage
microk8s.enable gpu
Everything works up to here, but
microk8s.enable kubeflow
fails with this error:
Resolving charm: cs:~kubeflow-charmers/pytorch-operator-163
Resolving charm: cs:~kubeflow-charmers/seldon-core-15
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-159
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-78": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-78": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-78/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow', '--channel', 'stable', '--overlay', '/tmp/tmpeuizwk4k')' returned non-zero exit status 1
Failed to enable kubeflow
Any idea what is causing it?
atamahjoubfar
on 10 Jun 2020
Looks like a networking issue - it needs to be able to retrieve the charms from jujucharms.com. I think we should do some connectivity tests before we launch into the orchestration of kubeflow, so we catch and warn more appropriately if that is going to fail as it has here.
markshuttle
on 10 Jun 2020
Looks like a networking issue - it needs to be able to retrieve the charms from jujucharms.com. I think we should do some connectivity tests before we launch into the orchestration of kubeflow, so we catch and warn more appropriately if that is going to fail as it has here.
But I can ping jujucharms.com from the same server.
atamahjoubfar
on 10 Jun 2020
@atamahjoubfar could you share the output of:
KUBEFLOW_DEBUG=true microk8s.enable kubeflow
@atamahjoubfar could you share the output of:
KUBEFLOW_DEBUG=true microk8s.enable kubeflowIt fails at
07:54:17 DEBUG httpbakery client.go:245 } -> error <nil>
07:54:17 INFO cmd bundle.go:364 Resolving charm: cs:~kubeflow-charmers/tf-job-operator-159
07:54:17 DEBUG httpbakery client.go:243 client do GET https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/tf-job-operator-159/meta/any?include=id&include=supported-series&include=published {
07:54:17 DEBUG httpbakery client.go:245 } -> error <nil>
07:54:17 DEBUG httpbakery client.go:243 client do GET https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-78/meta/any?include=id&include=supported-series&include=published {
07:54:17 DEBUG httpbakery client.go:245 } -> error <nil>
07:54:29 DEBUG juju.api monitor.go:35 RPC connection died
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-78": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-78": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-78/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
07:54:29 DEBUG cmd supercommand.go:519 error stack:
cannot retrieve charm "cs:~kubeflow-charmers/ambassador-78": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-78/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
/workspace/_build/src/github.com/juju/juju/rpc/client.go:178:
/workspace/_build/src/github.com/juju/juju/api/apiclient.go:1187:
/workspace/_build/src/github.com/juju/juju/api/client.go:459:
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/store.go:68:
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/bundle.go:543: cannot add charm "cs:~kubeflow-charmers/ambassador-78"
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/bundle.go:475:
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/bundle.go:164:
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/deploy.go:969: cannot deploy bundle
/workspace/_build/src/github.com/juju/juju/cmd/juju/application/deploy.go:1540:
Command '('microk8s-juju.wrapper', '--debug', 'deploy', 'cs:kubeflow', '--channel', 'stable', '--overlay', '/tmp/tmp6uznal1p')' returned non-zero exit status 1
Failed to enable kubeflow
@atamahjoubfar: That looks like transient network issues. Does it work if you try it again?
knkski
on 10 Jun 2020
@atamahjoubfar: That looks like transient network issues. Does it work if you try it again?
I tried again, and unfortunately, the problem persists. Is there any way to manually download the charms and enable kubeflow from disk?
atamahjoubfar
on 11 Jun 2020
@atamahjoubfar, @markshuttle: I've opened up #1296 to do a basic network connectivity check before enabling Kubeflow
@atamahjoubfar: If you follow the installation instructions in the README here for deploying the Kubeflow bundle instead of via microk8s.enable kubeflow, does it work for you?
knkski
on 11 Jun 2020
@kawa23, @MathewStylianidis: I believe the issue that you're facing is due to your hostname having capital letters, see this error message:
mai 11 15:06:58 SESTVSDV0002 microk8s.daemon-kubelet[24507]: E0511 15:06:58.330800 24507 csi_plugin.go:271] Failed to initialize CSINodeInfo: error updating CSINode annotation: timed out waiting for the condition; caused by: csinodes.storage.k8s.io "sestvsdv0002" is forbidden: User "system:node:SESTVSDV0002" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope: can only access CSINode with the same name as the requesting node
If you change your hostname to only have lowercase letters, does it work for you?
knkski
on 11 Jun 2020
@knkski @markshuttle: I tried to follow these instructions:
https://github.com/juju-solutions/bundle-kubeflow/#setup-microk8s
But got the same error:
+ juju deploy -m kubeflow kubeflow --channel stable --overlay=/tmp/tmp1rzpf5m9
Located bundle "cs:bundle/kubeflow-185"
Resolving charm: cs:~kubeflow-charmers/ambassador-78
Resolving charm: cs:~kubeflow-charmers/argo-controller-162
...
Resolving charm: cs:~kubeflow-charmers/seldon-core-15
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-159
Executing changes:
- upload charm cs:~kubeflow-charmers/ambassador-78 for series kubernetes
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-78": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-78": cannot get archive: Get "https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-78/archive?channel=stable": dial tcp: i/o timeout
Command '('juju', 'deploy', '-m', 'kubeflow', 'kubeflow', '--channel', 'stable', '--overlay=/tmp/tmp1rzpf5m9')' returned non-zero exit status 1.
I installed the microk8s from the edge channel, and now I can see the error message that you added in #1296.
Couldn't contact api.jujucharms.com
Please check your network connectivity before enabling Kubeflow.
Failed to enable kubeflow
However, from the same server, the following wget works:
wget https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-88/icon.svg
I can ping jujucharms.com, but not api.jujucharms.com. Any ideas?
atamahjoubfar
on 16 Jun 2020
@atamahjoubfar: Can you paste the output from this command?
microk8s.kubectl get configmap -n kube-system coredns -oyaml
@atamahjoubfar: Can you paste the output from this command?
microk8s.kubectl get configmap -n kube-system coredns -oyaml
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
log . {
class error
}
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
prometheus :9153
forward . 8.8.8.8 8.8.4.4
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"Corefile":".:53 {\n errors\n health {\n lameduck 5s\n }\n ready\n log . {\n class error\n }\n kubernetes cluster.local in-addr.arpa ip6.arpa {\n pods insecure\n fallthrough in-addr.arpa ip6.arpa\n }\n prometheus :9153\n forward . 8.8.8.8 8.8.4.4\n cache 30\n loop\n reload\n loadbalance\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"EnsureExists","k8s-app":"kube-dns"},"name":"coredns","namespace":"kube-system"}}
creationTimestamp: "2020-06-23T22:20:52Z"
labels:
addonmanager.kubernetes.io/mode: EnsureExists
k8s-app: kube-dns
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:Corefile: {}
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:labels:
.: {}
f:addonmanager.kubernetes.io/mode: {}
f:k8s-app: {}
manager: kubectl
operation: Update
time: "2020-06-23T22:20:52Z"
name: coredns
namespace: kube-system
resourceVersion: "1124"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 588deb7d-bf61-4096-a553-da7696eea724
Does this help to identify the issue? Thank you.
atamahjoubfar
on 24 Jun 2020
Got same error on Ubuntu 20.04 Desktop. I am not behind any firewall/proxy and can manually fetch https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-104/archive?channel=stable which downloads ambassador.zip (2.6 MB). I do not face this error on Unbutu 20.04 Server (in AWS)
Tried with both
microk8s v1.18.4
microk8s v1.18.6 1560 latest/edge
microk8s.kubectl get configmap -n kube-system coredns -oyaml gives almost the same output as above except for difference in creationTimestamp, time, uid, resourceVersion: "390"
Output of microk8s.enable kubeflow
...
Deploying Kubeflow...
Kubeflow could not be enabled:
Located bundle "cs:bundle/kubeflow-206"
Resolving charm: cs:~kubeflow-charmers/ambassador-104
Resolving charm: cs:~kubeflow-charmers/argo-controller-190
...
Resolving charm: cs:~kubeflow-charmers/seldon-core-44
Resolving charm: cs:~kubeflow-charmers/tf-job-operator-188
ERROR cannot deploy bundle: cannot add charm "cs:~kubeflow-charmers/ambassador-104": cannot retrieve charm "cs:~kubeflow-charmers/ambassador-104": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/~kubeflow-charmers/ambassador-104/archive?channel=stable: dial tcp: lookup api.jujucharms.com on 10.152.183.10:53: server misbehaving
Command '('microk8s-juju.wrapper', 'deploy', 'cs:kubeflow-206', '--channel', 'stable', '--overlay', '/tmp/tmpf6ehtx10')' returned non-zero exit status 1
Failed to enable kubeflow
Errors from coredns pod
.:53
[INFO] plugin/reload: Running configuration MD5 = be0f52d3c13480652e0c73672f2fa263
CoreDNS-1.6.6
linux/amd64, go1.13.5, 6a7a75e
[INFO] 127.0.0.1:59495 - 39173 "HINFO IN 2577415620770853004.8780143945028532492. udp 57 false 512" NOERROR - 0 6.011622149s
[ERROR] plugin/errors: 2 2577415620770853004.8780143945028532492. HINFO: read udp 10.1.21.36:38427->8.8.8.8:53: i/o timeout
[INFO] 127.0.0.1:37728 - 42831 "HINFO IN 2577415620770853004.8780143945028532492. udp 57 false 512" NOERROR - 0 2.000464057s
...
[ERROR] plugin/errors: 2 2577415620770853004.8780143945028532492. HINFO: read udp 10.1.21.36:44631->8.8.4.4:53: i/o timeout
[INFO] 10.1.21.38:33385 - 46047 "AAAA IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000755655s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. AAAA: read udp 10.1.21.36:57766->8.8.8.8:53: i/o timeout
[INFO] 10.1.21.38:36178 - 22767 "A IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000677982s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. A: read udp 10.1.21.36:44211->8.8.8.8:53: i/o timeout
[INFO] 10.1.21.38:60455 - 56232 "AAAA IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000352953s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. AAAA: read udp 10.1.21.36:45297->8.8.8.8:53: i/o timeout
[INFO] 10.1.21.38:50299 - 52151 "A IN api.jujucharms.com.localdomain. udp 48 false 512" NOERROR - 0 2.000257257s
[ERROR] plugin/errors: 2 api.jujucharms.com.localdomain. A: read udp 10.1.21.36:40710->8.8.4.4:53: i/o timeout
[INFO] 10.1.21.38:39253 - 3642 "AAAA IN api.jujucharms.com. udp 36 false 512" NOERROR - 0 2.000834815s
[ERROR] plugin/errors: 2 api.jujucharms.com. AAAA: read udp 10.1.21.36:38538->8.8.8.8:53: i/o timeout
[INFO] 10.1.21.38:47372 - 10457 "A IN api.jujucharms.com. udp 36 false 512" NOERROR - 0 2.000768237s
[ERROR] plugin/errors: 2 api.jujucharms.com. A: read udp 10.1.21.36:33745->8.8.4.4:53: i/o timeout
[INFO] 10.1.21.38:47852 - 7227 "AAAA IN api.jujucharms.com. udp 36 false 512" NOERROR - 0 2.000442768s
[ERROR] plugin/errors: 2 api.jujucharms.com. AAAA: read udp 10.1.21.36:42672->8.8.4.4:53: i/o timeout
[INFO] 10.1.21.38:60290 - 23521 "A IN api.jujucharms.com. udp 36 false 512" NOERROR - 0 2.000334072s
[ERROR] plugin/errors: 2 api.jujucharms.com. A: read udp 10.1.21.36:52312->8.8.8.8:53: i/o timeout
bipinm
on 21 Jul 2020
@bipinm: I'm going to open up a new issue with that output, that looks like the crux of the issue, and unrelated to the original issue for this thread:
knkski
on 22 Jul 2020
@knkski Thanks. Was actually referring to post from @atamahjoubfar on Jun 10th. i got the same error.
Will continue to check further when i get time and post any updates in the new issue you have created
bipinm
on 23 Jul 2020
I think all of the issues in this thread have been resolved. If not, feel free to open up a new issue
knkski
on 27 Jul 2020
Related issues
xrpn
·
4Comments
NumesSanguis
·
3Comments
arno01
·
4Comments
ceefour
·
3Comments
carmine
·
4Comments
Most helpful comment
You could
snap install jujuand thencp /snap/juju/current/bin/juju /var/snap/microk8s/current/bin/. Note this is a hack.