Required information
- Distribution: Ubuntu
- Distribution version: Xenial
- The output of "lxc info" or if that fails:
config:
core.https_address: '[::]:4443'
core.trust_password: true
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
addresses:
- 176.xxx.xxx.xxx:4443
- 10.159.23.1:4443
architectures:
- x86_64
- i686
certificate: |
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
certificate_fingerprint: driver: lxc
driver_version: 3.1.0
kernel: Linux
kernel_architecture: x86_64
kernel_version: 4.4.0-141-generic
server: lxd
server_pid: 32108
server_version: "3.11"
storage: btrfs
storage_version: "4.4"
server_clustered: false
server_name: ubuntu-xenial
project: default
Issue description
I´m trying to spin up mikrok8s in a bionic lxd container. The snap installation succeeds but kubelet service is not starting:
microk8s.inspect
Inspecting services
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-apiserver is running
Service snap.microk8s.daemon-proxy is running
FAIL: Service snap.microk8s.daemon-kubelet is not running
For more details look at: sudo journalctl -u snap.microk8s.daemon-kubelet
Service snap.microk8s.daemon-scheduler is running
Service snap.microk8s.daemon-controller-manager is running
Service snap.microk8s.daemon-etcd is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system info
Copy network configuration to the final report tarball
Copy processes list to the final report tarball
Copy snap list to the final report tarball
Inspect kubernetes cluster
microk8s.kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/heapster-v1.5.2-5c5498f57c-zgcxr 0/4 Pending 0 40m
kube-system pod/kube-dns-6bfbdd666c-sgvl6 0/3 Pending 0 40m
kube-system pod/kubernetes-dashboard-6fd7f9c494-l5vsw 0/1 Pending 0 40m
kube-system pod/monitoring-influxdb-grafana-v4-78777c64c8-wwtgp 0/2 Pending 0 40m
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default service/kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 54m
kube-system service/heapster ClusterIP 10.152.183.61 <none> 80/TCP 40m
kube-system service/kube-dns ClusterIP 10.152.183.10 <none> 53/UDP,53/TCP 40m
kube-system service/kubernetes-dashboard ClusterIP 10.152.183.183 <none> 443/TCP 40m
kube-system service/monitoring-grafana ClusterIP 10.152.183.175 <none> 80/TCP 40m
kube-system service/monitoring-influxdb ClusterIP 10.152.183.114 <none> 8083/TCP,8086/TCP 40m
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
kube-system deployment.apps/heapster-v1.5.2 0/1 1 0 40m
kube-system deployment.apps/kube-dns 0/1 1 0 40m
kube-system deployment.apps/kubernetes-dashboard 0/1 1 0 40m
kube-system deployment.apps/monitoring-influxdb-grafana-v4 0/1 1 0 40m
NAMESPACE NAME DESIRED CURRENT READY AGE
kube-system replicaset.apps/heapster-v1.5.2-5c5498f57c 1 1 0 40m
kube-system replicaset.apps/kube-dns-6bfbdd666c 1 1 0 40m
kube-system replicaset.apps/kubernetes-dashboard-6fd7f9c494 1 1 0 40m
kube-system replicaset.apps/monitoring-influxdb-grafana-v4-78777c64c8 1 1 0 40m
microk8s.kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:16443
Heapster is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Grafana is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
InfluxDB is running at https://127.0.0.1:16443/api/v1/namespaces/kube-system/services/monitoring-influxdb:http/proxy
md5-71041ead6496480c8c83da403113f03e
root@microk8s:~# microk8s.kubectl get nodes
NAME STATUS ROLES AGE VERSION
microk8s NotReady <none> 56m v1.14.0
md5-153e92ef5b8978f68be30c3c3e3b2359
364 fs.go:556] stat failed on /dev/dm-0 with error: no such file or directory
Apr 8 11:31:31 microk8s microk8s.daemon-kubelet[1364]: F0408 11:31:31.078576 1364 kubelet.go:1359] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/snap/microk8s/common/var/lib/kubelet": could not find device with major: 0, minor: 272 in cached partitions map
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Main process exited, code=exited, status=255/n/a
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Failed with result 'exit-code'.
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Service hold-off time over, scheduling restart.
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Scheduled restart job, restart counter is at 6.
Apr 8 11:31:31 microk8s systemd[1]: Stopped Service for snap application microk8s.daemon-kubelet.
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Start request repeated too quickly.
Apr 8 11:31:31 microk8s systemd[1]: snap.microk8s.daemon-kubelet.service: Failed with result 'exit-code'.
Apr 8 11:31:31 microk8s systemd[1]: Failed to start Service for snap application microk8s.daemon-kubelet.
md5-d80cf9b126aaee64e26989a0a85a0df7
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20190402)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: "20190402"
image.version: "18.04"
linux.kernel_modules: ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
raw.lxc: |
lxc.apparmor.profile = unconfined
lxc.cgroup.devices.allow = a
lxc.mount.auto=proc:rw sys:rw
lxc.cap.drop =
security.nesting: "true"
security.privileged: "true"
volatile.base_image: 663f6663aed66a22dd708c4b07514748221522b810008c55002fcc1dd81af377
volatile.eth0.hwaddr: 00:16:3e:d2:3c:28
volatile.idmap.base: "0"
volatile.idmap.next: '[]'
volatile.last_state.idmap: '[]'
volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
md5-71041ead6496480c8c83da403113f03e
root@microk8s:~# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.38 6673 stable canonical✓ core
microk8s v1.14.0 492 stable canonical✓ classic
I also tried the edge branch but with the same result.
I´m not sure if this is a issue related to LXD or microk8s.
Thanks.
chuegel
All 6 comments
Yep, that's a known issue with microk8s and something the Kubernetes team at Canonical is working on.
The intent is that you should only need to set:
- security.privileged=true
- security.nesting=true
- linux.kernel_modules=ip_tables,ip6_tables,netlink_diag,nf_nat,overlay
And then everything will work fine. Right now kubelet is trying to be smart about storage and failing.
@ktsakalozos may be able to provide more details.
There's nothing for us to do about this on the LXD side, so closing the issue here.
stgraber
on 8 Apr 2019
👍1
Hi @huegelc,
As @stgraber mentioned the intent is to have MicroK8s working on LXC with a minimal profile. We have a PR upstream that works towards this direction but it has not landed yet. For now, to have MicroK8s on LXC we use the two profiles you can find here: https://github.com/ubuntu/microk8s/tree/master/tests/lxc
ktsakalozos
on 8 Apr 2019
Thank you, much appreciated.
chuegel
on 8 Apr 2019
Hi @huegelc,
As @stgraber mentioned the intent is to have MicroK8s working on LXC with a minimal profile. We have a PR upstream that works towards this direction but it has not landed yet. For now, to have MicroK8s on LXC we use the two profiles you can find here: https://github.com/ubuntu/microk8s/tree/master/tests/lxc
@ktsakalozos What is the link to that upstream issue? I cannot tell if this is supposed to work yet or not. It would be useful to track that issue.
drewboswell
on 23 Apr 2020
Sorry for the late reply @drewboswell. I totally missed this.
The PR I was referring to is [1]. In the meantime we have revised the MicroK8s docs with instructions on how to install it on lxc [2].
[1] https://github.com/google/cadvisor/pull/2189
[2] https://microk8s.io/docs/lxd
ktsakalozos
on 23 Jun 2020
❤2
Thanks @ktsakalozos super helpful!
drewboswell
on 23 Jun 2020
Was this page helpful?
0 / 5 - 0 ratings
Related issues
simos
·
3Comments
shaun-ba
·
3Comments
rrva
·
5Comments
killua-eu
·
3Comments
AndreiPashkin
·
5Comments
Most helpful comment
Sorry for the late reply @drewboswell. I totally missed this.
The PR I was referring to is [1]. In the meantime we have revised the MicroK8s docs with instructions on how to install it on lxc [2].
[1] https://github.com/google/cadvisor/pull/2189
[2] https://microk8s.io/docs/lxd