Conjure-up: Possible calltrace running conjure-up canonical-kubernetes localhost

[naisanza]

Report

Thank you for trying conjure-up! Before reporting a bug please make sure you've gone through this checklist:

• [x] Is this problem already documented at https://docs.ubuntu.com/conjure-up/en/troubleshoot#common-spell-problems
• [x] Is conjure-up running inside a virtual machine?
• [x] Is this problem reproducible with sudo snap refresh conjure-up --edge?

Please provide the output of the following commands

admin@k8smaster-conjure:~$ which juju
/snap/bin/juju

admin@k8smaster-conjure:~$ juju version
2.2.3-xenial-amd64

admin@k8smaster-conjure:~$ which conjure-up
/snap/bin/conjure-up

admin@k8smaster-conjure:~$ conjure-up --version
conjure-up 2.3.1

admin@k8smaster-conjure:~$ which lxc
/snap/bin/lxc

admin@k8smaster-conjure:~$ lxc config show
config: {}

admin@k8smaster-conjure:~$ lxc version
2.18

admin@k8smaster-conjure:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.3 LTS"

Please attach tarball of ~/.cache/conjure-up:

conjure-up.tar.gz

Sosreport

Please attach a sosreport:
sosreport-k8smaster-conjure.1191-20171017181645.tar.xz.zip

The resulting output file can be attached to this issue.
conjure-up.tar.gz

What Spell was Selected?

canonical-kubernetes

What provider (aws, maas, localhost, etc)?

localhost

MAAS Users

Which version of MAAS?
n/a

Commands ran

Please outline what commands were run to install and execute conjure-up:

I used this ansible playbook https://github.com/TheShellLand/antsable/blob/master/playbooks/k8s_master_conjure-up.yml

Additional Information

admin@k8smaster-conjure:~$ conjure-up canonical-kubernetes localhost
[info] Summoning canonical-kubernetes to localhost
[info] Bootstrapping Juju controller.
[info] Bootstrap complete.
[info] Running step: pre-deploy.
[info] Deploying etcd...
[info] Deploying kubeapi-load-balancer...
[info] Deploying kubernetes-master...
[info] Deploying flannel...
[info] Deploying kubernetes-worker...
[info] Deploying easyrsa...
[error] cannot add charm to storage: cannot clean up after failed storage operation because: read tcp 127.0.0.1:55984->127.0.0.1:37017: i/o timeout: cannot add resource "buckets/fb9cea43-2c68-4cb0-8ed0-3d0231cba98d/charms/cs:~containers/kubeapi-load-balancer-31-d8026dd9-cfc2-4815-8b64-702fc2472afc" to store at storage path "4c4bc00e-d984-405c-8aab-20b738808566": failed to write data: read tcp 127.0.0.1:55984->127.0.0.1:37017: i/o timeout
[warning] Shutting down

[6840596.891665] resolvconf: page allocation stalls for 35228ms, order:0, mode:0x14280ca(GFP_HIGHUSER_MO
VABLE|__GFP_ZERO)
[6840596.891675] CPU: 7 PID: 9313 Comm: resolvconf Tainted: P        W  O    4.10.15-1-pve #1
[6840596.891676] Hardware name: Supermicro X7DCL/X7DCL, BIOS 1.0b 05/12/2008
[6840596.891677] Call Trace:
[6840596.891688]  dump_stack+0x63/0x81
[6840596.891692]  warn_alloc+0x13a/0x170
[6840596.891694]  __alloc_pages_slowpath+0x927/0xba0
[6840596.891697]  ? find_next_bit+0x18/0x20
[6840596.891701]  ? mem_cgroup_commit_charge+0x7e/0x510
[6840596.891702]  __alloc_pages_nodemask+0x209/0x260
[6840596.891705]  alloc_pages_vma+0xab/0x250
[6840596.891709]  handle_mm_fault+0xe30/0x1330
[6840596.891714]  __do_page_fault+0x23e/0x4e0
[6840596.891715]  do_page_fault+0x22/0x30
[6840596.891719]  page_fault+0x28/0x30
[6840596.891721] RIP: 0033:0x7feed5a4c509
[6840596.891722] RSP: 002b:00007ffdc153b510 EFLAGS: 00010246
[6840596.891724] RAX: 000000009d122d10 RBX: 0000000000000000 RCX: 00007feed5a4c41a
[6840596.891725] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[6840596.891726] RBP: 00007ffdc153b540 R08: 0000000000000000 R09: 00007feed5f62700
[6840596.891727] R10: 00007feed5f629d0 R11: 0000000000000246 R12: 00000000000051cb
[6840596.891728] R13: 0000000000000000 R14: 0000000000000000 R15: 00007feed5f70040
[6840596.891729] Mem-Info:
[6840596.891734] active_anon:3374256 inactive_anon:784014 isolated_anon:343
                  active_file:1068065 inactive_file:497923 isolated_file:0
                  unevictable:4975 dirty:90935 writeback:34 unstable:0
                  slab_reclaimable:331508 slab_unreclaimable:1926596
                  mapped:240372 shmem:198856 pagetables:47496 bounce:0
                  free:48960 free_pcp:0 free_cma:0
[6840596.891739] Node 0 active_anon:13497024kB inactive_anon:3136056kB active_file:4272260kB inactive_fi
le:1991692kB unevictable:19900kB isolated(anon):1372kB isolated(file):0kB mapped:961488kB dirty:363740kB
 writeback:136kB shmem:795424kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 34816kB writeback_tmp:0kB 
unstable:0kB pages_scanned:0 all_unreclaimable? no
[6840596.891740] Node 0 DMA free:15892kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB a
ctive_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocke
d:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0k
B local_pcp:0kB free_cma:0kB
[6840596.891744] lowmem_reserve[]: 0 3474 32124 32124 32124
[6840596.891747] Node 0 DMA32 free:121900kB min:7304kB low:10860kB high:14416kB active_anon:696956kB ina
ctive_anon:149260kB active_file:399652kB inactive_file:301808kB unevictable:0kB writepending:84576kB pre
sent:3652928kB managed:3587360kB mlocked:0kB slab_reclaimable:252108kB slab_unreclaimable:1608720kB kern
el_stack:1072kB pagetables:7052kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[6840596.891751] lowmem_reserve[]: 0 0 28649 28649 28649
[6840596.891754] Node 0 Normal free:58048kB min:60244kB low:89580kB high:118916kB active_anon:12800068kB
 inactive_anon:2986796kB active_file:3872608kB inactive_file:1689884kB unevictable:19900kB writepending:
279300kB present:29884416kB managed:29341076kB mlocked:19904kB slab_reclaimable:1073924kB slab_unreclaim
able:6097664kB kernel_stack:54032kB pagetables:182932kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0
kB
[6840596.891758] lowmem_reserve[]: 0 0 0 0 0
[6840596.891761] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512k
B 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[6840596.891772] Node 0 DMA32: 9031*4kB (UMEH) 9664*8kB (UMEH) 456*16kB (UMH) 36*32kB (UM) 0*64kB 0*128k
B 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 121884kB
[6840596.891781] Node 0 Normal: 9077*4kB (UMEH) 484*8kB (UMEH) 936*16kB (UMH) 86*32kB (UMH) 1*64kB (U) 0
*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 57972kB
[6840596.891791] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[6840596.891792] 1804777 total pagecache pages
[6840596.891794] 38229 pages in swap cache
[6840596.891795] Swap cache stats: add 19906820, delete 19868520, find 959469929/965218433
[6840596.891796] Free swap  = 7998028kB
[6840596.891796] Total swap = 15335416kB
[6840596.891797] 8388330 pages RAM
[6840596.891797] 0 pages HighMem/MovableOnly
[6840596.891798] 152248 pages reserved
[6840596.891799] 0 pages cma reserved
[6840596.891799] 0 pages hwpoisoned
[6840596.906400] kvm: page allocation stalls for 28676ms, order:0, mode:0x1c200c0(GFP_USER|__GFP_WRITE)
[6840596.906412] CPU: 4 PID: 9329 Comm: kvm Tainted: P        W  O    4.10.15-1-pve #1
[6840596.906413] Hardware name: Supermicro X7DCL/X7DCL, BIOS 1.0b 05/12/2008
[6840596.906414] Call Trace:
[6840596.906425]  dump_stack+0x63/0x81
[6840596.906428]  warn_alloc+0x13a/0x170
[6840596.906430]  __alloc_pages_slowpath+0x927/0xba0
[6840596.906435]  ? _cond_resched+0x2b/0x40
[6840596.906436]  __alloc_pages_nodemask+0x209/0x260
[6840596.906440]  alloc_pages_current+0x95/0x140
[6840596.906442]  __page_cache_alloc+0xc4/0xe0
[6840596.906444]  pagecache_get_page+0xab/0x2a0
[6840596.906448]  ? I_BDEV+0x20/0x20
[6840596.906449]  grab_cache_page_write_begin+0x20/0x40
[6840596.906451]  block_write_begin+0x2d/0xc0
[6840596.906452]  blkdev_write_begin+0x23/0x30
[6840596.906454]  generic_perform_write+0xc8/0x1c0
[6840596.906456]  __generic_file_write_iter+0x185/0x1d0
[6840596.906458]  blkdev_write_iter+0x8b/0x100
[6840596.906461]  ? rw_copy_check_uvector+0x69/0x130
[6840596.906462]  do_iter_readv_writev+0xb6/0x140
[6840596.906464]  do_readv_writev+0x19e/0x240
[6840596.906466]  vfs_writev+0x3c/0x50
[6840596.906467]  do_pwritev+0xb2/0xd0
[6840596.906469]  SyS_pwritev+0x11/0x20
[6840596.906471]  entry_SYSCALL_64_fastpath+0x1e/0xad
[6840596.906473] RIP: 0033:0x7f38cd1b6f83
[6840596.906474] RSP: 002b:00007f37e40645a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000128
[6840596.906476] RAX: ffffffffffffffda RBX: 00007f3817827340 RCX: 00007f38cd1b6f83
[6840596.906477] RDX: 0000000000000077 RSI: 00007f38c1849000 RDI: 0000000000000015
[6840596.906478] RBP: 00007f3817827340 R08: 0000000000000000 R09: 0000000000000000
[6840596.906479] R10: 000000000b1a8000 R11: 0000000000000293 R12: 00007f38c1866b00
[6840596.906480] R13: 00007f38c18bcce8 R14: 00007f37e4067700 R15: 00007f380340e400
[6840596.906481] Mem-Info:
[6840596.906486] active_anon:3374256 inactive_anon:784014 isolated_anon:343
                  active_file:1068065 inactive_file:497923 isolated_file:0
                  unevictable:4975 dirty:90935 writeback:34 unstable:0
                  slab_reclaimable:331508 slab_unreclaimable:1926596
                  mapped:240372 shmem:198856 pagetables:47496 bounce:0
                  free:48960 free_pcp:0 free_cma:0
[6840596.906490] Node 0 active_anon:13497024kB inactive_anon:3136056kB active_file:4272260kB inactive_fi
le:1991692kB unevictable:19900kB isolated(anon):1372kB isolated(file):0kB mapped:961488kB dirty:363740kB
 writeback:136kB shmem:795424kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 34816kB writeback_tmp:0kB 
unstable:0kB pages_scanned:0 all_unreclaimable? no
[6840596.906491] Node 0 DMA free:15892kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB a
ctive_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocke
d:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0k
B local_pcp:0kB free_cma:0kB
[6840596.906495] lowmem_reserve[]: 0 3474 32124 32124 32124
[6840596.906498] Node 0 DMA32 free:121900kB min:7304kB low:10860kB high:14416kB active_anon:696956kB ina
ctive_anon:149260kB active_file:399652kB inactive_file:301808kB unevictable:0kB writepending:84576kB pre
sent:3652928kB managed:3587360kB mlocked:0kB slab_reclaimable:252108kB slab_unreclaimable:1608720kB kern
el_stack:1072kB pagetables:7052kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[6840596.906503] lowmem_reserve[]: 0 0 28649 28649 28649
[6840596.906506] Node 0 Normal free:58048kB min:60244kB low:89580kB high:118916kB active_anon:12800068kB
 inactive_anon:2986796kB active_file:3872608kB inactive_file:1689884kB unevictable:19900kB writepending:
279300kB present:29884416kB managed:29341076kB mlocked:19904kB slab_reclaimable:1073924kB slab_unreclaim
able:6097664kB kernel_stack:54032kB pagetables:182932kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0
kB
[6840596.906510] lowmem_reserve[]: 0 0 0 0 0
[6840596.906513] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512k
B 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[6840596.906524] Node 0 DMA32: 9032*4kB (UMEH) 9664*8kB (UMEH) 456*16kB (UMH) 36*32kB (UM) 0*64kB 0*128k
B 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 121888kB
[6840596.906533] Node 0 Normal: 9087*4kB (UMEH) 484*8kB (UMEH) 936*16kB (UMH) 86*32kB (UMH) 1*64kB (U) 0
*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 58012kB
[6840596.906543] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[6840596.906544] 1804777 total pagecache pages
[6840596.906546] 38229 pages in swap cache
[6840596.906547] Swap cache stats: add 19906826, delete 19868526, find 959469929/965218437
[6840596.906547] Free swap  = 7998004kB
[6840596.906548] Total swap = 15335416kB
[6840596.906549] 8388330 pages RAM
[6840596.906550] 0 pages HighMem/MovableOnly
[6840596.906550] 152248 pages reserved
[6840596.906551] 0 pages cma reserved
[6840596.906551] 0 pages hwpoisoned
[6840596.994463] x2gosqlitewrapp: page allocation stalls for 12300ms, order:0, mode:0x14280ca(GFP_HIGHUS
ER_MOVABLE|__GFP_ZERO)
[6840596.994477] CPU: 2 PID: 9345 Comm: x2gosqlitewrapp Tainted: P        W  O    4.10.15-1-pve #1
[6840596.994478] Hardware name: Supermicro X7DCL/X7DCL, BIOS 1.0b 05/12/2008
[6840596.994479] Call Trace:
[6840596.994490]  dump_stack+0x63/0x81
[6840596.994494]  warn_alloc+0x13a/0x170
[6840596.994496]  __alloc_pages_slowpath+0x927/0xba0
[6840596.994598]  ? rrm_exit+0x46/0x80 [zfs]
[6840596.994601]  ? page_counter_uncharge+0x22/0x40
[6840596.994602]  __alloc_pages_nodemask+0x209/0x260
[6840596.994605]  alloc_pages_vma+0xab/0x250
[6840596.994609]  handle_mm_fault+0xe30/0x1330
[6840596.994613]  __do_page_fault+0x23e/0x4e0
[6840596.994615]  do_page_fault+0x22/0x30
[6840596.994618]  page_fault+0x28/0x30
[6840596.994620] RIP: 0033:0x7f01434af786
[6840596.994622] RSP: 002b:00007ffc1f4d8720 EFLAGS: 00010206
[6840596.994623] RAX: 0000000000007801 RBX: 00007f01437f1b20 RCX: 0000000000000811
[6840596.994624] RDX: 00000000013b9ff0 RSI: 00000000013ba800 RDI: 0000000000000002
[6840596.994625] RBP: 0000000000000811 R08: 00007f01437f1b98 R09: 0000000000000000
[6840596.994626] R10: 00007f01437f1b78 R11: 00007f01437f1b78 R12: 00007f01437f1b78
[6840596.994627] R13: 00007f01437f1b78 R14: 000000000000270f R15: 00007f01437f1b78
[6840596.994628] Mem-Info:
[6840596.994634] active_anon:3374256 inactive_anon:784014 isolated_anon:343
                  active_file:1068065 inactive_file:497923 isolated_file:0
                  unevictable:4975 dirty:90935 writeback:34 unstable:0
                  slab_reclaimable:331508 slab_unreclaimable:1926596
                  mapped:240372 shmem:198856 pagetables:47496 bounce:0
                  free:48600 free_pcp:0 free_cma:0
[6840596.994639] Node 0 active_anon:13497024kB inactive_anon:3136056kB active_file:4272260kB inactive_fi
le:1991692kB unevictable:19900kB isolated(anon):1372kB isolated(file):0kB mapped:961488kB dirty:363740kB
 writeback:136kB shmem:795424kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 34816kB writeback_tmp:0kB 
unstable:0kB pages_scanned:0 all_unreclaimable? no
[6840596.994640] Node 0 DMA free:15892kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB a
ctive_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocke
d:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0k
B local_pcp:0kB free_cma:0kB
[6840596.994644] lowmem_reserve[]: 0 3474 32124 32124 32124
[6840596.994647] Node 0 DMA32 free:121900kB min:7304kB low:10860kB high:14416kB active_anon:696956kB ina
ctive_anon:149260kB active_file:399652kB inactive_file:301808kB unevictable:0kB writepending:84576kB pre
sent:3652928kB managed:3587360kB mlocked:0kB slab_reclaimable:252108kB slab_unreclaimable:1608720kB kern
el_stack:1072kB pagetables:7052kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[6840596.994651] lowmem_reserve[]: 0 0 28649 28649 28649
[6840596.994655] Node 0 Normal free:56608kB min:60244kB low:89580kB high:118916kB active_anon:12800068kB
 inactive_anon:2986796kB active_file:3872608kB inactive_file:1689884kB unevictable:19900kB writepending:
279300kB present:29884416kB managed:29341076kB mlocked:19904kB slab_reclaimable:1073924kB slab_unreclaim
able:6097664kB kernel_stack:54032kB pagetables:182932kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0
kB
[6840596.994659] lowmem_reserve[]: 0 0 0 0 0
[6840596.994662] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512k
B 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[6840596.994673] Node 0 DMA32: 9033*4kB (UMEH) 9665*8kB (UMEH) 456*16kB (UMH) 36*32kB (UM) 0*64kB 0*128k
B 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 121900kB
[6840596.994682] Node 0 Normal: 8791*4kB (UMEH) 498*8kB (UMEH) 936*16kB (UMH) 86*32kB (UMH) 1*64kB (U) 0
*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56940kB
[6840596.994692] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[6840596.994693] 1804777 total pagecache pages
[6840596.994695] 38193 pages in swap cache
[6840596.994696] Swap cache stats: add 19906890, delete 19868626, find 959469929/965218437
[6840596.994697] Free swap  = 7997748kB
[6840596.994697] Total swap = 15335416kB
[6840596.994698] 8388330 pages RAM
[6840596.994698] 0 pages HighMem/MovableOnly
[6840596.994699] 152248 pages reserved
[6840596.994700] 0 pages cma reserved
[6840596.994700] 0 pages hwpoisoned

[adam-stokes]

Does this happen reliably? This isn't necessarily an issue with conjure-up but with your system specifically.

[naisanza]

@battlemidget running conjure-up via its text GUI caused the KVM to lock up 2 out of 2 times I've tried yesterday with a different message than the command-line method. I'll try and capture it today

The server is running 43 LXC containers, and 4 KVMS, and I have never seen it behave this way before with anything I've tried running before.

[adam-stokes]

@naisanza thanks, I've never come across this stacktrace before either. I'll talk to the juju/lxd guys to see if they have an idea. Another thought, could you be running out of memory?

[naisanza]

@battlemidget it's unlikely running out of memory, it's currently at 78% memory utilization, and 50% swap free

The KVM itself is sitting at 92.4% memory utilization, however, and never goes over it. I've only given it 2.5-3 GB to test conjure-up with, and I'm guessing the services running from conjure-up are meant to take up all available memory.

I know kubernetes currently has an issue with init, where if a VM is strapped for memory, a lot of odd behaviors happen, like certificates not being created before a pod dependent on it spawns. But I believe those were with very low resource limitations

[adam-stokes]

Can you try with kubernetes-core? That is lighter on resources, just want to rule out hardware limitations

[naisanza]

@battlemidget is the conjure-up command for kubernetes-core:

conjure-up canonical-kubernetes-core localhost

[adam-stokes]

@naisanza conjure-up kubernetes-core localhost

[naisanza]

@battlemidget

Try 1: Ran via ansible playbook

[info] Summoning kubernetes-core to localhost
[info] Bootstrapping Juju controller.
[info] Bootstrap complete.
[info] Running step: pre-deploy.
[info] Adding machines: 0: (xenial, ), 1: (xenial, cores=4 mem=4G)
[info] Added machines: 0, 1
[info] Deploying easyrsa...
[info] Deploying flannel...
[info] Deploying kubernetes-master...
[info] Deploying etcd...
[info] Deploying kubernetes-worker...
[error] while adding pending resource info for "flannel-s390x": read tcp 10.169.95.224:37260->10.169.95.224:37017: i/o timeout
[warning] Shutting down
[info] Summoning kubernetes-core to localhost
[info] Bootstrapping Juju controller.
[info] Bootstrap complete.
[info] Running step: pre-deploy.
[info] Adding machines: 0: (xenial) 1: (xenial cores=4 mem=4G)
[info] Added machines: 0 1
[info] Deploying easyrsa...
[info] Deploying flannel...
[info] Deploying kubernetes-master...
[info] Deploying etcd...
[info] Deploying kubernetes-worker...
[error] while adding pending resource info for flannel-s390x: read tcp 10.169.95.224:37260->10.169.95.224:37017: i/o timeout
[warning] Shutting down

Try 2:

admin@ubuntu:~$ conjure-up kubernetes-core localhost
[info] Summoning kubernetes-core to localhost
[info] Bootstrapping Juju controller.
[info] Bootstrap complete.
[info] Running step: pre-deploy.
[info] Adding machines: 0: (xenial, ), 1: (xenial, cores=4 mem=4G)
[info] Added machines: 0, 1
[info] Deploying etcd...
[info] Deploying kubernetes-master...
[info] Deploying kubernetes-worker...
[info] Deploying flannel...
[info] Deploying easyrsa...
[error] cannot add charm to storage: cannot clean up after failed storage operation because: read tcp 127.0.0.1:57210->127.0.0.1:37017: i/o timeout: cannot clean up after failed storage operation because: read tcp 127.0.0.1:57210->127.0.0.1:37017: i/o timeout: cannot update managed resource catalog: read tcp 127.0.0.1:57210->127.0.0.1:37017: i/o timeout
[warning] Shutting down

Try 3:

admin@ubuntu:~$ conjure-up kubernetes-core localhost
[info] Summoning kubernetes-core to localhost
[info] Bootstrapping Juju controller.
[error] Unable to bootstrap (cloud type: localhost)
[warning] Shutting down

Try 4: with 1 cpu, 2 cores

admin@ubuntu:~$ conjure-up kubernetes-core localhost
[info] Summoning kubernetes-core to localhost
[info] Bootstrapping Juju controller.
[info] Bootstrap complete.
[info] Running step: pre-deploy.
[info] Adding machines: 0: (xenial, ), 1: (xenial, cores=4 mem=4G)
[info] Added machines: 0, 1
[info] Deploying kubernetes-worker...
[info] Deploying flannel...
[info] Deploying kubernetes-master...
[info] Deploying etcd...
[info] Deploying easyrsa...
[error] while adding pending resource info for "etcd": read tcp 10.78.182.93:45310->10.78.182.93:37017: i/o timeout
[warning] Shutting down

[6931554.906639] kvm: page allocation stalls for 14792ms, order:0, mode:0x1c200c0(GFP_USER|__GF[23/1928]
[6931554.906653] CPU: 6 PID: 16677 Comm: kvm Tainted: P        W  O    4.10.15-1-pve #1
[6931554.906654] Hardware name: Supermicro X7DCL/X7DCL, BIOS 1.0b 05/12/2008
[6931554.906656] Call Trace:
[6931554.906667]  dump_stack+0x63/0x81
[6931554.906671]  warn_alloc+0x13a/0x170
[6931554.906673]  __alloc_pages_slowpath+0x927/0xba0
[6931554.906674]  __alloc_pages_nodemask+0x209/0x260
[6931554.906678]  alloc_pages_current+0x95/0x140
[6931554.906680]  __page_cache_alloc+0xc4/0xe0
[6931554.906682]  pagecache_get_page+0xab/0x2a0
[6931554.906685]  ? I_BDEV+0x20/0x20
[6931554.906687]  grab_cache_page_write_begin+0x20/0x40
[6931554.906688]  block_write_begin+0x2d/0xc0
[6931554.906690]  blkdev_write_begin+0x23/0x30
[6931554.906691]  generic_perform_write+0xc8/0x1c0
[6931554.906693]  __generic_file_write_iter+0x185/0x1d0
[6931554.906695]  blkdev_write_iter+0x8b/0x100
[6931554.906698]  ? rw_copy_check_uvector+0x69/0x130
[6931554.906700]  do_iter_readv_writev+0xb6/0x140
[6931554.906702]  do_readv_writev+0x19e/0x240
[6931554.906705]  ? pick_next_task_fair+0x47a/0x4b0
[6931554.906708]  ? __switch_to+0x23c/0x520
[6931554.906709]  ? __check_object_size+0x100/0x1d7
[6931554.906711]  vfs_writev+0x3c/0x50
[6931554.906713]  do_pwritev+0xb2/0xd0
[6931554.906714]  SyS_pwritev+0x11/0x20
[6931554.906717]  entry_SYSCALL_64_fastpath+0x1e/0xad
[6931554.906719] RIP: 0033:0x7fb45600df83
[6931554.906720] RSP: 002b:00007fb3537ce5a0 EFLAGS: 00000293 ORIG_RAX: 0000000000000128
[6931554.906722] RAX: ffffffffffffffda RBX: 00005572aac32618 RCX: 00007fb45600df83
[6931554.906723] RDX: 000000000000001f RSI: 00007fb382ffb600 RDI: 0000000000000015
[6931554.906724] RBP: 00007fb44a96a964 R08: 0000000000000000 R09: 00000000ffffffff
[6931554.906725] R10: 000000059fc56000 R11: 0000000000000293 R12: 00007fb3825afcf0
[6931554.906726] R13: 00007fb44a8bcce8 R14: 0000000000000000 R15: 00007fb46e2e4040
[6931554.906727] Mem-Info:
[6931554.906732] active_anon:3350661 inactive_anon:764146 isolated_anon:155
                  active_file:604300 inactive_file:1023206 isolated_file:0
                  unevictable:4975 dirty:87357 writeback:22 unstable:0
                  slab_reclaimable:332762 slab_unreclaimable:1915294
                  mapped:225688 shmem:160621 pagetables:47263 bounce:0
                  free:49194 free_pcp:0 free_cma:0
[6931554.906737] Node 0 active_anon:13402644kB inactive_anon:3056584kB active_file:2417200kB inactive_fi
le:4092824kB unevictable:19900kB isolated(anon):620kB isolated(file):0kB mapped:902752kB dirty:349428kB 
writeback:88kB shmem:642484kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 38912kB writeback_tmp:0kB un
stable:0kB pages_scanned:0 all_unreclaimable? no
[6931554.906738] Node 0 DMA free:15892kB min:32kB low:44kB high:56kB active_anon:0kB inactive_anon:0kB a
ctive_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15976kB managed:15892kB mlocke
d:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0k
B local_pcp:0kB free_cma:0kB
[6931554.906743] lowmem_reserve[]: 0 3474 32124 32124 32124
[6931554.906746] Node 0 DMA32 free:121828kB min:7304kB low:10860kB high:14416kB active_anon:514712kB ina
ctive_anon:144984kB active_file:227720kB inactive_file:689548kB unevictable:0kB writepending:14252kB pre
sent:3652928kB managed:3587360kB mlocked:0kB slab_reclaimable:260132kB slab_unreclaimable:1579312kB kern
el_stack:1220kB pagetables:5740kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[6931554.906750] lowmem_reserve[]: 0 0 28649 28649 28649
[6931554.906753] Node 0 Normal free:59056kB min:60244kB low:89580kB high:118916kB active_anon:12887932kB
 inactive_anon:2911600kB active_file:2189480kB inactive_file:3403276kB unevictable:19900kB writepending:335264kB present:29884416kB managed:29341076kB mlocked:19904kB slab_reclaimable:1070916kB slab_unreclaimable:6081864kB kernel_stack:53968kB pagetables:183312kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[6931554.906757] lowmem_reserve[]: 0 0 0 0 0
[6931554.906760] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15892kB
[6931554.906771] Node 0 DMA32: 2645*4kB (UMEH) 637*8kB (UMEH) 1148*16kB (UMEH) 1152*32kB (UMEH) 672*64kB (UM) 55*128kB (UM) 4*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 121980kB
[6931554.906781] Node 0 Normal: 14452*4kB (UMEH) 45*8kB (UMEH) 14*16kB (UMH) 10*32kB (H) 8*64kB (H) 4*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 59992kB
[6931554.906792] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[6931554.906793] 1852653 total pagecache pages
[6931554.906795] 62892 pages in swap cache
[6931554.906796] Swap cache stats: add 21093344, delete 21030364, find 976679694/982781767
[6931554.906797] Free swap  = 7585836kB
[6931554.906797] Total swap = 15335416kB
[6931554.906798] 8388330 pages RAM
[6931554.906799] 0 pages HighMem/MovableOnly
[6931554.906799] 152248 pages reserved
[6931554.906800] 0 pages cma reserved
[6931554.906800] 0 pages hwpoisoned

[adam-stokes]

I'll need to talk to some of the other teams to figure out what may be happening. This is the first time this issue has ever come up.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.