Nvidia-docker: Error response from daemon: OCI runtime create failed

Created on 11 Mar 2018 · 10Comments · Source: NVIDIA/nvidia-docker

1. Issue or feature description

Issue testing Xenial installation for nvidia-docker.

2. Steps to reproduce the issue

1). nvidia-docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:402: container init caused \"process_linux.go:385: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=cuda>=9.0 --pid=19639 /var/lib/docker/aufs/mnt/407489219b8518801044a038285066ca5138ca3216e7c27348eebaee807e2013]\\\\nnvidia-container-cli: initialization error: driver error: failed to process request\\\\n\\\"\"": unknown.

3. Information to attach (optional if deemed irrelevant)

[X] Kernel version from uname -a

Linux ubuntu-s-2vcpu-2gb-sfo2-01 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

[X] Any relevant kernel output lines from dmesg

[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Initializing cgroup subsys cpuacct
[    0.000000] Linux version 4.4.0-116-generic (buildd@lgw01-amd64-021) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9) ) #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 (Ubuntu 4.4.0-116.140-generic 4.4.98)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-116-generic root=UUID=a422faca-b2c8-42c6-b320-66f702b75fd3 ro console=tty1 console=ttyS0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Supporting XSAVE feature 0x01: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x02: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x04: 'AVX registers'
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] x86/fpu: Using 'eager' FPU context switches.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007fffafff] usable
[    0.000000] BIOS-e820: [mem 0x000000007fffb000-0x000000007fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.4 present.
[    0.000000] DMI: DigitalOcean Droplet, BIOS 20171212 12/12/2017
[    0.000000] Hypervisor detected: KVM
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x7fffb max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF uncachable
[    0.000000]   C0000-FFFFF write-protect
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0080000000 mask FF80000000 uncachable
[    0.000000]   1 disabled
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] found SMP MP-table at [mem 0x000f1270-0x000f127f] mapped at [ffff8800000f1270]
[    0.000000] Scanning 1 areas for low memory corruption
[    0.000000] Base memory trampoline at [ffff880000099000] 99000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x02215000, 0x02215fff] PGTABLE
[    0.000000] BRK [0x02216000, 0x02216fff] PGTABLE
[    0.000000] BRK [0x02217000, 0x02217fff] PGTABLE
[    0.000000] BRK [0x02218000, 0x02218fff] PGTABLE
[    0.000000] BRK [0x02219000, 0x02219fff] PGTABLE
[    0.000000] RAMDISK: [mem 0x34bbc000-0x365d5fff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000F10C0 000014 (v00 BOCHS )
[    0.000000] ACPI: RSDT 0x000000007FFFE410 000034 (v01 BOCHS  BXPCRSDT 00000001 BXPC 00000001)
[    0.000000] ACPI: FACP 0x000000007FFFFF80 000074 (v01 BOCHS  BXPCFACP 00000001 BXPC 00000001)
[    0.000000] ACPI: DSDT 0x000000007FFFE450 001135 (v01 BOCHS  BXPCDSDT 00000001 BXPC 00000001)
[    0.000000] ACPI: FACS 0x000000007FFFFF40 000040
[    0.000000] ACPI: SSDT 0x000000007FFFF6C0 00087A (v01 BOCHS  BXPCSSDT 00000001 BXPC 00000001)
[    0.000000] ACPI: APIC 0x000000007FFFF5D0 000080 (v01 BOCHS  BXPCAPIC 00000001 BXPC 00000001)
[    0.000000] ACPI: HPET 0x000000007FFFF590 000038 (v01 BOCHS  BXPCHPET 00000001 BXPC 00000001)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000007fffafff]
[    0.000000] NODE_DATA(0) allocated [mem 0x7fff6000-0x7fffafff]
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 0:7ffee001, primary cpu clock
[    0.000000] kvm-clock: using sched offset of 4272993793017 cycles
[    0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x000000007fffafff]
[    0.000000]   Normal   empty
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x000000007fffafff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000007fffafff]
[    0.000000] On node 0 totalpages: 524185
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3998 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 8128 pages used for memmap
[    0.000000]   DMA32 zone: 520187 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0xb008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ5 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] ACPI: IRQ10 used by override.
[    0.000000] ACPI: IRQ11 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] smpboot: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.000000] e820: [mem 0x80000000-0xfeffbfff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on KVM
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 34 pages/cpu @ffff88007fc00000 s99544 r8192 d31528 u1048576
[    0.000000] pcpu-alloc: s99544 r8192 d31528 u1048576 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 0 1 
[    0.000000] KVM setup async PF for cpu 0
[    0.000000] kvm-stealtime: cpu 0, msr 7fc101c0
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 515972
[    0.000000] Policy zone: DMA32
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.4.0-116-generic root=UUID=a422faca-b2c8-42c6-b320-66f702b75fd3 ro console=tty1 console=ttyS0
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 2018040K/2096740K available (8530K kernel code, 1309K rwdata, 3992K rodata, 1508K init, 1316K bss, 78700K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=2, Nodes=1
[    0.000000] Kernel/User page tables isolation: enabled
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=2
[    0.000000] NR_IRQS:33024 nr_irqs:440 16
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] console [tty1] enabled
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Detected 2199.998 MHz processor
[    0.261261] Calibrating delay loop (skipped) preset value.. 4399.99 BogoMIPS (lpj=8799992)
[    0.268124] pid_max: default: 32768 minimum: 301
[    0.269730] ACPI: Core revision 20150930
[    0.272184] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    0.274454] Security Framework initialized
[    0.275949] Yama: becoming mindful.
[    0.277321] AppArmor: AppArmor initialized
[    0.278999] Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.282441] Inode-cache hash table entries: 131072 (order: 8, 1048576 bytes)
[    0.285310] Mount-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.287643] Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes)
[    0.290308] Initializing cgroup subsys io
[    0.291840] Initializing cgroup subsys memory
[    0.293394] Initializing cgroup subsys devices
[    0.296013] Initializing cgroup subsys freezer
[    0.297634] Initializing cgroup subsys net_cls
[    0.299224] Initializing cgroup subsys perf_event
[    0.300927] Initializing cgroup subsys net_prio
[    0.302692] Initializing cgroup subsys hugetlb
[    0.304541] Initializing cgroup subsys pids
[    0.306118] FEATURE SPEC_CTRL Not Present
[    0.308305] mce: CPU supports 10 MCE banks
[    0.309745] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.311804] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.313879] Spectre V2 mitigation: Mitigation: Full generic retpoline
[    0.316320] Spectre V2 mitigation: Speculation control IBPB not-supported IBRS not-supported
[    0.320952] Freeing SMP alternatives memory: 32K
[    0.329597] ftrace: allocating 32186 entries in 126 pages
[    0.380860] smpboot: APIC(0) Converting physical 0 to logical package 0
[    0.383159] smpboot: APIC(1) Converting physical 1 to logical package 1
[    0.385448] smpboot: Max logical packages: 2
[    0.387158] x2apic enabled
[    0.389668] Switched APIC routing to physical x2apic.
[    0.392627] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[    0.394723] TSC deadline timer enabled
[    0.394754] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[    0.398363] Performance Events: 16-deep LBR, Broadwell events, Intel PMU driver.
[    0.437124] ... version:                2
[    0.438716] ... bit width:              48
[    0.440197] ... generic registers:      4
[    0.441631] ... value mask:             0000ffffffffffff
[    0.443470] ... max period:             000000007fffffff
[    0.445246] ... fixed-purpose events:   3
[    0.446810] ... event mask:             000000070000000f
[    0.448957] KVM setup paravirtual spinlock
[    0.453584] x86: Booting SMP configuration:
[    0.455188] .... node  #0, CPUs:      #1
[    0.456694] kvm-clock: cpu 1, msr 0:7ffee041, secondary cpu clock
[    0.479265] x86: Booted up 1 node, 2 CPUs
[    0.479270] KVM setup async PF for cpu 1
[    0.479275] kvm-stealtime: cpu 1, msr 7fd101c0
[    0.485073] smpboot: Total of 2 processors activated (8799.99 BogoMIPS)
[    0.488002] devtmpfs: initialized
[    0.490964] evm: security.selinux
[    0.492322] evm: security.SMACK64
[    0.493656] evm: security.SMACK64EXEC
[    0.495067] evm: security.SMACK64TRANSMUTE
[    0.496515] evm: security.SMACK64MMAP
[    0.497882] evm: security.ima
[    0.498985] evm: security.capability
[    0.500410] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.503661] futex hash table entries: 512 (order: 3, 32768 bytes)
[    0.505865] pinctrl core: initialized pinctrl subsystem
[    0.507751] RTC time:  2:22:46, date: 03/11/18
[    0.509466] NET: Registered protocol family 16
[    0.519703] cpuidle: using governor ladder
[    0.531733] cpuidle: using governor menu
[    0.533095] PCCT header not found.
[    0.534408] ACPI: bus type PCI registered
[    0.535823] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.538074] PCI: Using configuration type 1 for base access
[    0.552544] ACPI: Added _OSI(Module Device)
[    0.554246] ACPI: Added _OSI(Processor Device)
[    0.571185] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.574103] ACPI: Added _OSI(Processor Aggregator Device)
[    0.577982] ACPI: Interpreter enabled
[    0.579378] ACPI: (supports S0 S3 S4 S5)
[    0.580727] ACPI: Using IOAPIC for interrupt routing
[    0.582410] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.588422] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.590619] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[    0.593076] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    0.595480] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.599985] acpiphp: Slot [3] registered
[    0.601527] acpiphp: Slot [4] registered
[    0.603089] acpiphp: Slot [5] registered
[    0.607173] acpiphp: Slot [6] registered
[    0.608718] acpiphp: Slot [7] registered
[    0.610283] acpiphp: Slot [8] registered
[    0.611883] acpiphp: Slot [9] registered
[    0.613462] acpiphp: Slot [10] registered
[    0.615224] acpiphp: Slot [11] registered
[    0.616804] acpiphp: Slot [12] registered
[    0.618362] acpiphp: Slot [13] registered
[    0.619928] acpiphp: Slot [14] registered
[    0.621484] acpiphp: Slot [15] registered
[    0.623047] acpiphp: Slot [16] registered
[    0.624616] acpiphp: Slot [17] registered
[    0.626189] acpiphp: Slot [18] registered
[    0.627788] acpiphp: Slot [19] registered
[    0.629376] acpiphp: Slot [20] registered
[    0.630931] acpiphp: Slot [21] registered
[    0.632474] acpiphp: Slot [22] registered
[    0.633938] acpiphp: Slot [23] registered
[    0.635382] acpiphp: Slot [24] registered
[    0.638176] acpiphp: Slot [25] registered
[    0.639766] acpiphp: Slot [26] registered
[    0.641320] acpiphp: Slot [27] registered
[    0.642742] acpiphp: Slot [28] registered
[    0.644186] acpiphp: Slot [29] registered
[    0.645581] acpiphp: Slot [30] registered
[    0.647030] acpiphp: Slot [31] registered
[    0.648453] PCI host bridge to bus 0000:00
[    0.649895] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.651997] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.654190] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.656789] pci_bus 0000:00: root bus resource [mem 0x80000000-0xfebfffff window]
[    0.659288] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.661077] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    0.661493] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    0.662051] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    0.663752] pci 0000:00:01.1: reg 0x20: [io  0xc100-0xc10f]
[    0.664763] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.667819] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.670205] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.672774] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.675027] pci 0000:00:01.2: [8086:7020] type 00 class 0x0c0300
[    0.676585] pci 0000:00:01.2: reg 0x20: [io  0xc080-0xc09f]
[    0.677791] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    0.678220] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    0.680695] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    0.683459] pci 0000:00:02.0: [1b36:0100] type 00 class 0x030000
[    0.690906] pci 0000:00:02.0: reg 0x10: [mem 0xf8000000-0xfbffffff]
[    0.695718] pci 0000:00:02.0: reg 0x14: [mem 0xfc000000-0xfcffffff]
[    0.701422] pci 0000:00:02.0: reg 0x18: [mem 0xfd050000-0xfd051fff]
[    0.706773] pci 0000:00:02.0: reg 0x1c: [io  0xc0a0-0xc0bf]
[    0.720778] pci 0000:00:02.0: reg 0x30: [mem 0xfd040000-0xfd04ffff pref]
[    0.721972] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000
[    0.722667] pci 0000:00:03.0: reg 0x10: [io  0xc0c0-0xc0df]
[    0.723211] pci 0000:00:03.0: reg 0x14: [mem 0xfd052000-0xfd052fff]
[    0.727267] pci 0000:00:03.0: reg 0x30: [mem 0xfd000000-0xfd03ffff pref]
[    0.727568] pci 0000:00:04.0: [1af4:1004] type 00 class 0x010000
[    0.728660] pci 0000:00:04.0: reg 0x10: [io  0xc000-0xc03f]
[    0.729541] pci 0000:00:04.0: reg 0x14: [mem 0xfd053000-0xfd053fff]
[    0.735799] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000
[    0.736444] pci 0000:00:05.0: reg 0x10: [io  0xc040-0xc07f]
[    0.737203] pci 0000:00:05.0: reg 0x14: [mem 0xfd054000-0xfd054fff]
[    0.741621] pci 0000:00:06.0: [1af4:1002] type 00 class 0x00ff00
[    0.742134] pci 0000:00:06.0: reg 0x10: [io  0xc0e0-0xc0ff]
[    0.749027] ACPI: PCI Interrupt Link [LNKA] (IRQs 5 *10 11)
[    0.752002] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.754717] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.757475] ACPI: PCI Interrupt Link [LNKD] (IRQs 5 10 *11)
[    0.761744] ACPI: PCI Interrupt Link [LNKS] (IRQs *9)
[    0.765002] ACPI: Enabled 16 GPEs in block 00 to 0F
[    0.767806] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.769829] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.772824] vgaarb: loaded
[    0.773999] vgaarb: bridge control possible 0000:00:02.0
[    0.776407] SCSI subsystem initialized
[    0.778224] libata version 3.00 loaded.
[    0.778290] ACPI: bus type USB registered
[    0.779931] usbcore: registered new interface driver usbfs
[    0.781893] usbcore: registered new interface driver hub
[    0.783830] usbcore: registered new device driver usb
[    0.786024] PCI: Using ACPI for IRQ routing
[    0.787732] PCI: pci_cache_line_size set to 64 bytes
[    0.787998] e820: reserve RAM buffer [mem 0x0009fc00-0x0009ffff]
[    0.788005] e820: reserve RAM buffer [mem 0x7fffb000-0x7fffffff]
[    0.788325] NetLabel: Initializing
[    0.791023] NetLabel:  domain hash size = 128
[    0.792640] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.794568] NetLabel:  unlabeled traffic allowed by default
[    0.796924] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.799606] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.801945] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
[    0.807657] amd_nb: Cannot enumerate AMD northbridges
[    0.809619] clocksource: Switched to clocksource kvm-clock
[    0.820755] AppArmor: AppArmor Filesystem Enabled
[    0.822652] pnp: PnP ACPI init
[    0.824003] pnp 00:00: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.824047] pnp 00:01: Plug and Play ACPI device, IDs PNP0303 (active)
[    0.824074] pnp 00:02: Plug and Play ACPI device, IDs PNP0f13 (active)
[    0.824099] pnp 00:03: [dma 2]
[    0.824120] pnp 00:03: Plug and Play ACPI device, IDs PNP0700 (active)
[    0.824199] pnp 00:04: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.824427] pnp: PnP ACPI: found 5 devices
[    0.833277] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    0.836450] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    0.836453] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    0.836455] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    0.836457] pci_bus 0000:00: resource 7 [mem 0x80000000-0xfebfffff window]
[    0.836501] NET: Registered protocol family 2
[    0.838344] TCP established hash table entries: 16384 (order: 5, 131072 bytes)
[    0.841118] TCP bind hash table entries: 16384 (order: 6, 262144 bytes)
[    0.843549] TCP: Hash tables configured (established 16384 bind 16384)
[    0.845791] UDP hash table entries: 1024 (order: 3, 32768 bytes)
[    0.847801] UDP-Lite hash table entries: 1024 (order: 3, 32768 bytes)
[    0.850018] NET: Registered protocol family 1
[    0.852979] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    0.855026] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    0.857228] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    0.859828] ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11
[    0.862332] pci 0000:00:02.0: Video device with shadowed ROM
[    0.862381] PCI: CLS 0 bytes, default 64
[    0.862472] Unpacking initramfs...
[    1.358550] Freeing initrd memory: 26728K
[    1.369436] Scanning for low memory corruption every 60 seconds
[    1.371944] audit: initializing netlink subsys (disabled)
[    1.373894] audit: type=2000 audit(1520734967.327:1): initialized
[    1.376367] Initialise system trusted keyring
[    1.378147] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    1.380220] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    1.383842] zbud: loaded
[    1.385333] VFS: Disk quotas dquot_6.6.0
[    1.386797] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.389299] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    1.391564] fuse init (API version 7.23)
[    1.393226] Key type big_key registered
[    1.395882] Allocating IMA MOK and blacklist keyrings.
[    1.398239] Key type asymmetric registered
[    1.399768] Asymmetric key parser 'x509' registered
[    1.401632] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 249)
[    1.404501] io scheduler noop registered
[    1.406025] io scheduler deadline registered (default)
[    1.408059] io scheduler cfq registered
[    1.409741] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.411808] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.414194] intel_idle: does not run on family 6 model 79
[    1.414275] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    1.416892] ACPI: Power Button [PWRF]
[    1.418381] GHES: HEST is not enabled!
[    1.420044] ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 10
[    1.422115] virtio-pci 0000:00:03.0: virtio_pci: leaving for legacy driver
[    1.426456] virtio-pci 0000:00:04.0: virtio_pci: leaving for legacy driver
[    1.429430] ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 10
[    1.431714] virtio-pci 0000:00:05.0: virtio_pci: leaving for legacy driver
[    1.434829] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[    1.436979] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
[    1.440265] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    1.467096] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.472191] Linux agpgart interface v0.103
[    1.476471] loop: module loaded
[    1.479744]  vda: vda1 vda14 vda15
[    1.481402] ata_piix 0000:00:01.1: version 2.13
[    1.482292] scsi host0: ata_piix
[    1.483718] scsi host1: ata_piix
[    1.485015] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[    1.488959] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[    1.491658] libphy: Fixed MDIO Bus: probed
[    1.493301] tun: Universal TUN/TAP device driver, 1.6
[    1.495217] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    1.499016] PPP generic driver version 2.4.2
[    1.500814] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    1.503796] ehci-pci: EHCI PCI platform driver
[    1.505682] ehci-platform: EHCI generic platform driver
[    1.507552] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    1.510191] ohci-pci: OHCI PCI platform driver
[    1.512393] ohci-platform: OHCI generic platform driver
[    1.514684] uhci_hcd: USB Universal Host Controller Interface driver
[    1.519381] uhci_hcd 0000:00:01.2: UHCI Host Controller
[    1.521704] uhci_hcd 0000:00:01.2: new USB bus registered, assigned bus number 1
[    1.524536] uhci_hcd 0000:00:01.2: detected 2 ports
[    1.526290] uhci_hcd 0000:00:01.2: irq 11, io base 0x0000c080
[    1.528251] usb usb1: New USB device found, idVendor=1d6b, idProduct=0001
[    1.530514] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    1.532997] usb usb1: Product: UHCI Host Controller
[    1.534671] usb usb1: Manufacturer: Linux 4.4.0-116-generic uhci_hcd
[    1.536633] usb usb1: SerialNumber: 0000:00:01.2
[    1.538506] hub 1-0:1.0: USB hub found
[    1.539869] hub 1-0:1.0: 2 ports detected
[    1.541443] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    1.544988] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.546633] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.548726] mousedev: PS/2 mouse device common for all mice
[    1.552813] rtc_cmos 00:00: RTC can wake from S4
[    1.555029] rtc_cmos 00:00: rtc core: registered rtc_cmos as rtc0
[    1.557308] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1
[    1.557826] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram, hpet irqs
[    1.557845] i2c /dev entries driver
[    1.557953] device-mapper: uevent: version 1.0.3
[    1.560475] device-mapper: ioctl: 4.34.0-ioctl (2015-10-28) initialised: [email protected]
[    1.560509] ledtrig-cpu: registered to indicate activity on CPUs
[    1.561002] NET: Registered protocol family 10
[    1.561346] NET: Registered protocol family 17
[    1.561368] Key type dns_resolver registered
[    1.561679] microcode: CPU0 sig=0x406f1, pf=0x1, revision=0x1
[    1.562945] microcode: CPU1 sig=0x406f1, pf=0x1, revision=0x1
[    1.563002] microcode: Microcode Update Driver: v2.01 <[email protected]>, Peter Oruba
[    1.563246] registered taskstats version 1
[    1.563277] Loading compiled-in X.509 certificates
[    1.564622] Loaded X.509 cert 'Build time autogenerated kernel key: aa84f01984fbd529c105810b59e379d349698248'
[    1.564651] zswap: loaded using pool lzo/zbud
[    1.567209] Key type trusted registered
[    1.571239] Key type encrypted registered
[    1.571246] AppArmor: AppArmor sha1 policy hashing enabled
[    1.571258] ima: No TPM chip found, activating TPM-bypass!
[    1.571296] evm: HMAC attrs: 0x1
[    1.571702]   Magic number: 10:837:358
[    1.582993] rtc_cmos 00:00: setting system clock to 2018-03-11 02:22:47 UTC (1520734967)
[    1.583137] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[    1.583137] EDD information not available.
[    1.583208] PM: Hibernation image not present or could not be loaded.
[    1.647590] Freeing unused kernel memory: 1508K
[    1.649376] Write protecting the kernel read-only data: 14336k
[    1.652773] Freeing unused kernel memory: 1700K
[    1.654864] Freeing unused kernel memory: 104K
[    1.670542] random: systemd-udevd: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.670918] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.670940] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.680960] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681010] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681178] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681235] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681290] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681362] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.681441] random: udevadm: uninitialized urandom read (16 bytes read, 3 bits of entropy available)
[    1.741556] [drm] Initialized drm 1.1.0 20060810
[    1.744040] scsi host2: Virtio SCSI HBA
[    1.750234] FDC 0 is a S82078B
[    1.756281] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4
[    1.759825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3
[    1.760272] [drm] Device Version 0.0
[    1.760272] [drm] Compression level 0 log level 0
[    1.760273] [drm] Currently using mode #0, list at 0x488
[    1.760274] [drm] 12286 io pages at offset 0x1000000
[    1.760274] [drm] 16777216 byte draw area at offset 0x0
[    1.760275] [drm] RAM header offset: 0x3ffe000
[    1.760275] [drm] rom modes offset 0x488 for 128 modes
[    1.760302] [TTM] Zone  kernel: Available graphics memory: 1024056 kiB
[    1.760303] [TTM] Initializing pool allocator
[    1.760306] [TTM] Initializing DMA pool allocator
[    1.760313] [drm] qxl: 16M of VRAM memory size
[    1.760313] [drm] qxl: 63M of IO pages memory ready (VRAM domain)
[    1.760314] [drm] qxl: 16M of Surface memory size
[    1.766215] [drm] main mem slot 1 [f8000000,3ffe000]
[    1.766217] [drm] surface mem slot 2 [fc000000,1000000]
[    1.766244] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    1.766244] [drm] No driver support for vblank timestamp query.
[    1.766698] [drm] fb mappable at 0xF8000000, size 3145728
[    1.766700] [drm] fb: depth 24, pitch 4096, width 1024, height 768
[    1.808725] fbcon: qxldrmfb (fb0) is primary device
[    1.829451] Console: switching to colour frame buffer device 128x48
[    1.843451] qxl 0000:00:02.0: fb0: qxldrmfb frame buffer device
[    1.845546] AVX2 version of gcm_enc/dec engaged.
[    1.847213] AES CTR mode by8 optimization enabled
[    1.853706] [drm] Initialized qxl 0.1.0 20120117 for 0000:00:02.0 on minor 0
[    1.972723] md: linear personality registered for level -1
[    1.977476] md: multipath personality registered for level -4
[    1.981889] md: raid0 personality registered for level 0
[    1.986346] md: raid1 personality registered for level 1
[    2.057632] raid6: sse2x1   gen()  7672 MB/s
[    2.125631] raid6: sse2x1   xor()  4861 MB/s
[    2.193627] raid6: sse2x2   gen()  6180 MB/s
[    2.261630] raid6: sse2x2   xor()  6198 MB/s
[    2.329633] raid6: sse2x4   gen() 10254 MB/s
[    2.365710] tsc: Refined TSC clocksource calibration: 2199.990 MHz
[    2.367933] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1fb6296d523, max_idle_ns: 440795274677 ns
[    2.397640] raid6: sse2x4   xor()  4812 MB/s
[    2.465631] raid6: avx2x1   gen() 10128 MB/s
[    2.533625] raid6: avx2x2   gen() 16133 MB/s
[    2.601636] raid6: avx2x4   gen() 12365 MB/s
[    2.603042] raid6: using algorithm avx2x2 gen() 16133 MB/s
[    2.604862] raid6: using avx2x2 recovery algorithm
[    2.608251] xor: automatically using best checksumming function:
[    2.649639]    avx       : 14954.000 MB/sec
[    2.654990] async_tx: api initialized (async)
[    2.668362] md: raid6 personality registered for level 6
[    2.670699] md: raid5 personality registered for level 5
[    2.672865] md: raid4 personality registered for level 4
[    2.685512] md: raid10 personality registered for level 10
[    2.727122] Btrfs loaded
[    2.763103] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: (null)
[    2.837545] systemd[1]: systemd 229 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN)
[    2.845740] systemd[1]: Detected virtualization kvm.
[    2.849658] systemd[1]: Detected architecture x86-64.
[    2.855713] systemd[1]: Set hostname to <ubuntu-s-2vcpu-2gb-sfo2-01>.
[    2.939525] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[    2.945115] systemd[1]: Listening on Syslog Socket.
[    2.948826] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    2.954680] systemd[1]: Created slice User and Session Slice.
[    2.959541] systemd[1]: Reached target User and Group Name Lookups.
[    2.965202] systemd[1]: Listening on udev Kernel Socket.
[    2.970787] systemd[1]: Listening on udev Control Socket.
[    2.976154] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[    2.982257] systemd[1]: Started Trigger resolvconf update for networkd DNS.
[    2.988165] systemd[1]: Listening on LVM2 poll daemon socket.
[    2.993883] systemd[1]: Created slice System Slice.
[    2.998096] systemd[1]: Created slice system-serial\x2dgetty.slice.
[    3.002575] systemd[1]: Reached target Encrypted Volumes.
[    3.007502] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    3.012374] systemd[1]: Listening on Journal Audit Socket.
[    3.017131] systemd[1]: Listening on LVM2 metadata daemon socket.
[    3.022846] systemd[1]: Listening on Journal Socket (/dev/log).
[    3.028442] systemd[1]: Listening on Journal Socket.
[    3.045718] systemd[1]: Mounting Debug File System...
[    3.051301] systemd[1]: Starting Set console keymap...
[    3.056676] systemd[1]: Starting Journal Service...
[    3.061876] systemd[1]: Mounting POSIX Message Queue File System...
[    3.067980] systemd[1]: Starting Nameserver information manager...
[    3.075727] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[    3.085289] systemd[1]: Starting Load Kernel Modules...
[    3.090574] systemd[1]: Starting Uncomplicated firewall...
[    3.095213] Loading iSCSI transport class v2.0-870.
[    3.097314] systemd[1]: Starting Create list of required static device nodes for the current kernel...
[    3.107304] systemd[1]: Mounting Huge Pages File System...
[    3.114401] systemd[1]: Starting Remount Root and Kernel File Systems...
[    3.117203] iscsi: registered transport (tcp)
[    3.122954] systemd[1]: Reached target Swap.
[    3.126545] EXT4-fs (vda1): re-mounted. Opts: (null)
[    3.128766] systemd[1]: Reached target Slices.
[    3.150610] systemd[1]: Mounted POSIX Message Queue File System.
[    3.154687] systemd[1]: Mounted Huge Pages File System.
[    3.158449] systemd[1]: Mounted Debug File System.
[    3.160046] iscsi: registered transport (iser)
[    3.163439] systemd[1]: Started Journal Service.
[    3.343685] systemd-journald[605]: Received request to flush runtime journal from PID 1
[    3.561358] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0xb100, revision 0
[    3.837471] intel_rapl: no valid rapl domains found in package 0
[    4.189082] ppdev: user-space parallel port driver
[    4.214077] audit: type=1400 audit(1520734970.140:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=990 comm="apparmor_parser"
[    4.214086] audit: type=1400 audit(1520734970.140:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=990 comm="apparmor_parser"
[    4.214091] audit: type=1400 audit(1520734970.140:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=990 comm="apparmor_parser"
[    4.214096] audit: type=1400 audit(1520734970.140:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=990 comm="apparmor_parser"
[    4.216476] audit: type=1400 audit(1520734970.140:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/sbin/dhclient" pid=991 comm="apparmor_parser"
[    4.216484] audit: type=1400 audit(1520734970.140:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=991 comm="apparmor_parser"
[    4.216489] audit: type=1400 audit(1520734970.140:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=991 comm="apparmor_parser"
[    4.216501] audit: type=1400 audit(1520734970.140:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=991 comm="apparmor_parser"
[    4.218777] audit: type=1400 audit(1520734970.144:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/lxc-start" pid=992 comm="apparmor_parser"
[    5.300434] cgroup: new mount options do not match the existing superblock, will be ignored
[    5.374856] random: nonblocking pool is initialized
[    5.818774] aufs 4.x-rcN-20160111
[    5.900348] bridge: automatic filtering via arp/ip/ip6tables has been deprecated. Update your scripts to load br_netfilter if you need this.
[    5.903025] Bridge firewalling registered
[    5.909387] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    5.926560] ip_tables: (C) 2000-2006 Netfilter Core Team
[    5.985753] Initializing XFRM netlink socket
[    5.992681] Netfilter messages via NETLINK v0.30.
[    5.996364] ctnetlink v0.93: registering with nfnetlink.
[    6.035267] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[    6.114917] aufs au_opts_verify:1597:dockerd[1436]: dirperm1 breaks the protection by the permission bits on the lower branch
[   41.887409] aufs au_opts_verify:1597:dockerd[1437]: dirperm1 breaks the protection by the permission bits on the lower branch
[   41.911270] aufs au_opts_verify:1597:dockerd[1437]: dirperm1 breaks the protection by the permission bits on the lower branch
[   41.931121] aufs au_opts_verify:1597:dockerd[1391]: dirperm1 breaks the protection by the permission bits on the lower branch
[   41.938149] device vethf14b12d entered promiscuous mode
[   41.938229] IPv6: ADDRCONF(NETDEV_UP): vethf14b12d: link is not ready
[   42.118147] eth0: renamed from veth53da118
[   42.129413] IPv6: ADDRCONF(NETDEV_CHANGE): vethf14b12d: link becomes ready
[   42.129445] docker0: port 1(vethf14b12d) entered forwarding state
[   42.129455] docker0: port 1(vethf14b12d) entered forwarding state
[   42.129479] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[   42.196583] docker0: port 1(vethf14b12d) entered disabled state
[   42.196643] veth53da118: renamed from eth0
[   42.222188] docker0: port 1(vethf14b12d) entered disabled state
[   42.223148] device vethf14b12d left promiscuous mode
[   42.223152] docker0: port 1(vethf14b12d) entered disabled state
[  147.271313] aufs au_opts_verify:1597:dockerd[1438]: dirperm1 breaks the protection by the permission bits on the lower branch
[  147.295234] aufs au_opts_verify:1597:dockerd[1438]: dirperm1 breaks the protection by the permission bits on the lower branch
[  147.317553] aufs au_opts_verify:1597:dockerd[1363]: dirperm1 breaks the protection by the permission bits on the lower branch
[  147.330621] device veth89ace85 entered promiscuous mode
[  147.331799] IPv6: ADDRCONF(NETDEV_UP): veth89ace85: link is not ready
[  147.469855] eth0: renamed from veth4238435
[  147.481824] IPv6: ADDRCONF(NETDEV_CHANGE): veth89ace85: link becomes ready
[  147.481854] docker0: port 1(veth89ace85) entered forwarding state
[  147.481864] docker0: port 1(veth89ace85) entered forwarding state
[  147.554239] docker0: port 1(veth89ace85) entered disabled state
[  147.554291] veth4238435: renamed from eth0
[  147.584205] docker0: port 1(veth89ace85) entered disabled state
[  147.585143] device veth89ace85 left promiscuous mode
[  147.585147] docker0: port 1(veth89ace85) entered disabled state
[  826.604994] aufs au_opts_verify:1597:dockerd[1756]: dirperm1 breaks the protection by the permission bits on the lower branch
[  826.624520] aufs au_opts_verify:1597:dockerd[1756]: dirperm1 breaks the protection by the permission bits on the lower branch
[  826.655444] aufs au_opts_verify:1597:dockerd[1391]: dirperm1 breaks the protection by the permission bits on the lower branch
[  826.662159] device vethb4114b1 entered promiscuous mode
[  826.662279] IPv6: ADDRCONF(NETDEV_UP): vethb4114b1: link is not ready
[  826.787833] eth0: renamed from veth0d27535
[  826.799530] IPv6: ADDRCONF(NETDEV_CHANGE): vethb4114b1: link becomes ready
[  826.799572] docker0: port 1(vethb4114b1) entered forwarding state
[  826.799582] docker0: port 1(vethb4114b1) entered forwarding state
[  826.875317] docker0: port 1(vethb4114b1) entered disabled state
[  826.875377] veth0d27535: renamed from eth0
[  826.903258] docker0: port 1(vethb4114b1) entered disabled state
[  826.904179] device vethb4114b1 left promiscuous mode
[  826.904183] docker0: port 1(vethb4114b1) entered disabled state
[  944.053306] aufs au_opts_verify:1597:dockerd[1436]: dirperm1 breaks the protection by the permission bits on the lower branch
[  944.069822] aufs au_opts_verify:1597:dockerd[1436]: dirperm1 breaks the protection by the permission bits on the lower branch
[  944.092416] aufs au_opts_verify:1597:dockerd[1391]: dirperm1 breaks the protection by the permission bits on the lower branch
[  944.095695] device vethe93f058 entered promiscuous mode
[  944.095771] IPv6: ADDRCONF(NETDEV_UP): vethe93f058: link is not ready
[  944.217979] eth0: renamed from veth0872f07
[  944.229969] IPv6: ADDRCONF(NETDEV_CHANGE): vethe93f058: link becomes ready
[  944.229999] docker0: port 1(vethe93f058) entered forwarding state
[  944.230008] docker0: port 1(vethe93f058) entered forwarding state
[  944.416041] docker0: port 1(vethe93f058) entered disabled state
[  944.416099] veth0872f07: renamed from eth0
[  944.440572] docker0: port 1(vethe93f058) entered disabled state
[  944.441756] device vethe93f058 left promiscuous mode
[  944.441761] docker0: port 1(vethe93f058) entered disabled state
[  990.921177] aufs au_opts_verify:1597:dockerd[3261]: dirperm1 breaks the protection by the permission bits on the lower branch
[  995.969934] aufs au_opts_verify:1597:dockerd[3261]: dirperm1 breaks the protection by the permission bits on the lower branch
[  995.992631] aufs au_opts_verify:1597:dockerd[3261]: dirperm1 breaks the protection by the permission bits on the lower branch
[  996.014803] aufs au_opts_verify:1597:dockerd[3261]: dirperm1 breaks the protection by the permission bits on the lower branch
[  996.019463] device veth3b09f49 entered promiscuous mode
[  996.019538] IPv6: ADDRCONF(NETDEV_UP): veth3b09f49: link is not ready
[  996.179212] eth0: renamed from vethdcec3c1
[  996.195295] IPv6: ADDRCONF(NETDEV_CHANGE): veth3b09f49: link becomes ready
[  996.195327] docker0: port 1(veth3b09f49) entered forwarding state
[  996.195337] docker0: port 1(veth3b09f49) entered forwarding state
[  996.262447] docker0: port 1(veth3b09f49) entered disabled state
[  996.262489] vethdcec3c1: renamed from eth0
[  996.289775] docker0: port 1(veth3b09f49) entered disabled state
[  996.290756] device veth3b09f49 left promiscuous mode
[  996.290760] docker0: port 1(veth3b09f49) entered disabled state
[ 2388.120021] aufs au_opts_verify:1597:dockerd[3435]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2388.138973] aufs au_opts_verify:1597:dockerd[3435]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2388.158810] aufs au_opts_verify:1597:dockerd[3259]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2388.163254] device vethc859349 entered promiscuous mode
[ 2388.163328] IPv6: ADDRCONF(NETDEV_UP): vethc859349: link is not ready
[ 2388.274342] eth0: renamed from veth4d22e90
[ 2388.289932] IPv6: ADDRCONF(NETDEV_CHANGE): vethc859349: link becomes ready
[ 2388.289960] docker0: port 1(vethc859349) entered forwarding state
[ 2388.289967] docker0: port 1(vethc859349) entered forwarding state
[ 2388.348899] docker0: port 1(vethc859349) entered disabled state
[ 2388.348949] veth4d22e90: renamed from eth0
[ 2388.373704] docker0: port 1(vethc859349) entered disabled state
[ 2388.374542] device vethc859349 left promiscuous mode
[ 2388.374545] docker0: port 1(vethc859349) entered disabled state
[ 2490.342061] aufs au_opts_verify:1597:dockerd[3264]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2490.365300] aufs au_opts_verify:1597:dockerd[3264]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2490.386077] aufs au_opts_verify:1597:dockerd[3261]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2490.391059] device vethecad6d6 entered promiscuous mode
[ 2490.391332] IPv6: ADDRCONF(NETDEV_UP): vethecad6d6: link is not ready
[ 2490.535027] eth0: renamed from veth717bf5d
[ 2490.546995] IPv6: ADDRCONF(NETDEV_CHANGE): vethecad6d6: link becomes ready
[ 2490.547018] docker0: port 1(vethecad6d6) entered forwarding state
[ 2490.547025] docker0: port 1(vethecad6d6) entered forwarding state
[ 2490.613839] docker0: port 1(vethecad6d6) entered disabled state
[ 2490.613899] veth717bf5d: renamed from eth0
[ 2490.637424] docker0: port 1(vethecad6d6) entered disabled state
[ 2490.638418] device vethecad6d6 left promiscuous mode
[ 2490.638421] docker0: port 1(vethecad6d6) entered disabled state
[ 2695.963050] aufs au_opts_verify:1597:dockerd[3247]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2695.987882] aufs au_opts_verify:1597:dockerd[3247]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2696.012935] aufs au_opts_verify:1597:dockerd[3435]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 2696.015827] device veth5dbce35 entered promiscuous mode
[ 2696.016045] IPv6: ADDRCONF(NETDEV_UP): veth5dbce35: link is not ready
[ 2696.163361] eth0: renamed from veth19d0578
[ 2696.178852] IPv6: ADDRCONF(NETDEV_CHANGE): veth5dbce35: link becomes ready
[ 2696.178872] docker0: port 1(veth5dbce35) entered forwarding state
[ 2696.178879] docker0: port 1(veth5dbce35) entered forwarding state
[ 2696.246486] docker0: port 1(veth5dbce35) entered disabled state
[ 2696.246538] veth19d0578: renamed from eth0
[ 2696.269248] docker0: port 1(veth5dbce35) entered disabled state
[ 2696.270464] device veth5dbce35 left promiscuous mode
[ 2696.270468] docker0: port 1(veth5dbce35) entered disabled state
[ 3168.047092] aufs au_opts_verify:1597:dockerd[3260]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3168.069255] aufs au_opts_verify:1597:dockerd[3260]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3168.100338] aufs au_opts_verify:1597:dockerd[3435]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3168.101339] device vetha2e3e8b entered promiscuous mode
[ 3168.101410] IPv6: ADDRCONF(NETDEV_UP): vetha2e3e8b: link is not ready
[ 3168.248365] eth0: renamed from veth8c4ef72
[ 3168.259795] IPv6: ADDRCONF(NETDEV_CHANGE): vetha2e3e8b: link becomes ready
[ 3168.259835] docker0: port 1(vetha2e3e8b) entered forwarding state
[ 3168.259845] docker0: port 1(vetha2e3e8b) entered forwarding state
[ 3168.320111] docker0: port 1(vetha2e3e8b) entered disabled state
[ 3168.320154] veth8c4ef72: renamed from eth0
[ 3168.354552] docker0: port 1(vetha2e3e8b) entered disabled state
[ 3168.355404] device vetha2e3e8b left promiscuous mode
[ 3168.355417] docker0: port 1(vetha2e3e8b) entered disabled state
[ 3280.124971] aufs au_opts_verify:1597:dockerd[3264]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3280.158479] aufs au_opts_verify:1597:dockerd[3264]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3280.180173] aufs au_opts_verify:1597:dockerd[3435]: dirperm1 breaks the protection by the permission bits on the lower branch
[ 3280.181197] device veth1059e61 entered promiscuous mode
[ 3280.181254] IPv6: ADDRCONF(NETDEV_UP): veth1059e61: link is not ready
[ 3280.337319] eth0: renamed from veth352876e
[ 3280.349333] IPv6: ADDRCONF(NETDEV_CHANGE): veth1059e61: link becomes ready
[ 3280.349358] docker0: port 1(veth1059e61) entered forwarding state
[ 3280.349366] docker0: port 1(veth1059e61) entered forwarding state
[ 3280.414837] docker0: port 1(veth1059e61) entered disabled state
[ 3280.414883] veth352876e: renamed from eth0
[ 3280.448557] docker0: port 1(veth1059e61) entered disabled state
[ 3280.449941] device veth1059e61 left promiscuous mode
[ 3280.449946] docker0: port 1(veth1059e61) entered disabled state

[X] Driver information from nvidia-smi -a

Not applicable.
[X] Docker version from docker version

17.12.1-ce
[X] NVIDIA packages version from dpkg -l '*nvidia*' _or_ rpm -qa '*nvidia*'

_or_: command not found
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                    Version                          Architecture                     Description
+++-=======================================================-================================-================================-====================================================================================================================
ii  libnvidia-container-tools                               1.0.0~beta.1-1                   amd64                            NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64                              1.0.0~beta.1-1                   amd64                            NVIDIA container runtime library
ii  nvidia-container-runtime                                2.0.0+docker17.12.1-1            amd64                            NVIDIA container runtime
ii  nvidia-container-runtime-hook                           1.3.0-1                          amd64                            NVIDIA container runtime hook
un  nvidia-docker                                           <none>                           <none>                           (no description available)
ii  nvidia-docker2                                          2.0.3+docker17.12.1-1            all                              nvidia-docker CLI wrapper

[X] NVIDIA container library version from nvidia-container-cli -V

version: 1.0.0

[X] NVIDIA container library logs (see troubleshooting)

$ nvidia-container-cli -V
version: 1.0.0
build date: 2018-03-06T01:53+00:00
build revision: be797da00b156493e80f1ae6f38d69f23c932554
build compiler: gcc-5 5.4.0 20160609
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
root@ubuntu-s-2vcpu-2gb-sfo2-01:~# cat /etc/nvidia-container-runtime/config.toml
disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-runtime-hook.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
ldconfig = "@/sbin/ldconfig.real"

[X] Docker command, image and tag used

command used:  nvidia-docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
image: nvidia/cuda
tag:  latest

Source

conradwt

Most helpful comment

I met this problem while I suspend my computer, then I restart my computer, this error was solved. I guess it was because the docker daemon missed driver library path. So I suggest if you have installed the nvidia driver, cuda and nvidia-docker exactly followed the official docs, you should restart you computer.

helloworld77 on 29 Mar 2018

👍11

All 10 comments

Driver information from nvidia-smi -a
Not applicable.

What do you mean? You don't have the NVIDIA driver installed?

flx42 on 11 Mar 2018

@flx42 The nvidia-smi -a generates the following message:

$ nvidia-smi -a
nvidia-smi: command not found

The host doesn't have an Nvidia driver because it doesn't support Nvidia GPU hardware at this time. Thus, I'll be setting up a stack on GCP which does have GPU support.

conradwt on 11 Mar 2018

You do need a GPU to run the containers, but you can very well build any GPU app on a machine without a GPU. But you don't need nvidia-docker to do that.

flx42 on 11 Mar 2018

I had the same problem, and I resolved it by adding to my ld library path the directory where are the nvidia libraries:

$ cat /etc/ld.so.conf.d/nvidia.conf 
/usr/lib/nvidia-390/

$ sudo ldconfig
$ sudo systemctl restart docker.service
$ nvidia-docker run --rm nvidia/cuda nvidia-smi
Sun Mar 25 00:19:50 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.25                 Driver Version: 390.25                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 00000000:02:00.0 Off |                  N/A |
| N/A   38C    P0    N/A /  N/A |      0MiB /  4046MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I'm using Ubuntu 17.10 and the following nvidia apt package:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 17.10
Release:        17.10
Codename:       artful

$ apt-cache policy nvidia-390
nvidia-390:
  Installed: 390.25-0ubuntu0~gpu17.10.1
  Candidate: 390.25-0ubuntu0~gpu17.10.1

szaffarano on 25 Mar 2018

👍3

helloworld77 on 29 Mar 2018

👍11

I happen the same problem, and I check that I should re-install cuda drivers.

linrio on 19 Aug 2018

👎4

I happen the same problem, and I check that I should re-install cuda drivers.

I think this is a red herring, there should be no CUDA requirement to install nvidia-docker. Do you mean you updated your NVIDIA drivers? This would likely be a fix, as the issues here are most likely related to a driver-dockerCUDA incompatibility (see this issue) . For example, with my driver version of 396.18, the latest version of nvidia-docker I could run was for CUDA 9.1, and my command was:

docker run --runtime=nvidia --rm nvidia/cuda:9.1-devel nvidia-smi

find your version by getting your exact driver:

cat /proc/driver/nvidia/version

and then get the matching CUDA version you need from the chart here

balloch on 2 Oct 2018

👍5

I happen the same problem, and I check that I should re-install cuda drivers.

I think this is a red herring, there should be no CUDA requirement to install nvidia-docker. Do you mean you updated your NVIDIA drivers? This would likely be a fix, as the issues here are most likely related to a driver-dockerCUDA incompatibility (see this issue) . For example, with my driver version of 396.18, the latest version of nvidia-docker I could run was for CUDA 9.1, and my command was:

docker run --runtime=nvidia --rm nvidia/cuda:9.1-devel nvidia-smi

find your version by getting your exact driver:

cat /proc/driver/nvidia/version

and then get the matching CUDA version you need from the chart here

i have the same issue..
ifmy GPU is tesla p4 and my driver version is 396.37, can i choose 9.2-devel
or
i must change my driver to 384.130-385.00, so i can run 10-devel

huyutuo on 7 Jan 2019

Hello,

I faced the same issue and I managed to solve it by upgrading the driver version. This issue seems to be caused by an inconsistency between the Driver and CUDA version.

Locate the libcuda.so (Note that it is important to take the version of libcuda.so, rather than the version shown on nvidia-smi.):

$ locate libcuda.so
/usr/lib/i386-linux-gnu/libcuda.so
/usr/lib/i386-linux-gnu/libcuda.so.1
/usr/lib/i386-linux-gnu/libcuda.so.384.xx
/usr/lib/x86_64-linux-gnu/libcuda.so
/usr/lib/x86_64-linux-gnu/libcuda.so.1
/usr/lib/x86_64-linux-gnu/libcuda.so.384.xx

In my case, the driver version is 384 and I have been trying to use an image from CUDA 9.2 and it is therefore giving me trouble. Upgrading to nvidia-396 helps resolve the problem (DO NOT forget to restart the computer after driver upgrade!).

ArmageddonKnight on 19 Feb 2019

Probably no need to reboot the machine.
sudo service docker restart solved the ... running prestart hook 0 caused \\\"fork/exec /usr/bin/dockerd (deleted): no such file or directory\\\"\"": unknown. issue for me (with container not related to this project at all).