Lxd: CUDA installed on host and in container, gpu added (using tutorial) results in: NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver.

Created on 8 Aug 2017  路  16Comments  路  Source: lxc/lxd

The template below is mostly useful for bug reports and support questions.
Feel free to remove anything which doesn't apply to you and add more information where it makes sense.

Required information

  • Distribution: Ubuntu
  • Distribution version: 16.04LTS
  • The output of "lxc info" or if that fails:
    config:
    core.https_address: '[::]:8443'
    core.trust_password: true
    api_extensions:

    • storage_zfs_remove_snapshots

    • container_host_shutdown_timeout

    • container_syscall_filtering

    • auth_pki

    • container_last_used_at

    • etag

    • patch

    • usb_devices

    • https_allowed_credentials

    • image_compression_algorithm

    • directory_manipulation

    • container_cpu_time

    • storage_zfs_use_refquota

    • storage_lvm_mount_options

    • network

    • profile_usedby

    • container_push

    • container_exec_recording

    • certificate_update

    • container_exec_signal_handling

    • gpu_devices

    • container_image_properties

    • migration_progress

    • id_map

    • network_firewall_filtering

    • network_routes

    • storage

    • file_delete

    • file_append

    • network_dhcp_expiry

    • storage_lvm_vg_rename

    • storage_lvm_thinpool_rename

    • network_vlan

    • image_create_aliases

    • container_stateless_copy

    • container_only_migration

    • storage_zfs_clone_copy

    • unix_device_rename

    • storage_lvm_use_thinpool

    • storage_rsync_bwlimit

    • network_vxlan_interface

    • storage_btrfs_mount_options

    • entity_description

    • image_force_refresh

    • storage_lvm_lv_resizing

    • id_map_base

    • file_symlinks

    • container_push_target

    • network_vlan_physical

    • storage_images_delete

    • container_edit_metadata

    • container_snapshot_stateful_migration

    • storage_driver_ceph

      api_status: stable

      api_version: "1.0"

      auth: trusted

      public: false

      environment:

      addresses:

    • 192.168.1.105:8443

      architectures:

    • x86_64

    • i686

      certificate: |

      -----BEGIN CERTIFICATE-----

      MIIFpDCCA4ygAwIBAgIQBulS6UX+T6a2aMLQCglDxzANBgkqhkiG9w0BAQsFADAz

      MRwwGgYDVQQKExNsaW51eGNvbnRhaW5lcnMub3JnMRMwEQYDVQQDDApyb290QG5v

      ZGU1MB4XDTE2MTEyMjEzMjgxMloXDTI2MTEyMDEzMjgxMlowMzEcMBoGA1UEChMT

      bGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBub2RlNTCCAiIwDQYJ

      KoZIhvcNAQEBBQADggIPADCCAgoCggIBANnt8D7kNw34A1EAEAMNsKJSXQGOp9DX

      xj+L2ZtvkgVCTTNvM28wavyn/ngS2/on3DJWu7mlaT2p5rGQOEtX7xo8nmfROJnj

      jra/moXy1Xv4Zyz0SG2RM+StYUUCfq9h0+y23DTbv6zZp1HQCSB5vihrIPnRFHah

      NMuV5QyRpbCxeB7Gg9r6YamzT/CeuL9M2bFgbUw6rK/xmtHo+41Li/63YMiUeFxE

      C7sPRaXFlTo1G7vuOkK+LWLwYL5//eoM3Zgc+pcS/qJdnnWKx3ncE5LyN5q9/wyg

      jh2baH0ryBagIvnMy41quYBoJRX9wF23tHJBv0FvpQcLnIqlFiolxw6oGxSYgiH/

      6euO5nOAwUn/BswCMF1HyiGqKma/qIqdpjhk2o7TBAyoCvRCyGWUpmChfC5yHfAW

      BL+I5FSsE12wXKT7P5+9F4C2WXi33v5uDsumAQf7lNyJhF0uaGXJ6W2Xn9sqmlrc

      29V7fMAb+JkV4aJkqyKoeqmR4beWUPDDaTTkXYQdOIaN8Gjn0fDZfTM3Db7Wcw/X

      rIsDVxxm4figYtnPQ2fdQ0JJB086x2dTG29awUZIrpOuycrh7VuREGrljcuG0rQm

      QL/LICfOM1Tive+MxGIeW7qLoNwynIdu2rFxq/gsAEpzy3XWabfOof8MrsoevWZk

      /e2Zn9QG+8T7AgMBAAGjgbMwgbAwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoG

      CCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwewYDVR0RBHQwcoIFbm9kZTWCEDE5Mi4x

      NjguMS4xMDUvMjSCEDE5Mi4xNjguMS4xMDUvMjSCG2ZlODA6OmVjNDo3YWZmOmZl

      ZDg6YjgzMC82NIIcZmU4MDo6OGNlYjpkMGZmOmZlY2M6NGJkYS82NIIKZmU4MDo6

      MS82NDANBgkqhkiG9w0BAQsFAAOCAgEAwCmTR1NpuvZE/vsOKmT1m8pp/DsL80oE

      nvdhc94OIWbclVEo+9SpNfGYnHO4WKV5CBHhR86XJEZJ3XRu+xdJvzrS3gprwepl

      veut0t1QK4k51si2UpCdaJC5/nI2UrD4DvPH/b1l2mUznT8Rmca3KFgbupFqRnMM

      IPATRud6Dr9PPolF5SBF9T7f9a0n1rQiI0x36h6OyjtpYqfHVokAYNaq4e9mQ32Z

      CJsismz2mG4AAlVjII3n8wqt5AX04eJEFbsZKuu+hCwW4V7G0j/e2qSdyJq4uUEz

      +ZaEn3hov6hywZiZSJLfxEFArr5aSJ5++n76pc69sxRJE5mJ96b1bA1J0l2F2E//

      Yb5hrBvINfXj/OkuNpMGQRkVZmiulWeK8DXwIEfDp79sk/wz7tTG1vzdP16MxHq0

      iby1EH6y635r48JWX+xrpFu3qQN4mErXzrLq5ADUyg8REpdrX2xJsUgMJWYKg8FN

      DbvcxWhCcrzZEnfrWQluE/yJ4zipAePyS94QxejrFUqXY1/Gy99OMod/6ctIlMs1

      ZUhCIsu+nw5OlcFEt2qWue0vBKxNAlSwMoySR8mb66LDtCKxEXseO8wKQvqCyBWR

      bQIHEeLLpNuOxP0WoZhmMiZH3hxEHX6R8IftLM1G504zxbQz2GxaxlGBK/1qAdDL

      8doVn5i+t4A=

      -----END CERTIFICATE-----

      certificate_fingerprint: 78a91b226234ba4c72bcf5e2ac04a4bda752fcbe736739399024f50f377aadd0

      driver: lxc

      driver_version: 2.0.8

      kernel: Linux

      kernel_architecture: x86_64

      kernel_version: 4.4.0-89-generic

      server: lxd

      server_pid: 2523

      server_version: "2.16"

      storage: dir

      storage_version: "1"

Issue description

GPU not available in container after it has been added.

Steps to reproduce

>>> nvidia-smi
Tue Aug  8 23:23:31 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:82:00.0     Off |                  N/A |
|  0%   34C    P0    54W / 250W |      0MiB / 12207MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

>>> lxc exec c1 -- apt install cuda-demo-suite-8-0 --no-install-recommends
Reading package lists... Done
Building dependency tree       
Reading state information... Done
cuda-demo-suite-8-0 is already the newest version (8.0.61-1).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.

>>> lxc config device add c1 gpu gpu
Device gpu added to c1

>>> lxc exec c1 -- nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.
Bug
Source
picture gregsifr

Most helpful comment

Yes, adding a unix-char device for nvidia0, nvidiactl and nvidia-uvm should work fine, this is effectively what the gpu device type does for you.

only path= should be needed, LXD will then read the major/minor from the host for you.

picture stgraber on 9 Aug 2017
👍2

All 16 comments

Do you have the exact same version of the nvidia stuff both inside and outside the container?

stgraber picture stgraber on 8 Aug 2017

I did a demo of this very feature earlier today during a talk and things worked properly here.

stgraber picture stgraber on 8 Aug 2017

In my setup, I have the cuda packages at version 8.0.61-1 on the host and in the container. Nvidia kernel module version is 375.66.

stgraber picture stgraber on 8 Aug 2017

I used the same version:

>>>  sudo dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
(Reading database ... 146472 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu1604_8.0.61-1_amd64.deb ...
Unpacking cuda-repo-ubuntu1604 (8.0.61-1) over (8.0.61-1) ...
Setting up cuda-repo-ubuntu1604 (8.0.61-1) ...
OK

>>>  sudo apt update
Ign:1 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Hit:2 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release    
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]                                           
Hit:5 http://au.archive.ubuntu.com/ubuntu xenial InRelease                                  
Get:6 http://au.archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]               
Get:7 http://au.archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]                
Get:8 http://au.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [607 kB]        
Get:9 http://au.archive.ubuntu.com/ubuntu xenial-updates/main i386 Packages [586 kB]        
Get:10 http://au.archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [15.5 kB]
Get:11 http://au.archive.ubuntu.com/ubuntu xenial-updates/multiverse i386 Packages [14.6 kB]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [2,752 B]
Get:13 http://security.ubuntu.com/ubuntu xenial-security/multiverse i386 Packages [2,908 B]
Fetched 1,536 kB in 2s (532 kB/s)     
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
>>>  sudo apt install cuda
Reading package lists... Done
Building dependency tree       
Reading state information... Done
cuda is already the newest version (8.0.61-1).
0 to upgrade, 0 to newly install, 0 to remove and 0 not to upgrade.

>>>  nvidia-smi 
Wed Aug  9 10:37:36 2017       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:82:00.0     Off |                  N/A |
|  0%   33C    P0    54W / 250W |      0MiB / 12207MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

>>>  /usr/local/cuda-8.0/extras/demo_suite/bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce GTX TITAN X
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(MB/s)
   33554432         12109.2

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(MB/s)
   33554432         12365.1

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)    Bandwidth(MB/s)
   33554432         249540.0

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

>>>  lxc exec c1 -- dpkg -i cuda-repo-ubuntu1604_8.0.61-1_amd64.deb

(Reading database ... 46711 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu1604_8.0.61-1_amd64.deb ...
Unpacking cuda-repo-ubuntu1604 (8.0.61-1) over (8.0.61-1) ...
Setting up cuda-repo-ubuntu1604 (8.0.61-1) ...
OK
>>>  lxc exec c1 -- apt update

Ign:1 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  InRelease
Hit:2 http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64  Release                
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]                              
Hit:5 http://archive.ubuntu.com/ubuntu xenial InRelease                    
Get:6 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]           
Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]
Get:8 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [607 kB]
Fetched 914 kB in 5s (174 kB/s)   
Reading package lists... Done
Building dependency tree       
Reading state information... Done
3 packages can be upgraded. Run 'apt list --upgradable' to see them.
>>>  lxc exec c1 -- apt install cuda-demo-suite-8-0 --no-install-recommends

Reading package lists... Done
Building dependency tree       
Reading state information... Done
cuda-demo-suite-8-0 is already the newest version (8.0.61-1).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
>>>  lxc exec c1 -- nvidia-smi

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

>>>  lxc config device add c1 gpu gpu
error: The device already exists

After doing the above I noticed that cuda was not added to PATH on the host (nvcc --version did not work). The outcome was the same after adding cuda to PATH though.

I've also noticed that after the installation procedure the headless Ubuntu Server 16.04LTS machine now starts X on startup.

gregsifr picture gregsifr on 9 Aug 2017

Can you post "lxc config show --expanded c1" as well as "ls -lh /dev/nvidia*" from inside the container?

stgraber picture stgraber on 9 Aug 2017

From the host:

>>> lxc config show --expanded c1
architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.delay: "5"
  environment.http_proxy: ""
  user.network_mode: ""
  volatile.base_image: 5d5fe2d3263a8a6badb81fafe53635eb2887aff386cfc7e3d7bea9453a25e9cf
  volatile.eth0.hwaddr: 00:16:3e:a6:49:c1
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  backup:
    path: /mnt/backup
    source: /mnt/backup
    type: disk
  data-backup:
    path: /mnt/data-backup
    source: /mnt/data-backup
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  gpu:
    type: gpu
  root:
    path: /
    pool: default
    type: disk
  write-cashe:
    path: /mnt/write-cache
    source: /mnt/write-cache
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

From within the lxd container:

>>> ls -lh /dev/nvidia*
ls: cannot access '/dev/nvidia*': No such file or directory
gregsifr picture gregsifr on 9 Aug 2017

Interesting... Anything in /dev/dri in the container?

stgraber picture stgraber on 9 Aug 2017 
>>> ls /dev/dri
card0  controlD64  renderD128
gregsifr picture gregsifr on 9 Aug 2017

Can you post "ls -lh /dev/dri", "ls -lh /dev/nvidia*" and "lspci -nnn" from the host?

stgraber picture stgraber on 9 Aug 2017 
>>> ls -lh /dev/dri
total 0
crw-rw----+ 1 root video 226,   0 Aug  9 15:54 card0
crw-rw----+ 1 root video 226,   1 Aug  9 15:54 card1
crw-rw----  1 root video 226,  64 Aug  9 15:54 controlD64
crw-rw----+ 1 root video 226, 128 Aug  9 15:54 renderD128

>>> ls -lh /dev/nvidia*
crw-rw-rw- 1 root root 195,   0 Aug  9 15:54 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Aug  9 15:54 /dev/nvidiactl
crw-rw-rw- 1 root root 242,   0 Aug  9 15:54 /dev/nvidia-uvm

>>> lspci -nnn
00:00.0 Host bridge [0600]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMI2 [8086:2f00] (rev 02)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
00:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
00:03.2 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f0a] (rev 02)
00:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02)
00:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02)
00:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02)
00:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02)
00:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02)
00:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02)
00:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02)
00:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02)
00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02)
00:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
00:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02)
00:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR [8086:8d7c] (rev 05)
00:11.4 SATA controller [0106]: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] [8086:8d62] (rev 05)
00:14.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB xHCI Host Controller [8086:8d31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #1 [8086:8d3a] (rev 05)
00:16.1 Communication controller [0780]: Intel Corporation C610/X99 series chipset MEI Controller #2 [8086:8d3b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 [8086:8d2d] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 [8086:8d10] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 [8086:8d18] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 [8086:8d26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation C610/X99 series chipset LPC Controller [8086:8d44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] [8086:8d02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation C610/X99 series chipset SMBus Controller [8086:8d22] (rev 05)
03:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
03:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection [8086:1521] (rev 01)
05:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
06:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
7f:08.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f80] (rev 02)
7f:08.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f32] (rev 02)
7f:08.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f83] (rev 02)
7f:09.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f90] (rev 02)
7f:09.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f33] (rev 02)
7f:09.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f93] (rev 02)
7f:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
7f:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
7f:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
7f:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
7f:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
7f:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
7f:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
7f:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe4] (rev 02)
7f:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe5] (rev 02)
7f:0c.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe6] (rev 02)
7f:0c.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe7] (rev 02)
7f:0d.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe8] (rev 02)
7f:0d.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe9] (rev 02)
7f:0d.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fea] (rev 02)
7f:0d.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2feb] (rev 02)
7f:0d.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fec] (rev 02)
7f:0d.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fed] (rev 02)
7f:0d.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fee] (rev 02)
7f:0d.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fef] (rev 02)
7f:0e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2ff0] (rev 02)
7f:0e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2ff1] (rev 02)
7f:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
7f:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
7f:0f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ffa] (rev 02)
7f:0f.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ffb] (rev 02)
7f:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02)
7f:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02)
7f:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02)
7f:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
7f:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
7f:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
7f:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
7f:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
7f:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
7f:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
7f:12.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 [8086:2f60] (rev 02)
7f:12.5 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 [8086:2f38] (rev 02)
7f:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
7f:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
7f:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02)
7f:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02)
7f:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
7f:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
7f:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02)
7f:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02)
7f:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02)
7f:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02)
7f:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
7f:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
7f:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
7f:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
7f:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
7f:16.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
7f:16.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder [8086:2f6a] (rev 02)
7f:16.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder [8086:2f6b] (rev 02)
7f:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
7f:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
7f:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02)
7f:17.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control [8086:2fd1] (rev 02)
7f:17.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers [8086:2fd2] (rev 02)
7f:17.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers [8086:2fd3] (rev 02)
7f:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
7f:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
7f:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
7f:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)
7f:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
7f:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
7f:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
7f:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
7f:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
7f:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
7f:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
80:01.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 1 [8086:2f02] (rev 02)
80:03.0 PCI bridge [0604]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02)
80:04.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 0 [8086:2f20] (rev 02)
80:04.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 1 [8086:2f21] (rev 02)
80:04.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 2 [8086:2f22] (rev 02)
80:04.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 3 [8086:2f23] (rev 02)
80:04.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 4 [8086:2f24] (rev 02)
80:04.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 5 [8086:2f25] (rev 02)
80:04.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 6 [8086:2f26] (rev 02)
80:04.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DMA Channel 7 [8086:2f27] (rev 02)
80:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management [8086:2f28] (rev 02)
80:05.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Hot Plug [8086:2f29] (rev 02)
80:05.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 RAS, Control Status and Global Errors [8086:2f2a] (rev 02)
80:05.4 PIC [0800]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 I/O APIC [8086:2f2c] (rev 02)
81:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804]
82:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1)
82:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
ff:08.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f80] (rev 02)
ff:08.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f32] (rev 02)
ff:08.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 0 [8086:2f83] (rev 02)
ff:09.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f90] (rev 02)
ff:09.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f33] (rev 02)
ff:09.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 QPI Link 1 [8086:2f93] (rev 02)
ff:0b.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f81] (rev 02)
ff:0b.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f36] (rev 02)
ff:0b.2 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring [8086:2f37] (rev 02)
ff:0c.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe0] (rev 02)
ff:0c.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe1] (rev 02)
ff:0c.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe2] (rev 02)
ff:0c.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe3] (rev 02)
ff:0c.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe4] (rev 02)
ff:0c.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe5] (rev 02)
ff:0c.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe6] (rev 02)
ff:0c.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe7] (rev 02)
ff:0d.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe8] (rev 02)
ff:0d.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fe9] (rev 02)
ff:0d.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fea] (rev 02)
ff:0d.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2feb] (rev 02)
ff:0d.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fec] (rev 02)
ff:0d.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fed] (rev 02)
ff:0d.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fee] (rev 02)
ff:0d.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2fef] (rev 02)
ff:0e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2ff0] (rev 02)
ff:0e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Unicast Registers [8086:2ff1] (rev 02)
ff:0f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff8] (rev 02)
ff:0f.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ff9] (rev 02)
ff:0f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ffa] (rev 02)
ff:0f.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Buffered Ring Agent [8086:2ffb] (rev 02)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffc] (rev 02)
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffd] (rev 02)
ff:0f.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers [8086:2ffe] (rev 02)
ff:10.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f1d] (rev 02)
ff:10.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 PCIe Ring Interface [8086:2f34] (rev 02)
ff:10.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1e] (rev 02)
ff:10.6 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f7d] (rev 02)
ff:10.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers [8086:2f1f] (rev 02)
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2fa0] (rev 02)
ff:12.1 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 0 [8086:2f30] (rev 02)
ff:12.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 [8086:2f60] (rev 02)
ff:12.5 Performance counters [1101]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Home Agent 1 [8086:2f38] (rev 02)
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2faa] (rev 02)
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder [8086:2fab] (rev 02)
ff:13.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast [8086:2fae] (rev 02)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2faf] (rev 02)
ff:14.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control [8086:2fb0] (rev 02)
ff:14.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control [8086:2fb1] (rev 02)
ff:14.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers [8086:2fb2] (rev 02)
ff:14.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers [8086:2fb3] (rev 02)
ff:14.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbc] (rev 02)
ff:14.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbd] (rev 02)
ff:14.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbe] (rev 02)
ff:14.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 [8086:2fbf] (rev 02)
ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:16.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers [8086... (rev 02)
ff:16.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder [8086:2f6a] (rev 02)
ff:16.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel Target Address Decoder [8086:2f6b] (rev 02)
ff:16.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast [8086:2f6e] (rev 02)
ff:16.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO Global Broadcast [8086:2f6f] (rev 02)
ff:17.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control [8086:2fd0] (rev 02)
ff:17.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 Thermal Control [8086:2fd1] (rev 02)
ff:17.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 ERROR Registers [8086:2fd2] (rev 02)
ff:17.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 1 ERROR Registers [8086:2fd3] (rev 02)
ff:17.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb8] (rev 02)
ff:17.5 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fb9] (rev 02)
ff:17.6 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fba] (rev 02)
ff:17.7 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 [8086:2fbb] (rev 02)
ff:1e.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f98] (rev 02)
ff:1e.1 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f99] (rev 02)
ff:1e.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9a] (rev 02)
ff:1e.3 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2fc0] (rev 02)
ff:1e.4 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 Power Control Unit [8086:2f9c] (rev 02)
ff:1f.0 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f88] (rev 02)
ff:1f.2 System peripheral [0880]: Intel Corporation Xeon E7 v3/Xeon E5 v3/Core i7 VCU [8086:2f8a] (rev 02)
gregsifr picture gregsifr on 9 Aug 2017

Interesting, so it's as if LXD just ignored the nvidia card entirely and only passed your BMC's GPU (aspeed) to the container.

Can you try:

lxc config device remove c1 gpu
lxc config device add c1 gpu gpu id=1

See if LXD behaves differently when directly asked for the second GPU.

stgraber picture stgraber on 9 Aug 2017

lxc exec c1 -- nvidia-smi continues to fail. I rerun the previous commands:

>>> lxc exec c1 -- nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

>>> lxc config show --expanded c1
architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.delay: "5"
  environment.http_proxy: ""
  user.network_mode: ""
  volatile.base_image: 5d5fe2d3263a8a6badb81fafe53635eb2887aff386cfc7e3d7bea9453a25e9cf
  volatile.eth0.hwaddr: 00:16:3e:a6:49:c1
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
  volatile.last_state.power: RUNNING
devices:
  backup:
    path: /mnt/backup
    source: /mnt/backup
    type: disk
  data-backup:
    path: /mnt/data-backup
    source: /mnt/data-backup
    type: disk
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  gpu:
    id: "1"
    type: gpu
  root:
    path: /
    pool: default
    type: disk
  write-cashe:
    path: /mnt/write-cache
    source: /mnt/write-cache
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

From within the container:

>>> ls /dev/dri
renderD128

>>> ls -lh /dev/nvidia*
ls: cannot access '/dev/nvidia*': No such file or directory
gregsifr picture gregsifr on 9 Aug 2017

Ok, I think I managed to reproduce the problem here when adding a third GPU which is non-nvidia to our test system. It's not entirely obvious to me what's causing the problem here though and the obvious workarounds of forcing things with a product/vendor id or pci id doesn't seem to work either.

@brauner Can you take a look at this? I've got the canonical-lxd system setup to reproduce the issue.

stgraber picture stgraber on 9 Aug 2017

Could the device be added as unix-char device in the meantime?

Something along the lines of: lxc config device add c1 gpu unix-char path=/dev/nvidia0 for each /dev/nvidia0, /dev/nvidiactl, /dev/nvidia-uvm. Would the major need to be added also?

gregsifr picture gregsifr on 9 Aug 2017

Yes, adding a unix-char device for nvidia0, nvidiactl and nvidia-uvm should work fine, this is effectively what the gpu device type does for you.

only path= should be needed, LXD will then read the major/minor from the host for you.

stgraber picture stgraber on 9 Aug 2017
👍2

I am have the same problem, Radeon card0 nvidia card1, the lxd does not have /dev/nvidia0 so I tried to do the unix-char trick but I get "The device already exists" but I do not see it in /dev of container

greenpdx picture greenpdx on 30 Mar 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

srkunze picture srkunze  路  3Comments
fwaggle picture fwaggle  路  4Comments
rrva picture rrva  路  5Comments
AndreiPashkin picture AndreiPashkin  路  5Comments
simos picture simos  路  3Comments