Salt: Helium salt-server 100% CPU usage and never recovers

[getabc]

Firstly the ubuntu salt-master environment,

root@automate:~# salt-master --versions-report
           Salt: 2014.7.0
         Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.3.0
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.0.1
           RAET: Not Installed
            ZMQ: 4.0.4
           Mako: 0.9.1

/etc/salt/master

interface: 10.156.47.9
max_open_files: 8192
worker_threads: 20
timeout: 30 
open_mode: True
auto_accept: True

file_roots:                                                                                         
  base:                                                                                             
    - /srv/salt/base 

fileserver_backend:                                                                                 
  - git                                                                                             
  - roots   

pillar_roots:                                                                                       
  base:                                                                                             
    - /srv/pillar

The Windows minion are configured with 2014.7.0

ipc_mode: tcp
master: automate.gs.au.adshel.net
retry_dns: 30
master_port: 4506

root_dir: c:\salt
pki_dir: /conf/pki/minion

multiprocessing: False
win_repo_cachefile: 'salt://win/repo/winrepo.p'

#Autentication
random_reauth_delay: 270
recon_default: 1000
recon_max: 30000
recon_randomize: True
auth_timeout: 60

At this point the salt-master server is running normally with 130 minions connected.

Command issued

root@automate:/etc/salt# salt '*st*' test.ping

Data returned as normal, but one process is stuck at 100%.

root@automate:/etc/salt# ps -aux | grep salt-master
root     13609  0.0  1.2 143532 26504 pts/0    S+   10:12   0:00 /usr/bin/python /usr/bin/salt-master -l debug
root     13616  0.7  2.9 220172 59908 pts/0    S+   10:12   0:06 /usr/bin/python /usr/bin/salt-master -l debug
root     13617  0.0  1.3 225460 27488 pts/0    Sl+  10:12   0:00 /usr/bin/python /usr/bin/salt-master -l debug
root     13620  0.0  1.1 225460 23720 pts/0    Sl+  10:12   0:00 /usr/bin/python /usr/bin/salt-master -l debug
root     13623  0.0  1.1 143532 22568 pts/0    S+   10:12   0:00 /usr/bin/python /usr/bin/salt-master -l debug
root     13624  0.3  2.2 455408 46660 pts/0    Sl+  10:12   0:02 /usr/bin/python /usr/bin/salt-master -l debug
root     13625  0.3  2.2 455408 46668 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13626  0.3  2.2 455404 46668 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13627  0.3  2.2 455412 46600 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13628  0.3  2.2 455408 46676 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13629  0.3  2.2 455412 46680 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13630  0.3  2.2 455420 46688 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13631  0.3  2.2 455420 46688 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13632  0.3  2.2 520948 46736 pts/0    Sl+  10:12   0:02 /usr/bin/python /usr/bin/salt-master -l debug
root     13633  0.3  2.2 455420 46684 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13634  0.3  2.2 455416 46664 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13635  0.3  2.2 455416 46680 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13637  0.3  2.2 455420 46692 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13641  0.3  2.2 455420 46688 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13642  0.3  2.2 455424 46676 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13643  0.3  2.2 455424 46692 pts/0    Sl+  10:12   0:02 /usr/bin/python /usr/bin/salt-master -l debug
root     13646  0.3  2.2 455424 46700 pts/0    Sl+  10:12   0:02 /usr/bin/python /usr/bin/salt-master -l debug
root     13649  0.3  2.2 455424 46688 pts/0    Sl+  10:12   0:02 /usr/bin/python /usr/bin/salt-master -l debug
root     13652  0.3  2.2 455424 46676 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13657  0.3  2.2 455432 46700 pts/0    Sl+  10:12   0:03 /usr/bin/python /usr/bin/salt-master -l debug
root     13666 68.3  7.7 852956 159332 pts/0   Sl+  10:12  10:12 /usr/bin/python /usr/bin/salt-master -l debug

No file activity found when I strace the process

root@automate:/etc/salt# sudo strace -c -p 13666
Process 13666 attached
<no result>

The salt-server does not respond to any requests until I restart the salt-master process.

[getabc]

Removed the pillar_roots in the master file and everything works perfectly.

pillar_roots:                                                                                       
  base:                                                                                             
    - /srv/pillar

I haven't setup any pillar configuration files.

[getabc]

after a few salt commands the 100% CPU salt-master process is back

[getabc]

added two more CPU's to the virtual machine and everything back to normal. Haven't enabled pillar_roots again.

[pitatus]

what did your salt-master VM environment look like before adding the CPU's? (memory and number of CPU's) did you see anything in the system logs at the time to help identify what might have been going on? (/var/log/kern.log, /var/log/syslog)

[getabc]

root@automate:~# uname -a
Linux automate.gs.au.adshel.net 3.13.0-40-generic #69-Ubuntu SMP Thu Nov 13 17:53:56 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

root@automate:~# head -n1 /etc/issue
Ubuntu 14.04.1 LTS \n \l

root@automate:~# grep MemTotal /proc/meminfo
MemTotal:        2046380 kB

root@automate:~# grep "model name" /proc/cpuinfo
model name      : Six-Core AMD Opteron(tm) Processor 2435
model name      : Six-Core AMD Opteron(tm) Processor 2435
model name      : Six-Core AMD Opteron(tm) Processor 2435

The original VM contained only one CPU, now upgraded to three and system has become stable.

This morning I re-enabled the pillar_roots setting and had a few minutes at 100% spike, but after a few salt-master restarts the server is back to normal.

I noticed the following in my syslog,

Dec  5 22:49:06 u14-04 kernel: [    0.731261] pci 0000:00:15.3: bridge window [io  0x1000-0x0fff] to [bus 06] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731323] pci 0000:00:15.4: bridge window [io  0x1000-0x0fff] to [bus 07] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731384] pci 0000:00:15.5: bridge window [io  0x1000-0x0fff] to [bus 08] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731444] pci 0000:00:15.6: bridge window [io  0x1000-0x0fff] to [bus 09] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731505] pci 0000:00:15.7: bridge window [io  0x1000-0x0fff] to [bus 0a] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731736] pci 0000:00:16.3: bridge window [io  0x1000-0x0fff] to [bus 0e] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731796] pci 0000:00:16.4: bridge window [io  0x1000-0x0fff] to [bus 0f] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731857] pci 0000:00:16.5: bridge window [io  0x1000-0x0fff] to [bus 10] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731917] pci 0000:00:16.6: bridge window [io  0x1000-0x0fff] to [bus 11] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.731977] pci 0000:00:16.7: bridge window [io  0x1000-0x0fff] to [bus 12] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732211] pci 0000:00:17.3: bridge window [io  0x1000-0x0fff] to [bus 16] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732272] pci 0000:00:17.4: bridge window [io  0x1000-0x0fff] to [bus 17] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732332] pci 0000:00:17.5: bridge window [io  0x1000-0x0fff] to [bus 18] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732392] pci 0000:00:17.6: bridge window [io  0x1000-0x0fff] to [bus 19] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732452] pci 0000:00:17.7: bridge window [io  0x1000-0x0fff] to [bus 1a] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732626] pci 0000:00:18.2: bridge window [io  0x1000-0x0fff] to [bus 1d] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732686] pci 0000:00:18.3: bridge window [io  0x1000-0x0fff] to [bus 1e] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732748] pci 0000:00:18.4: bridge window [io  0x1000-0x0fff] to [bus 1f] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732808] pci 0000:00:18.5: bridge window [io  0x1000-0x0fff] to [bus 20] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732868] pci 0000:00:18.6: bridge window [io  0x1000-0x0fff] to [bus 21] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732929] pci 0000:00:18.7: bridge window [io  0x1000-0x0fff] to [bus 22] add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732943] pci 0000:00:15.3: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732946] pci 0000:00:15.4: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732948] pci 0000:00:15.5: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732950] pci 0000:00:15.6: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732951] pci 0000:00:15.7: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732953] pci 0000:00:16.3: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732955] pci 0000:00:16.4: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732957] pci 0000:00:16.5: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732959] pci 0000:00:16.6: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732961] pci 0000:00:16.7: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732963] pci 0000:00:17.3: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732965] pci 0000:00:17.4: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732967] pci 0000:00:17.5: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732969] pci 0000:00:17.6: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732971] pci 0000:00:17.7: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732972] pci 0000:00:18.2: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732974] pci 0000:00:18.3: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732976] pci 0000:00:18.4: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732978] pci 0000:00:18.5: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732980] pci 0000:00:18.6: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732982] pci 0000:00:18.7: res[13]=[io  0x1000-0x0fff] get_res_add_size add_size 1000
Dec  5 22:49:06 u14-04 kernel: [    0.732988] pci 0000:00:0f.0: BAR 6: assigned [mem 0x80000000-0x80007fff pref]
Dec  5 22:49:06 u14-04 kernel: [    0.733367] pci 0000:00:10.0: BAR 6: assigned [mem 0x80008000-0x8000bfff pref]
Dec  5 22:49:06 u14-04 kernel: [    0.733685] pci 0000:00:15.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.733869] pci 0000:00:15.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734051] pci 0000:00:15.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734233] pci 0000:00:15.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734414] pci 0000:00:15.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734596] pci 0000:00:16.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734777] pci 0000:00:16.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.734959] pci 0000:00:16.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.735140] pci 0000:00:16.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.735321] pci 0000:00:16.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.735502] pci 0000:00:17.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.735684] pci 0000:00:17.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.735865] pci 0000:00:17.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736046] pci 0000:00:17.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736227] pci 0000:00:17.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736408] pci 0000:00:18.2: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736590] pci 0000:00:18.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736771] pci 0000:00:18.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.736952] pci 0000:00:18.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.737137] pci 0000:00:18.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.737319] pci 0000:00:18.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.737847] pci 0000:00:18.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738029] pci 0000:00:18.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738211] pci 0000:00:18.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738392] pci 0000:00:18.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738574] pci 0000:00:18.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738756] pci 0000:00:18.2: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.738937] pci 0000:00:17.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.739118] pci 0000:00:17.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.739300] pci 0000:00:17.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.739482] pci 0000:00:17.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.739664] pci 0000:00:17.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.739845] pci 0000:00:16.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.740026] pci 0000:00:16.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.740207] pci 0000:00:16.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.746124] pci 0000:00:16.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.746310] pci 0000:00:16.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.746493] pci 0000:00:15.7: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.746674] pci 0000:00:15.6: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.746855] pci 0000:00:15.5: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.747036] pci 0000:00:15.4: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.747216] pci 0000:00:15.3: BAR 13: can't assign io (size 0x1000)
Dec  5 22:49:06 u14-04 kernel: [    0.747399] pci 0000:00:01.0: PCI bridge to [bus 01]
Dec  5 22:49:06 u14-04 kernel: [    0.747660] pci 0000:00:11.0: PCI bridge to [bus 02]
Dec  5 22:49:06 u14-04 kernel: [    0.747838] pci 0000:00:11.0:   bridge window [io  0x2000-0x3fff]
Dec  5 22:49:06 u14-04 kernel: [    0.748047] pci 0000:00:11.0:   bridge window [mem 0xd1900000-0xd23fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.748253] pci 0000:00:11.0:   bridge window [mem 0xdc400000-0xdc9fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.748572] pci 0000:03:00.0: BAR 6: assigned [mem 0xd4400000-0xd440ffff pref]
Dec  5 22:49:06 u14-04 kernel: [    0.748845] pci 0000:00:15.0: PCI bridge to [bus 03]
Dec  5 22:49:06 u14-04 kernel: [    0.749021] pci 0000:00:15.0:   bridge window [io  0x4000-0x4fff]
Dec  5 22:49:06 u14-04 kernel: [    0.749229] pci 0000:00:15.0:   bridge window [mem 0xd2400000-0xd24fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.749436] pci 0000:00:15.0:   bridge window [mem 0xd4400000-0xd44fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.749846] pci 0000:00:15.1: PCI bridge to [bus 04]
Dec  5 22:49:06 u14-04 kernel: [    0.750030] pci 0000:00:15.1:   bridge window [io  0x8000-0x8fff]
Dec  5 22:49:06 u14-04 kernel: [    0.750239] pci 0000:00:15.1:   bridge window [mem 0xd2800000-0xd28fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.750510] pci 0000:00:15.1:   bridge window [mem 0xd4800000-0xd48fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.750917] pci 0000:00:15.2: PCI bridge to [bus 05]
Dec  5 22:49:06 u14-04 kernel: [    0.751138] pci 0000:00:15.2:   bridge window [io  0xc000-0xcfff]
Dec  5 22:49:06 u14-04 kernel: [    0.751397] pci 0000:00:15.2:   bridge window [mem 0xd2c00000-0xd2cfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.751655] pci 0000:00:15.2:   bridge window [mem 0xdcb00000-0xdcbfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.752060] pci 0000:00:15.3: PCI bridge to [bus 06]
Dec  5 22:49:06 u14-04 kernel: [    0.752303] pci 0000:00:15.3:   bridge window [mem 0xd3000000-0xd30fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.752562] pci 0000:00:15.3:   bridge window [mem 0xdcd00000-0xdcdfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.752964] pci 0000:00:15.4: PCI bridge to [bus 07]
Dec  5 22:49:06 u14-04 kernel: [    0.758877] pci 0000:00:15.4:   bridge window [mem 0xd3400000-0xd34fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.759138] pci 0000:00:15.4:   bridge window [mem 0xdcf00000-0xdcffffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.759559] pci 0000:00:15.5: PCI bridge to [bus 08]
Dec  5 22:49:06 u14-04 kernel: [    0.759802] pci 0000:00:15.5:   bridge window [mem 0xd3800000-0xd38fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.760058] pci 0000:00:15.5:   bridge window [mem 0xdd100000-0xdd1fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.760534] pci 0000:00:15.6: PCI bridge to [bus 09]
Dec  5 22:49:06 u14-04 kernel: [    0.760834] pci 0000:00:15.6:   bridge window [mem 0xd3c00000-0xd3cfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.761153] pci 0000:00:15.6:   bridge window [mem 0xdd300000-0xdd3fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.761687] pci 0000:00:15.7: PCI bridge to [bus 0a]
Dec  5 22:49:06 u14-04 kernel: [    0.761996] pci 0000:00:15.7:   bridge window [mem 0xd4000000-0xd40fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.762323] pci 0000:00:15.7:   bridge window [mem 0xdd500000-0xdd5fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.762840] pci 0000:00:16.0: PCI bridge to [bus 0b]
Dec  5 22:49:06 u14-04 kernel: [    0.763119] pci 0000:00:16.0:   bridge window [io  0x5000-0x5fff]
Dec  5 22:49:06 u14-04 kernel: [    0.763452] pci 0000:00:16.0:   bridge window [mem 0xd2500000-0xd25fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.763779] pci 0000:00:16.0:   bridge window [mem 0xd4500000-0xd45fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.764288] pci 0000:00:16.1: PCI bridge to [bus 0c]
Dec  5 22:49:06 u14-04 kernel: [    0.764566] pci 0000:00:16.1:   bridge window [io  0x9000-0x9fff]
Dec  5 22:49:06 u14-04 kernel: [    0.764905] pci 0000:00:16.1:   bridge window [mem 0xd2900000-0xd29fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.770374] pci 0000:00:16.1:   bridge window [mem 0xd4900000-0xd49fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.770907] pci 0000:00:16.2: PCI bridge to [bus 0d]
Dec  5 22:49:06 u14-04 kernel: [    0.771184] pci 0000:00:16.2:   bridge window [io  0xd000-0xdfff]
Dec  5 22:49:06 u14-04 kernel: [    0.771502] pci 0000:00:16.2:   bridge window [mem 0xd2d00000-0xd2dfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.771817] pci 0000:00:16.2:   bridge window [mem 0xd4b00000-0xd4bfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.772315] pci 0000:00:16.3: PCI bridge to [bus 0e]
Dec  5 22:49:06 u14-04 kernel: [    0.772620] pci 0000:00:16.3:   bridge window [mem 0xd3100000-0xd31fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.772937] pci 0000:00:16.3:   bridge window [mem 0xd4d00000-0xd4dfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.773433] pci 0000:00:16.4: PCI bridge to [bus 0f]
Dec  5 22:49:06 u14-04 kernel: [    0.773778] pci 0000:00:16.4:   bridge window [mem 0xd3500000-0xd35fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.774096] pci 0000:00:16.4:   bridge window [mem 0xd4f00000-0xd4ffffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.774593] pci 0000:00:16.5: PCI bridge to [bus 10]
Dec  5 22:49:06 u14-04 kernel: [    0.774894] pci 0000:00:16.5:   bridge window [mem 0xd3900000-0xd39fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.775210] pci 0000:00:16.5:   bridge window [mem 0xd5100000-0xd51fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.775708] pci 0000:00:16.6: PCI bridge to [bus 11]
Dec  5 22:49:06 u14-04 kernel: [    0.776008] pci 0000:00:16.6:   bridge window [mem 0xd3d00000-0xd3dfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.776324] pci 0000:00:16.6:   bridge window [mem 0xd5300000-0xd53fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.776821] pci 0000:00:16.7: PCI bridge to [bus 12]
Dec  5 22:49:06 u14-04 kernel: [    0.777122] pci 0000:00:16.7:   bridge window [mem 0xd4100000-0xd41fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.777440] pci 0000:00:16.7:   bridge window [mem 0xd5500000-0xd55fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.778002] pci 0000:00:17.0: PCI bridge to [bus 13]
Dec  5 22:49:06 u14-04 kernel: [    0.783042] pci 0000:00:17.0:   bridge window [io  0x6000-0x6fff]
Dec  5 22:49:06 u14-04 kernel: [    0.783253] pci 0000:00:17.0:   bridge window [mem 0xd2600000-0xd26fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.783466] pci 0000:00:17.0:   bridge window [mem 0xd4600000-0xd46fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.783807] pci 0000:00:17.1: PCI bridge to [bus 14]
Dec  5 22:49:06 u14-04 kernel: [    0.783984] pci 0000:00:17.1:   bridge window [io  0xa000-0xafff]
Dec  5 22:49:06 u14-04 kernel: [    0.784192] pci 0000:00:17.1:   bridge window [mem 0xd2a00000-0xd2afffff]
Dec  5 22:49:06 u14-04 kernel: [    0.784398] pci 0000:00:17.1:   bridge window [mem 0xdca00000-0xdcafffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.784723] pci 0000:00:17.2: PCI bridge to [bus 15]
Dec  5 22:49:06 u14-04 kernel: [    0.784900] pci 0000:00:17.2:   bridge window [io  0xe000-0xefff]
Dec  5 22:49:06 u14-04 kernel: [    0.785108] pci 0000:00:17.2:   bridge window [mem 0xd2e00000-0xd2efffff]
Dec  5 22:49:06 u14-04 kernel: [    0.785315] pci 0000:00:17.2:   bridge window [mem 0xdcc00000-0xdccfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.785663] pci 0000:00:17.3: PCI bridge to [bus 16]
Dec  5 22:49:06 u14-04 kernel: [    0.785860] pci 0000:00:17.3:   bridge window [mem 0xd3200000-0xd32fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.786069] pci 0000:00:17.3:   bridge window [mem 0xdce00000-0xdcefffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.786403] pci 0000:00:17.4: PCI bridge to [bus 17]
Dec  5 22:49:06 u14-04 kernel: [    0.786600] pci 0000:00:17.4:   bridge window [mem 0xd3600000-0xd36fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.786807] pci 0000:00:17.4:   bridge window [mem 0xdd000000-0xdd0fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.787135] pci 0000:00:17.5: PCI bridge to [bus 18]
Dec  5 22:49:06 u14-04 kernel: [    0.787332] pci 0000:00:17.5:   bridge window [mem 0xd3a00000-0xd3afffff]
Dec  5 22:49:06 u14-04 kernel: [    0.787539] pci 0000:00:17.5:   bridge window [mem 0xdd200000-0xdd2fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.787867] pci 0000:00:17.6: PCI bridge to [bus 19]
Dec  5 22:49:06 u14-04 kernel: [    0.788067] pci 0000:00:17.6:   bridge window [mem 0xd3e00000-0xd3efffff]
Dec  5 22:49:06 u14-04 kernel: [    0.788275] pci 0000:00:17.6:   bridge window [mem 0xdd400000-0xdd4fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.788602] pci 0000:00:17.7: PCI bridge to [bus 1a]
Dec  5 22:49:06 u14-04 kernel: [    0.788799] pci 0000:00:17.7:   bridge window [mem 0xd4200000-0xd42fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.789007] pci 0000:00:17.7:   bridge window [mem 0xdd600000-0xdd6fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.789331] pci 0000:00:18.0: PCI bridge to [bus 1b]
Dec  5 22:49:06 u14-04 kernel: [    0.789509] pci 0000:00:18.0:   bridge window [io  0x7000-0x7fff]
Dec  5 22:49:06 u14-04 kernel: [    0.789752] pci 0000:00:18.0:   bridge window [mem 0xd2700000-0xd27fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.789960] pci 0000:00:18.0:   bridge window [mem 0xd4700000-0xd47fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.790288] pci 0000:00:18.1: PCI bridge to [bus 1c]
Dec  5 22:49:06 u14-04 kernel: [    0.801670] pci 0000:00:18.1:   bridge window [io  0xb000-0xbfff]
Dec  5 22:49:06 u14-04 kernel: [    0.802359] pci 0000:00:18.1:   bridge window [mem 0xd2b00000-0xd2bfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.803032] pci 0000:00:18.1:   bridge window [mem 0xd4a00000-0xd4afffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.807665] pci 0000:00:18.2: PCI bridge to [bus 1d]
Dec  5 22:49:06 u14-04 kernel: [    0.808311] pci 0000:00:18.2:   bridge window [mem 0xd2f00000-0xd2ffffff]
Dec  5 22:49:06 u14-04 kernel: [    0.808985] pci 0000:00:18.2:   bridge window [mem 0xd4c00000-0xd4cfffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.810140] pci 0000:00:18.3: PCI bridge to [bus 1e]
Dec  5 22:49:06 u14-04 kernel: [    0.810532] pci 0000:00:18.3:   bridge window [mem 0xd3300000-0xd33fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.810740] pci 0000:00:18.3:   bridge window [mem 0xd4e00000-0xd4efffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.811071] pci 0000:00:18.4: PCI bridge to [bus 1f]
Dec  5 22:49:06 u14-04 kernel: [    0.811268] pci 0000:00:18.4:   bridge window [mem 0xd3700000-0xd37fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.811475] pci 0000:00:18.4:   bridge window [mem 0xd5000000-0xd50fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.811801] pci 0000:00:18.5: PCI bridge to [bus 20]
Dec  5 22:49:06 u14-04 kernel: [    0.811999] pci 0000:00:18.5:   bridge window [mem 0xd3b00000-0xd3bfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.812205] pci 0000:00:18.5:   bridge window [mem 0xd5200000-0xd52fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.812531] pci 0000:00:18.6: PCI bridge to [bus 21]
Dec  5 22:49:06 u14-04 kernel: [    0.812728] pci 0000:00:18.6:   bridge window [mem 0xd3f00000-0xd3ffffff]
Dec  5 22:49:06 u14-04 kernel: [    0.812936] pci 0000:00:18.6:   bridge window [mem 0xd5400000-0xd54fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.813262] pci 0000:00:18.7: PCI bridge to [bus 22]
Dec  5 22:49:06 u14-04 kernel: [    0.813459] pci 0000:00:18.7:   bridge window [mem 0xd4300000-0xd43fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.813693] pci 0000:00:18.7:   bridge window [mem 0xd5600000-0xd56fffff 64bit pref]Dec  5 22:49:06 u14-04 kernel: [    0.814033] pci_bus 0000:00: resource 4 [mem 0x000a0000-0x000bffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814035] pci_bus 0000:00: resource 5 [mem 0x000cc000-0x000cffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814037] pci_bus 0000:00: resource 6 [mem 0x000d0000-0x000d3fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814038] pci_bus 0000:00: resource 7 [mem 0x000d4000-0x000d7fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814040] pci_bus 0000:00: resource 8 [mem 0x000d8000-0x000dbfff]
Dec  5 22:49:06 u14-04 kernel: [    0.814042] pci_bus 0000:00: resource 9 [mem 0x80000000-0xfebfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814043] pci_bus 0000:00: resource 10 [io  0x0000-0x0cf7]
Dec  5 22:49:06 u14-04 kernel: [    0.814045] pci_bus 0000:00: resource 11 [io  0x0d00-0xfeff]
Dec  5 22:49:06 u14-04 kernel: [    0.814049] pci_bus 0000:02: resource 0 [io  0x2000-0x3fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814051] pci_bus 0000:02: resource 1 [mem 0xd1900000-0xd23fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814053] pci_bus 0000:02: resource 2 [mem 0xdc400000-0xdc9fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814054] pci_bus 0000:02: resource 4 [mem 0x000a0000-0x000bffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814056] pci_bus 0000:02: resource 5 [mem 0x000cc000-0x000cffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814058] pci_bus 0000:02: resource 6 [mem 0x000d0000-0x000d3fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814059] pci_bus 0000:02: resource 7 [mem 0x000d4000-0x000d7fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814061] pci_bus 0000:02: resource 8 [mem 0x000d8000-0x000dbfff]
Dec  5 22:49:06 u14-04 kernel: [    0.814063] pci_bus 0000:02: resource 9 [mem 0x80000000-0xfebfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814064] pci_bus 0000:02: resource 10 [io  0x0000-0x0cf7]
Dec  5 22:49:06 u14-04 kernel: [    0.814066] pci_bus 0000:02: resource 11 [io  0x0d00-0xfeff]
Dec  5 22:49:06 u14-04 kernel: [    0.814068] pci_bus 0000:03: resource 0 [io  0x4000-0x4fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814070] pci_bus 0000:03: resource 1 [mem 0xd2400000-0xd24fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814071] pci_bus 0000:03: resource 2 [mem 0xd4400000-0xd44fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814074] pci_bus 0000:04: resource 0 [io  0x8000-0x8fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814075] pci_bus 0000:04: resource 1 [mem 0xd2800000-0xd28fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814077] pci_bus 0000:04: resource 2 [mem 0xd4800000-0xd48fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814079] pci_bus 0000:05: resource 0 [io  0xc000-0xcfff]
Dec  5 22:49:06 u14-04 kernel: [    0.814081] pci_bus 0000:05: resource 1 [mem 0xd2c00000-0xd2cfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814083] pci_bus 0000:05: resource 2 [mem 0xdcb00000-0xdcbfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814085] pci_bus 0000:06: resource 1 [mem 0xd3000000-0xd30fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814087] pci_bus 0000:06: resource 2 [mem 0xdcd00000-0xdcdfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814089] pci_bus 0000:07: resource 1 [mem 0xd3400000-0xd34fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814091] pci_bus 0000:07: resource 2 [mem 0xdcf00000-0xdcffffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814093] pci_bus 0000:08: resource 1 [mem 0xd3800000-0xd38fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814095] pci_bus 0000:08: resource 2 [mem 0xdd100000-0xdd1fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814097] pci_bus 0000:09: resource 1 [mem 0xd3c00000-0xd3cfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814098] pci_bus 0000:09: resource 2 [mem 0xdd300000-0xdd3fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814101] pci_bus 0000:0a: resource 1 [mem 0xd4000000-0xd40fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814103] pci_bus 0000:0a: resource 2 [mem 0xdd500000-0xdd5fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814105] pci_bus 0000:0b: resource 0 [io  0x5000-0x5fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814107] pci_bus 0000:0b: resource 1 [mem 0xd2500000-0xd25fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814108] pci_bus 0000:0b: resource 2 [mem 0xd4500000-0xd45fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814110] pci_bus 0000:0c: resource 0 [io  0x9000-0x9fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814112] pci_bus 0000:0c: resource 1 [mem 0xd2900000-0xd29fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814114] pci_bus 0000:0c: resource 2 [mem 0xd4900000-0xd49fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814116] pci_bus 0000:0d: resource 0 [io  0xd000-0xdfff]
Dec  5 22:49:06 u14-04 kernel: [    0.814118] pci_bus 0000:0d: resource 1 [mem 0xd2d00000-0xd2dfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814119] pci_bus 0000:0d: resource 2 [mem 0xd4b00000-0xd4bfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814121] pci_bus 0000:0e: resource 1 [mem 0xd3100000-0xd31fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814123] pci_bus 0000:0e: resource 2 [mem 0xd4d00000-0xd4dfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814125] pci_bus 0000:0f: resource 1 [mem 0xd3500000-0xd35fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814127] pci_bus 0000:0f: resource 2 [mem 0xd4f00000-0xd4ffffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814129] pci_bus 0000:10: resource 1 [mem 0xd3900000-0xd39fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814131] pci_bus 0000:10: resource 2 [mem 0xd5100000-0xd51fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814133] pci_bus 0000:11: resource 1 [mem 0xd3d00000-0xd3dfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814134] pci_bus 0000:11: resource 2 [mem 0xd5300000-0xd53fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814136] pci_bus 0000:12: resource 1 [mem 0xd4100000-0xd41fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814138] pci_bus 0000:12: resource 2 [mem 0xd5500000-0xd55fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814140] pci_bus 0000:13: resource 0 [io  0x6000-0x6fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814142] pci_bus 0000:13: resource 1 [mem 0xd2600000-0xd26fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814143] pci_bus 0000:13: resource 2 [mem 0xd4600000-0xd46fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814145] pci_bus 0000:14: resource 0 [io  0xa000-0xafff]
Dec  5 22:49:06 u14-04 kernel: [    0.814147] pci_bus 0000:14: resource 1 [mem 0xd2a00000-0xd2afffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814149] pci_bus 0000:14: resource 2 [mem 0xdca00000-0xdcafffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814151] pci_bus 0000:15: resource 0 [io  0xe000-0xefff]
Dec  5 22:49:06 u14-04 kernel: [    0.814152] pci_bus 0000:15: resource 1 [mem 0xd2e00000-0xd2efffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814154] pci_bus 0000:15: resource 2 [mem 0xdcc00000-0xdccfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814156] pci_bus 0000:16: resource 1 [mem 0xd3200000-0xd32fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814158] pci_bus 0000:16: resource 2 [mem 0xdce00000-0xdcefffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814160] pci_bus 0000:17: resource 1 [mem 0xd3600000-0xd36fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814162] pci_bus 0000:17: resource 2 [mem 0xdd000000-0xdd0fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814168] pci_bus 0000:18: resource 1 [mem 0xd3a00000-0xd3afffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814170] pci_bus 0000:18: resource 2 [mem 0xdd200000-0xdd2fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814172] pci_bus 0000:19: resource 1 [mem 0xd3e00000-0xd3efffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814174] pci_bus 0000:19: resource 2 [mem 0xdd400000-0xdd4fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814176] pci_bus 0000:1a: resource 1 [mem 0xd4200000-0xd42fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814178] pci_bus 0000:1a: resource 2 [mem 0xdd600000-0xdd6fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814180] pci_bus 0000:1b: resource 0 [io  0x7000-0x7fff]
Dec  5 22:49:06 u14-04 kernel: [    0.814181] pci_bus 0000:1b: resource 1 [mem 0xd2700000-0xd27fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814183] pci_bus 0000:1b: resource 2 [mem 0xd4700000-0xd47fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814185] pci_bus 0000:1c: resource 0 [io  0xb000-0xbfff]
Dec  5 22:49:06 u14-04 kernel: [    0.814186] pci_bus 0000:1c: resource 1 [mem 0xd2b00000-0xd2bfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814188] pci_bus 0000:1c: resource 2 [mem 0xd4a00000-0xd4afffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814190] pci_bus 0000:1d: resource 1 [mem 0xd2f00000-0xd2ffffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814192] pci_bus 0000:1d: resource 2 [mem 0xd4c00000-0xd4cfffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814194] pci_bus 0000:1e: resource 1 [mem 0xd3300000-0xd33fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814195] pci_bus 0000:1e: resource 2 [mem 0xd4e00000-0xd4efffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814197] pci_bus 0000:1f: resource 1 [mem 0xd3700000-0xd37fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814199] pci_bus 0000:1f: resource 2 [mem 0xd5000000-0xd50fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814201] pci_bus 0000:20: resource 1 [mem 0xd3b00000-0xd3bfffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814202] pci_bus 0000:20: resource 2 [mem 0xd5200000-0xd52fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814204] pci_bus 0000:21: resource 1 [mem 0xd3f00000-0xd3ffffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814206] pci_bus 0000:21: resource 2 [mem 0xd5400000-0xd54fffff 64bit pref]
Dec  5 22:49:06 u14-04 kernel: [    0.814208] pci_bus 0000:22: resource 1 [mem 0xd4300000-0xd43fffff]
Dec  5 22:49:06 u14-04 kernel: [    0.814210] pci_bus 0000:22: resource 2 [mem 0xd5600000-0xd56fffff 64bit pref]
``

[pitatus]

sounds like the system is happier with the additional resources, does the system swap during the cpu spike? if you disable the master job cache does that help (/etc/salt/master, 'job_cache: False', restart master), if disabling the job cache helps you may have an IO issue (need faster disk channel, SSD's, etc), if swapping may just need more RAM assigned to the box.. (re-enable job cache after experimenting..)

[rallytime]

I am going to close this issue since we didn't hear back. However, if this remains an issue, please leave a comment and we will be glad to open this again and readdress the problem. Thanks!