Salt: GitFS cause extra high CPU usage on the master

Created on 3 Jul 2017 · 21Comments · Source: saltstack/salt

Description of Issue/Question

Over the last 6-12 months we've found state.apply to be very slow and causing very high CPU usage on our salt master. We have about 20 formulas, one external state tree with the master top file and one external pillar, each in separate Git repos.

While debugging I decided to disable the git fileserver in the master configuration and go back to using the basic roots file system, creating a CRON job to simulate gitfs by running git pull every 30 seconds.

This small change has made state.apply over 10x faster than it was before and has lowered CPU usage on the salt master so much that we've been able to reduce it from a 6vCPU 8GB instance to a 2vCPU 4GB instance and still achieve a lower average CPU usage than before.

Here is the before and after CPU usage on the salt master:

screen shot 2017-07-03 at 1 56 30 pm

Pending Discussion

Source

ghost

Most helpful comment

The current plan is to have this functionality present in the next feature release (this fall).

terminalmage on 13 Jun 2018

👍4

All 21 comments

@terminalmage Can you take a look at this when you have time? I am looking for some information on if this should be expected with 20+ git repositories on gitfs?

Thanks,
Daniel

gtmanfred on 3 Jul 2017

Well, it's hard to say, with no configuration examples, no versions report, no idea whether GitPython or Pygit2 is being used, and no useful information about the repositories.

terminalmage on 3 Jul 2017

We are running the following version:

Salt Version:
           Salt: 2016.11.6

Dependency Versions:
           cffi: 1.8.3
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: 0.24.2
        libnacl: 1.5.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.3.7
      pycparser: 2.17
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.24.2
         Python: 2.7.13 (default, Jan 19 2017, 14:48:08)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1

System Versions:
           dist: debian 9.0
        machine: x86_64
        release: 4.9.0-3-amd64
         system: Linux
        version: debian 9.0

For GitFS and our pillars we are using:

gitfs_provider: pygit2

We have about 20 gitfs_remotes and one gitfs_pillar. Of those 20 gifs_remotes, 19 are formulas, the last has our salt states and top file.

ghost on 4 Jul 2017

I assume the spikes are happening when the state.apply is executed? It'd be hard for me to say what is causing it without knowing what the layout is, but short of that we do have some logging meant to help troubleshoot this sort of issue, so let's start with that. You'll need to set log_level_logfile: trace in your master config file and restart the salt-master daemon. After doing so, execute a state.apply and check to see that you're getting the CPU over-utilization you're expecting. Once you've confirmed this, you can feel free to comment out the log_level_logfile line and restart the master, since it will generate a lot of logging.

The next thing to do would be to filter out a specific sort of line, by running the following command:

grep 'Master function call' /var/log/salt/master >~/timing_data.log

After you have done this, post the contents of the timing_data.log. This will help narrow down where in GitFS a given function is taking longer, which has been helpful in the past in diagnosing issues with CPU utilization.

terminalmage on 5 Jul 2017

Thanks @terminalmage!

ghost on 5 Jul 2017

Here is the trace of master function call when using GitFS for a single highstate execution:

2017-07-05 10:49:44,088 [salt.master      ][TRACE   ][28594] Master function call _pillar took 10.9189629555 seconds
2017-07-05 10:49:44,215 [salt.master      ][TRACE   ][28593] Master function call _master_opts took 0.0203490257263 seconds
2017-07-05 10:49:44,229 [salt.master      ][TRACE   ][28595] Master function call _file_envs took 8.79764556885e-05 seconds
2017-07-05 10:49:44,556 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.321863174438 seconds
2017-07-05 10:49:44,791 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.226008176804 seconds
2017-07-05 10:49:44,905 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.105545043945 seconds
2017-07-05 10:49:45,083 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.1703748703 seconds
2017-07-05 10:49:45,240 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.148874044418 seconds
2017-07-05 10:49:45,320 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0716469287872 seconds
2017-07-05 10:49:45,503 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.176724910736 seconds
2017-07-05 10:49:45,659 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.147601127625 seconds
2017-07-05 10:49:45,689 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0210168361664 seconds
2017-07-05 10:49:45,898 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.20565700531 seconds
2017-07-05 10:49:46,068 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.15965294838 seconds
2017-07-05 10:49:46,240 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.16241312027 seconds
2017-07-05 10:49:46,277 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.0281100273132 seconds
2017-07-05 10:49:46,436 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.15570807457 seconds
2017-07-05 10:49:46,597 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.152132987976 seconds
2017-07-05 10:49:46,710 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.105358123779 seconds
2017-07-05 10:49:46,922 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.203437805176 seconds
2017-07-05 10:49:47,087 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.154041051865 seconds
2017-07-05 10:49:47,275 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.178432941437 seconds
2017-07-05 10:49:47,307 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.0236959457397 seconds
2017-07-05 10:49:47,404 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0937528610229 seconds
2017-07-05 10:49:47,573 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.161285877228 seconds
2017-07-05 10:49:58,783 [salt.master      ][TRACE   ][28593] Master function call _pillar took 11.1976218224 seconds
2017-07-05 10:49:58,831 [salt.master      ][TRACE   ][28595] Master function call _file_envs took 8.32080841064e-05 seconds
2017-07-05 10:49:58,855 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0218138694763 seconds
2017-07-05 10:49:58,871 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0120658874512 seconds
2017-07-05 10:49:58,929 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223519802094 seconds
2017-07-05 10:49:58,942 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0110349655151 seconds
2017-07-05 10:49:58,993 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0226190090179 seconds
2017-07-05 10:49:59,008 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0115828514099 seconds
2017-07-05 10:49:59,057 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0214729309082 seconds
2017-07-05 10:49:59,071 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0112330913544 seconds
2017-07-05 10:49:59,123 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0228569507599 seconds
2017-07-05 10:49:59,138 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0122559070587 seconds
2017-07-05 10:49:59,191 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0260100364685 seconds
2017-07-05 10:49:59,206 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0116891860962 seconds
2017-07-05 10:49:59,253 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0217609405518 seconds
2017-07-05 10:49:59,268 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0118799209595 seconds
2017-07-05 10:49:59,293 [salt.master      ][TRACE   ][28595] Master function call _serve_file took 0.0214087963104 seconds
2017-07-05 10:49:59,317 [salt.master      ][TRACE   ][28594] Master function call _serve_file took 0.021281003952 seconds
2017-07-05 10:49:59,369 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225620269775 seconds
2017-07-05 10:49:59,383 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0113439559937 seconds
2017-07-05 10:49:59,432 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0181338787079 seconds
2017-07-05 10:49:59,448 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0136151313782 seconds
2017-07-05 10:49:59,473 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0221219062805 seconds
2017-07-05 10:49:59,487 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112581253052 seconds
2017-07-05 10:49:59,569 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0234570503235 seconds
2017-07-05 10:49:59,583 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0108828544617 seconds
2017-07-05 10:49:59,608 [salt.master      ][TRACE   ][28594] Master function call _serve_file took 0.0218970775604 seconds
2017-07-05 10:49:59,634 [salt.master      ][TRACE   ][28593] Master function call _serve_file took 0.0218651294708 seconds
2017-07-05 10:49:59,688 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0209059715271 seconds
2017-07-05 10:49:59,703 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112071037292 seconds
2017-07-05 10:49:59,727 [salt.master      ][TRACE   ][28593] Master function call _serve_file took 0.0214700698853 seconds
2017-07-05 10:49:59,752 [salt.master      ][TRACE   ][28595] Master function call _serve_file took 0.021537065506 seconds
2017-07-05 10:49:59,802 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0145699977875 seconds
2017-07-05 10:49:59,821 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0155220031738 seconds
2017-07-05 10:49:59,845 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0218901634216 seconds
2017-07-05 10:49:59,860 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0117919445038 seconds
2017-07-05 10:49:59,916 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0221951007843 seconds
2017-07-05 10:49:59,930 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0109469890594 seconds
2017-07-05 10:49:59,955 [salt.master      ][TRACE   ][28594] Master function call _serve_file took 0.0213811397552 seconds
2017-07-05 10:49:59,981 [salt.master      ][TRACE   ][28593] Master function call _serve_file took 0.02205991745 seconds
2017-07-05 10:50:00,037 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211970806122 seconds
2017-07-05 10:50:00,051 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.010960817337 seconds
2017-07-05 10:50:00,111 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0250208377838 seconds
2017-07-05 10:50:00,127 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.012433052063 seconds
2017-07-05 10:50:00,181 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0224249362946 seconds
2017-07-05 10:50:00,197 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0118989944458 seconds
2017-07-05 10:50:00,221 [salt.master      ][TRACE   ][28595] Master function call _serve_file took 0.0212099552155 seconds
2017-07-05 10:50:00,246 [salt.master      ][TRACE   ][28594] Master function call _serve_file took 0.0209012031555 seconds
2017-07-05 10:50:00,300 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227448940277 seconds
2017-07-05 10:50:00,323 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0198919773102 seconds
2017-07-05 10:50:00,366 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0132608413696 seconds
2017-07-05 10:50:00,384 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0157771110535 seconds
2017-07-05 10:50:00,409 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0210909843445 seconds
2017-07-05 10:50:00,423 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0110960006714 seconds
2017-07-05 10:50:00,474 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0235769748688 seconds
2017-07-05 10:50:00,496 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0192680358887 seconds
2017-07-05 10:50:00,554 [salt.master      ][TRACE   ][28594] Master function call _ext_nodes took 0.00363206863403 seconds
2017-07-05 10:50:00,581 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.014662027359 seconds
2017-07-05 10:50:00,614 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0144529342651 seconds
2017-07-05 10:50:00,651 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.0142278671265 seconds
2017-07-05 10:50:00,684 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.0156931877136 seconds
2017-07-05 10:50:00,732 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0140800476074 seconds
2017-07-05 10:50:00,763 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0210218429565 seconds
2017-07-05 10:50:00,777 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0114297866821 seconds
2017-07-05 10:50:00,804 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0142040252686 seconds
2017-07-05 10:50:00,834 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.014228105545 seconds
2017-07-05 10:50:00,864 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.014456987381 seconds
2017-07-05 10:50:00,898 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.014484167099 seconds
2017-07-05 10:50:00,930 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.0144629478455 seconds
2017-07-05 10:50:00,963 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.017422914505 seconds
2017-07-05 10:50:00,997 [salt.master      ][TRACE   ][28595] Master function call _file_list took 0.0146849155426 seconds
2017-07-05 10:50:01,069 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0171949863434 seconds
2017-07-05 10:50:01,090 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0181770324707 seconds
2017-07-05 10:50:01,115 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0215289592743 seconds
2017-07-05 10:50:01,130 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0109939575195 seconds
2017-07-05 10:50:01,180 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0155961513519 seconds
2017-07-05 10:50:01,200 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0178210735321 seconds
2017-07-05 10:50:01,226 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0216569900513 seconds
2017-07-05 10:50:01,241 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0119700431824 seconds
2017-07-05 10:50:01,281 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0151360034943 seconds
2017-07-05 10:50:01,299 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0147919654846 seconds
2017-07-05 10:50:01,327 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0245311260223 seconds
2017-07-05 10:50:01,342 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0121178627014 seconds
2017-07-05 10:50:01,379 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.018012046814 seconds
2017-07-05 10:50:01,410 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0272481441498 seconds
2017-07-05 10:50:01,438 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.023078918457 seconds
2017-07-05 10:50:01,452 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0115129947662 seconds
2017-07-05 10:50:01,499 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0229120254517 seconds
2017-07-05 10:50:01,515 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0124521255493 seconds
2017-07-05 10:50:01,564 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0224750041962 seconds
2017-07-05 10:50:01,580 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0126039981842 seconds
2017-07-05 10:50:01,620 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0219738483429 seconds
2017-07-05 10:50:01,634 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0113477706909 seconds
2017-07-05 10:50:01,674 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0234730243683 seconds
2017-07-05 10:50:01,689 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0119178295135 seconds
2017-07-05 10:50:01,723 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0230209827423 seconds
2017-07-05 10:50:01,740 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0141959190369 seconds
2017-07-05 10:50:01,809 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0228891372681 seconds
2017-07-05 10:50:01,824 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0117671489716 seconds
2017-07-05 10:50:01,867 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0282390117645 seconds
2017-07-05 10:50:01,882 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0116691589355 seconds
2017-07-05 10:50:01,912 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0217370986938 seconds
2017-07-05 10:50:01,928 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0129299163818 seconds
2017-07-05 10:50:01,965 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0213861465454 seconds
2017-07-05 10:50:01,980 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119440555573 seconds
2017-07-05 10:50:02,016 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227630138397 seconds
2017-07-05 10:50:02,030 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0114619731903 seconds
2017-07-05 10:50:02,061 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0232050418854 seconds
2017-07-05 10:50:02,078 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0144670009613 seconds
2017-07-05 10:50:02,111 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0187039375305 seconds
2017-07-05 10:50:02,129 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0149538516998 seconds
2017-07-05 10:50:02,156 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0240631103516 seconds
2017-07-05 10:50:02,183 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0229849815369 seconds
2017-07-05 10:50:02,242 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0477638244629 seconds
2017-07-05 10:50:02,265 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0197579860687 seconds
2017-07-05 10:50:02,294 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0226519107819 seconds
2017-07-05 10:50:02,309 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0124070644379 seconds
2017-07-05 10:50:02,341 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0224280357361 seconds
2017-07-05 10:50:02,355 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.011482000351 seconds
2017-07-05 10:50:02,402 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0325698852539 seconds
2017-07-05 10:50:02,419 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0138440132141 seconds
2017-07-05 10:50:02,448 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0229029655457 seconds
2017-07-05 10:50:02,463 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0122041702271 seconds
2017-07-05 10:50:02,501 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0238080024719 seconds
2017-07-05 10:50:02,516 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0119531154633 seconds
2017-07-05 10:50:02,562 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0233588218689 seconds
2017-07-05 10:50:02,577 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0120058059692 seconds
2017-07-05 10:50:02,604 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0217130184174 seconds
2017-07-05 10:50:02,619 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119700431824 seconds
2017-07-05 10:50:02,660 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0215849876404 seconds
2017-07-05 10:50:02,674 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.011470079422 seconds
2017-07-05 10:50:02,716 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0242440700531 seconds
2017-07-05 10:50:02,732 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0136168003082 seconds
2017-07-05 10:50:02,762 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0222320556641 seconds
2017-07-05 10:50:02,777 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0122067928314 seconds
2017-07-05 10:50:02,813 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.016233921051 seconds
2017-07-05 10:50:02,830 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0148808956146 seconds
2017-07-05 10:50:02,856 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.022351026535 seconds
2017-07-05 10:50:02,870 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.01154088974 seconds
2017-07-05 10:50:02,902 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.021253824234 seconds
2017-07-05 10:50:02,921 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0158801078796 seconds
2017-07-05 10:50:02,954 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0214791297913 seconds
2017-07-05 10:50:02,968 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0109689235687 seconds
2017-07-05 10:50:03,003 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0251610279083 seconds
2017-07-05 10:50:03,020 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0139570236206 seconds
2017-07-05 10:50:03,065 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0237009525299 seconds
2017-07-05 10:50:03,080 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0121109485626 seconds
2017-07-05 10:50:03,117 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0167019367218 seconds
2017-07-05 10:50:03,135 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0153319835663 seconds
2017-07-05 10:50:03,166 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0281529426575 seconds
2017-07-05 10:50:03,185 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0153708457947 seconds
2017-07-05 10:50:03,222 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0255839824677 seconds
2017-07-05 10:50:03,241 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0152990818024 seconds
2017-07-05 10:50:03,274 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0262989997864 seconds
2017-07-05 10:50:03,299 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0152449607849 seconds
2017-07-05 10:50:03,343 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.032989025116 seconds
2017-07-05 10:50:03,363 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0170960426331 seconds
2017-07-05 10:50:03,407 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0266621112823 seconds
2017-07-05 10:50:03,425 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0150759220123 seconds
2017-07-05 10:50:03,467 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.035099029541 seconds
2017-07-05 10:50:03,484 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0142660140991 seconds
2017-07-05 10:50:03,527 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0260589122772 seconds
2017-07-05 10:50:03,550 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0193769931793 seconds
2017-07-05 10:50:03,601 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0357930660248 seconds
2017-07-05 10:50:03,618 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0139391422272 seconds
2017-07-05 10:50:03,650 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0258300304413 seconds
2017-07-05 10:50:03,668 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0148918628693 seconds
2017-07-05 10:50:03,715 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0294380187988 seconds
2017-07-05 10:50:03,733 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.015056848526 seconds
2017-07-05 10:50:03,773 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0280129909515 seconds
2017-07-05 10:50:03,793 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0166881084442 seconds
2017-07-05 10:50:03,837 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.037456035614 seconds
2017-07-05 10:50:03,854 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0140559673309 seconds
2017-07-05 10:50:03,891 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0226979255676 seconds
2017-07-05 10:50:03,906 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.011696100235 seconds
2017-07-05 10:50:03,937 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0154960155487 seconds
2017-07-05 10:50:03,955 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0152719020844 seconds
2017-07-05 10:50:03,986 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0266599655151 seconds
2017-07-05 10:50:04,004 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0153639316559 seconds
2017-07-05 10:50:04,036 [salt.master      ][TRACE   ][28594] Master function call _serve_file took 0.0279569625854 seconds
2017-07-05 10:50:04,067 [salt.master      ][TRACE   ][28593] Master function call _serve_file took 0.0273089408875 seconds
2017-07-05 10:50:04,237 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0263888835907 seconds
2017-07-05 10:50:04,256 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0154271125793 seconds
2017-07-05 10:50:04,368 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0266759395599 seconds
2017-07-05 10:50:04,386 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0151028633118 seconds
2017-07-05 10:50:04,431 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0248639583588 seconds
2017-07-05 10:50:04,449 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0148570537567 seconds
2017-07-05 10:50:04,486 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0145928859711 seconds
2017-07-05 10:50:04,503 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0144798755646 seconds
2017-07-05 10:50:04,528 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0216701030731 seconds
2017-07-05 10:50:04,542 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0109479427338 seconds
2017-07-05 10:50:04,568 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0181310176849 seconds
2017-07-05 10:50:04,590 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0185029506683 seconds
2017-07-05 10:50:04,616 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0235960483551 seconds
2017-07-05 10:50:04,634 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0148508548737 seconds
2017-07-05 10:50:04,671 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0249028205872 seconds
2017-07-05 10:50:04,691 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0162200927734 seconds
2017-07-05 10:50:04,722 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0244288444519 seconds
2017-07-05 10:50:04,739 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0144851207733 seconds
2017-07-05 10:50:04,777 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0247349739075 seconds
2017-07-05 10:50:04,795 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0142509937286 seconds
2017-07-05 10:50:04,834 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.024689912796 seconds
2017-07-05 10:50:04,852 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0141608715057 seconds
2017-07-05 10:50:04,887 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0282008647919 seconds
2017-07-05 10:50:04,908 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0176999568939 seconds
2017-07-05 10:50:04,948 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0237720012665 seconds
2017-07-05 10:50:04,966 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.01433801651 seconds
2017-07-05 10:50:05,005 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0261490345001 seconds
2017-07-05 10:50:05,030 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0213670730591 seconds
2017-07-05 10:50:05,062 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0244688987732 seconds
2017-07-05 10:50:05,079 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0143570899963 seconds
2017-07-05 10:50:05,125 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.02414894104 seconds
2017-07-05 10:50:05,142 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0140478610992 seconds
2017-07-05 10:50:05,179 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0247230529785 seconds
2017-07-05 10:50:05,197 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.015044927597 seconds
2017-07-05 10:50:05,231 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0271730422974 seconds
2017-07-05 10:50:05,252 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0177221298218 seconds
2017-07-05 10:50:05,287 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0210580825806 seconds
2017-07-05 10:50:05,301 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112979412079 seconds
2017-07-05 10:50:05,335 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0160899162292 seconds
2017-07-05 10:50:05,355 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0168890953064 seconds
2017-07-05 10:50:05,394 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.036052942276 seconds
2017-07-05 10:50:05,410 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0118448734283 seconds
2017-07-05 10:50:05,454 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0292630195618 seconds
2017-07-05 10:50:05,472 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.014701128006 seconds
2017-07-05 10:50:05,506 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0257239341736 seconds
2017-07-05 10:50:05,523 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0137469768524 seconds
2017-07-05 10:50:05,572 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0256872177124 seconds
2017-07-05 10:50:05,589 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0142331123352 seconds
2017-07-05 10:50:05,622 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.022362947464 seconds
2017-07-05 10:50:05,637 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119500160217 seconds
2017-07-05 10:50:05,685 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0228679180145 seconds
2017-07-05 10:50:05,701 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0124979019165 seconds
2017-07-05 10:50:05,737 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0234889984131 seconds
2017-07-05 10:50:05,754 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0142388343811 seconds
2017-07-05 10:50:05,786 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0241720676422 seconds
2017-07-05 10:50:05,802 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0134570598602 seconds
2017-07-05 10:50:05,843 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0229299068451 seconds
2017-07-05 10:50:05,858 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0111970901489 seconds
2017-07-05 10:50:05,987 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0250618457794 seconds
2017-07-05 10:50:06,004 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0136671066284 seconds
2017-07-05 10:50:06,046 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0250380039215 seconds
2017-07-05 10:50:06,063 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0138049125671 seconds
2017-07-05 10:50:06,123 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0226640701294 seconds
2017-07-05 10:50:06,140 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0120811462402 seconds
2017-07-05 10:50:06,183 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0222859382629 seconds
2017-07-05 10:50:06,197 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0115170478821 seconds
2017-07-05 10:50:06,227 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211040973663 seconds
2017-07-05 10:50:06,241 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0108208656311 seconds
2017-07-05 10:50:06,277 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0230960845947 seconds
2017-07-05 10:50:06,290 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0108029842377 seconds
2017-07-05 10:50:06,323 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0179481506348 seconds
2017-07-05 10:50:06,344 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.017814874649 seconds
2017-07-05 10:50:06,373 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.024994134903 seconds
2017-07-05 10:50:06,390 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0142951011658 seconds
2017-07-05 10:50:06,425 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0244162082672 seconds
2017-07-05 10:50:06,442 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0139989852905 seconds
2017-07-05 10:50:06,473 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.024610042572 seconds
2017-07-05 10:50:06,490 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0137431621552 seconds
2017-07-05 10:50:06,524 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0234479904175 seconds
2017-07-05 10:50:06,540 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0131459236145 seconds
2017-07-05 10:50:06,582 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227918624878 seconds
2017-07-05 10:50:06,598 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0127348899841 seconds
2017-07-05 10:50:06,634 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0302309989929 seconds
2017-07-05 10:50:06,653 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0152490139008 seconds
2017-07-05 10:50:06,713 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0233199596405 seconds
2017-07-05 10:50:06,729 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.013396024704 seconds
2017-07-05 10:50:06,766 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.023188829422 seconds
2017-07-05 10:50:06,784 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0147659778595 seconds
2017-07-05 10:50:06,814 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.02374792099 seconds
2017-07-05 10:50:06,831 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0135579109192 seconds
2017-07-05 10:50:06,917 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0392649173737 seconds
2017-07-05 10:50:06,934 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0138099193573 seconds
2017-07-05 10:50:06,969 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0234889984131 seconds
2017-07-05 10:50:06,986 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0142030715942 seconds
2017-07-05 10:50:07,023 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0265259742737 seconds
2017-07-05 10:50:07,042 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.015606880188 seconds
2017-07-05 10:50:07,089 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.023601770401 seconds
2017-07-05 10:50:07,106 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0137388706207 seconds
2017-07-05 10:50:07,145 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0239059925079 seconds
2017-07-05 10:50:07,162 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0140450000763 seconds
2017-07-05 10:50:07,191 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0234789848328 seconds
2017-07-05 10:50:07,207 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0133218765259 seconds
2017-07-05 10:50:08,323 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0166389942169 seconds
2017-07-05 10:50:08,341 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0148658752441 seconds
2017-07-05 10:50:08,367 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0238559246063 seconds
2017-07-05 10:50:08,383 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0132839679718 seconds
2017-07-05 10:50:08,421 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0275449752808 seconds
2017-07-05 10:50:08,439 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0150458812714 seconds
2017-07-05 10:50:08,469 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0248610973358 seconds
2017-07-05 10:50:08,486 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0141561031342 seconds
2017-07-05 10:50:08,519 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0243828296661 seconds
2017-07-05 10:50:08,535 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0128619670868 seconds
2017-07-05 10:50:08,569 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0235240459442 seconds
2017-07-05 10:50:08,585 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0132241249084 seconds
2017-07-05 10:50:08,615 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0241370201111 seconds
2017-07-05 10:50:08,631 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0130438804626 seconds
2017-07-05 10:50:08,660 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227088928223 seconds
2017-07-05 10:50:08,675 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0124638080597 seconds
2017-07-05 10:50:08,716 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0247309207916 seconds
2017-07-05 10:50:08,737 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.018296957016 seconds
2017-07-05 10:50:08,766 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223739147186 seconds
2017-07-05 10:50:08,783 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0143890380859 seconds
2017-07-05 10:50:08,832 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0219869613647 seconds
2017-07-05 10:50:08,845 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0104739665985 seconds
2017-07-05 10:50:08,912 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0354290008545 seconds
2017-07-05 10:50:08,926 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0113608837128 seconds
2017-07-05 10:50:08,952 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0159940719604 seconds
2017-07-05 10:50:08,970 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0152580738068 seconds
2017-07-05 10:50:08,993 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0210168361664 seconds
2017-07-05 10:50:09,007 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.01060795784 seconds
2017-07-05 10:50:09,036 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0227508544922 seconds
2017-07-05 10:50:09,051 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0124640464783 seconds
2017-07-05 10:50:09,084 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0205538272858 seconds
2017-07-05 10:50:09,098 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112931728363 seconds
2017-07-05 10:50:09,127 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0209200382233 seconds
2017-07-05 10:50:09,141 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0109400749207 seconds
2017-07-05 10:50:09,178 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.021754026413 seconds
2017-07-05 10:50:09,192 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0115668773651 seconds
2017-07-05 10:50:10,580 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0226538181305 seconds
2017-07-05 10:50:10,607 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0238859653473 seconds
2017-07-05 10:50:10,621 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0113699436188 seconds
2017-07-05 10:50:10,652 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0205020904541 seconds
2017-07-05 10:50:10,676 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0216209888458 seconds
2017-07-05 10:50:10,692 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0131409168243 seconds
2017-07-05 10:50:11,711 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0214619636536 seconds
2017-07-05 10:50:11,735 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0213539600372 seconds
2017-07-05 10:50:11,750 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0117220878601 seconds
2017-07-05 10:50:11,786 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0226919651031 seconds
2017-07-05 10:50:11,803 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0146179199219 seconds
2017-07-05 10:50:11,833 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0222451686859 seconds
2017-07-05 10:50:11,848 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0121219158173 seconds
2017-07-05 10:50:11,882 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0230069160461 seconds
2017-07-05 10:50:11,912 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0217709541321 seconds
2017-07-05 10:50:11,926 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.010978937149 seconds
2017-07-05 10:50:11,960 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0231418609619 seconds
2017-07-05 10:50:11,977 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0144691467285 seconds
2017-07-05 10:50:12,007 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0218868255615 seconds
2017-07-05 10:50:12,024 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0135788917542 seconds
2017-07-05 10:50:12,634 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0222151279449 seconds
2017-07-05 10:50:12,659 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0222020149231 seconds
2017-07-05 10:50:12,674 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0122649669647 seconds
2017-07-05 10:50:12,716 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.022598028183 seconds
2017-07-05 10:50:12,731 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0116720199585 seconds
2017-07-05 10:50:12,761 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0229570865631 seconds
2017-07-05 10:50:12,779 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0151908397675 seconds
2017-07-05 10:50:12,915 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211899280548 seconds
2017-07-05 10:50:12,939 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0211091041565 seconds
2017-07-05 10:50:12,954 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0120270252228 seconds
2017-07-05 10:50:12,997 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0247600078583 seconds
2017-07-05 10:50:13,013 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.013298034668 seconds
2017-07-05 10:50:13,041 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0219900608063 seconds
2017-07-05 10:50:13,055 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0109889507294 seconds
2017-07-05 10:50:13,185 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0257759094238 seconds
2017-07-05 10:50:13,212 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0241820812225 seconds
2017-07-05 10:50:13,228 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0128500461578 seconds
2017-07-05 10:50:20,515 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0339040756226 seconds
2017-07-05 10:50:20,545 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0248649120331 seconds
2017-07-05 10:50:20,559 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0110640525818 seconds
2017-07-05 10:50:20,660 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0235381126404 seconds
2017-07-05 10:50:20,686 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0224049091339 seconds
2017-07-05 10:50:20,704 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0153450965881 seconds
2017-07-05 10:50:20,775 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0243558883667 seconds
2017-07-05 10:50:20,803 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0238568782806 seconds
2017-07-05 10:50:20,818 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.012521982193 seconds
2017-07-05 10:50:20,857 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0241498947144 seconds
2017-07-05 10:50:20,876 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0160269737244 seconds
2017-07-05 10:50:20,947 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0247449874878 seconds
2017-07-05 10:50:20,978 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0268790721893 seconds
2017-07-05 10:50:20,995 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0144629478455 seconds
2017-07-05 10:50:21,042 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0229041576385 seconds
2017-07-05 10:50:21,069 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0232169628143 seconds
2017-07-05 10:50:21,085 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.013445854187 seconds
2017-07-05 10:50:21,148 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0232310295105 seconds
2017-07-05 10:50:21,174 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0225868225098 seconds
2017-07-05 10:50:21,190 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.013797044754 seconds
2017-07-05 10:50:21,269 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0566420555115 seconds
2017-07-05 10:50:21,313 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0365760326385 seconds
2017-07-05 10:50:21,333 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0172989368439 seconds
2017-07-05 10:50:21,377 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.022775888443 seconds
2017-07-05 10:50:21,405 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0250318050385 seconds
2017-07-05 10:50:21,424 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0159330368042 seconds
2017-07-05 10:50:21,504 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0237250328064 seconds
2017-07-05 10:50:21,530 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0234851837158 seconds
2017-07-05 10:50:21,547 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0139510631561 seconds
2017-07-05 10:50:21,593 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0247311592102 seconds
2017-07-05 10:50:21,624 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0285730361938 seconds
2017-07-05 10:50:21,641 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0138251781464 seconds
2017-07-05 10:50:21,689 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0227019786835 seconds
2017-07-05 10:50:21,715 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0231251716614 seconds
2017-07-05 10:50:21,732 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.013060092926 seconds
2017-07-05 10:50:21,794 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0284280776978 seconds
2017-07-05 10:50:21,821 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0239040851593 seconds
2017-07-05 10:50:21,837 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0132651329041 seconds
2017-07-05 10:50:21,882 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0249481201172 seconds
2017-07-05 10:50:21,911 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.025297164917 seconds
2017-07-05 10:50:21,928 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0148949623108 seconds
2017-07-05 10:50:21,979 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0237600803375 seconds
2017-07-05 10:50:22,007 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0247881412506 seconds
2017-07-05 10:50:22,023 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0133311748505 seconds
2017-07-05 10:50:22,068 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0227329730988 seconds
2017-07-05 10:50:22,094 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.023411989212 seconds
2017-07-05 10:50:22,111 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0137240886688 seconds
2017-07-05 10:50:22,157 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0241739749908 seconds
2017-07-05 10:50:22,186 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0257232189178 seconds
2017-07-05 10:50:22,203 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0142149925232 seconds
2017-07-05 10:50:22,247 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0229909420013 seconds
2017-07-05 10:50:22,274 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0242629051208 seconds
2017-07-05 10:50:22,291 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.01384806633 seconds
2017-07-05 10:50:22,336 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0232391357422 seconds
2017-07-05 10:50:22,363 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0237770080566 seconds
2017-07-05 10:50:22,382 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0152790546417 seconds
2017-07-05 10:50:22,428 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0238649845123 seconds
2017-07-05 10:50:22,456 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0257189273834 seconds
2017-07-05 10:50:22,472 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0132839679718 seconds
2017-07-05 10:50:22,516 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223050117493 seconds
2017-07-05 10:50:22,543 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0239279270172 seconds
2017-07-05 10:50:22,563 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0169620513916 seconds
2017-07-05 10:50:22,647 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0225291252136 seconds
2017-07-05 10:50:22,673 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0227320194244 seconds
2017-07-05 10:50:22,689 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.013622045517 seconds
2017-07-05 10:50:22,742 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0233869552612 seconds
2017-07-05 10:50:22,769 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0245041847229 seconds
2017-07-05 10:50:22,786 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0137810707092 seconds
2017-07-05 10:50:22,833 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0230860710144 seconds
2017-07-05 10:50:22,860 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0242509841919 seconds
2017-07-05 10:50:22,878 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0151159763336 seconds
2017-07-05 10:50:22,921 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.022588968277 seconds
2017-07-05 10:50:22,947 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0229239463806 seconds
2017-07-05 10:50:22,963 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0139379501343 seconds
2017-07-05 10:50:23,007 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0242199897766 seconds
2017-07-05 10:50:23,036 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0257511138916 seconds
2017-07-05 10:50:23,052 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0133628845215 seconds
2017-07-05 10:50:23,097 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0231630802155 seconds
2017-07-05 10:50:23,124 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0238590240479 seconds
2017-07-05 10:50:23,140 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0132641792297 seconds
2017-07-05 10:50:23,185 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0228390693665 seconds
2017-07-05 10:50:23,212 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0233199596405 seconds
2017-07-05 10:50:23,228 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0127911567688 seconds
2017-07-05 10:50:23,274 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0249049663544 seconds
2017-07-05 10:50:23,304 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0268549919128 seconds
2017-07-05 10:50:23,323 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0162620544434 seconds
2017-07-05 10:50:23,376 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0234568119049 seconds
2017-07-05 10:50:23,404 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.024337053299 seconds
2017-07-05 10:50:23,420 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0138139724731 seconds
2017-07-05 10:50:23,467 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0232620239258 seconds
2017-07-05 10:50:23,497 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.027841091156 seconds
2017-07-05 10:50:23,514 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0139181613922 seconds
2017-07-05 10:50:23,560 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0245931148529 seconds
2017-07-05 10:50:23,590 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0263648033142 seconds
2017-07-05 10:50:23,606 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0134460926056 seconds
2017-07-05 10:50:27,209 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0219287872314 seconds
2017-07-05 10:50:27,234 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0218560695648 seconds
2017-07-05 10:50:27,248 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0109989643097 seconds
2017-07-05 10:50:28,825 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.024542093277 seconds
2017-07-05 10:50:28,853 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0251832008362 seconds
2017-07-05 10:50:28,876 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0199689865112 seconds
2017-07-05 10:50:28,915 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0260701179504 seconds
2017-07-05 10:50:28,935 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0171158313751 seconds
2017-07-05 10:50:28,967 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0255081653595 seconds
2017-07-05 10:50:28,984 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.014750957489 seconds
2017-07-05 10:50:29,018 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0253419876099 seconds
2017-07-05 10:50:29,035 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0149340629578 seconds
2017-07-05 10:50:29,091 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.02485704422 seconds
2017-07-05 10:50:29,118 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0244600772858 seconds
2017-07-05 10:50:29,136 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0152518749237 seconds
2017-07-05 10:50:29,177 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0294809341431 seconds
2017-07-05 10:50:29,197 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0171930789948 seconds
2017-07-05 10:50:29,231 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0277419090271 seconds
2017-07-05 10:50:29,249 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0150499343872 seconds
2017-07-05 10:50:29,307 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0244209766388 seconds
2017-07-05 10:50:29,332 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225880146027 seconds
2017-07-05 10:50:29,347 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0114870071411 seconds
2017-07-05 10:50:29,804 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0244069099426 seconds
2017-07-05 10:50:29,833 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0251030921936 seconds
2017-07-05 10:50:29,850 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0147030353546 seconds
2017-07-05 10:50:29,891 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0266718864441 seconds
2017-07-05 10:50:29,911 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0167579650879 seconds
2017-07-05 10:50:29,942 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0242440700531 seconds
2017-07-05 10:50:29,960 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0147948265076 seconds
2017-07-05 10:50:29,997 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0265259742737 seconds
2017-07-05 10:50:30,016 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0166940689087 seconds
2017-07-05 10:50:30,068 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0218169689178 seconds
2017-07-05 10:50:30,093 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227470397949 seconds
2017-07-05 10:50:30,108 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0114099979401 seconds
2017-07-05 10:50:30,481 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0236251354218 seconds
2017-07-05 10:50:30,507 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0232079029083 seconds
2017-07-05 10:50:30,522 [salt.master      ][TRACE   ][28595] Master function call _file_find took 0.0113430023193 seconds
2017-07-05 10:50:31,031 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.328094005585 seconds
2017-07-05 10:50:31,051 [salt.master      ][TRACE   ][28593] Master function call _dir_list took 0.0127019882202 seconds
2017-07-05 10:50:31,068 [salt.master      ][TRACE   ][28595] Master function call _dir_list took 0.0117418766022 seconds
2017-07-05 10:50:31,091 [salt.master      ][TRACE   ][28594] Master function call _file_list took 0.0204341411591 seconds
2017-07-05 10:50:31,133 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0385110378265 seconds
2017-07-05 10:50:31,158 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0209310054779 seconds
2017-07-05 10:50:31,172 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0115501880646 seconds
2017-07-05 10:50:31,228 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0491740703583 seconds
2017-07-05 10:50:31,277 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0450320243835 seconds
2017-07-05 10:50:31,297 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0169789791107 seconds
2017-07-05 10:50:31,324 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0229218006134 seconds
2017-07-05 10:50:31,348 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0201649665833 seconds
2017-07-05 10:50:31,362 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116169452667 seconds
2017-07-05 10:50:31,391 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0234789848328 seconds
2017-07-05 10:50:31,416 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0215950012207 seconds
2017-07-05 10:50:31,430 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112969875336 seconds
2017-07-05 10:50:31,458 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0233130455017 seconds
2017-07-05 10:50:31,483 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0220069885254 seconds
2017-07-05 10:50:31,500 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0140039920807 seconds
2017-07-05 10:50:31,528 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0236129760742 seconds
2017-07-05 10:50:31,553 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.021525144577 seconds
2017-07-05 10:50:31,568 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.011873960495 seconds
2017-07-05 10:50:31,596 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.023745059967 seconds
2017-07-05 10:50:31,621 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0216288566589 seconds
2017-07-05 10:50:31,636 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0124430656433 seconds
2017-07-05 10:50:31,666 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.024710893631 seconds
2017-07-05 10:50:31,692 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0224800109863 seconds
2017-07-05 10:50:31,706 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119431018829 seconds
2017-07-05 10:50:31,734 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0229558944702 seconds
2017-07-05 10:50:31,758 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0208730697632 seconds
2017-07-05 10:50:31,773 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116119384766 seconds
2017-07-05 10:50:31,801 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0231881141663 seconds
2017-07-05 10:50:31,825 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0207929611206 seconds
2017-07-05 10:50:31,839 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0111639499664 seconds
2017-07-05 10:50:31,872 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0282111167908 seconds
2017-07-05 10:50:31,897 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0219261646271 seconds
2017-07-05 10:50:31,913 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0129430294037 seconds
2017-07-05 10:50:31,939 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0215780735016 seconds
2017-07-05 10:50:31,962 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0202879905701 seconds
2017-07-05 10:50:31,976 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0111401081085 seconds
2017-07-05 10:50:32,011 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0310590267181 seconds
2017-07-05 10:50:32,034 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0201451778412 seconds
2017-07-05 10:50:32,048 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0111877918243 seconds
2017-07-05 10:50:32,077 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0243780612946 seconds
2017-07-05 10:50:32,102 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223951339722 seconds
2017-07-05 10:50:32,117 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119450092316 seconds
2017-07-05 10:50:32,146 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0233960151672 seconds
2017-07-05 10:50:32,171 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211460590363 seconds
2017-07-05 10:50:32,185 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.01176404953 seconds
2017-07-05 10:50:32,213 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0231928825378 seconds
2017-07-05 10:50:32,237 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0212428569794 seconds
2017-07-05 10:50:32,254 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.013286113739 seconds
2017-07-05 10:50:32,283 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0249109268188 seconds
2017-07-05 10:50:32,309 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0222818851471 seconds
2017-07-05 10:50:32,324 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0122289657593 seconds
2017-07-05 10:50:32,350 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0218160152435 seconds
2017-07-05 10:50:32,373 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.020378112793 seconds
2017-07-05 10:50:32,388 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.011559009552 seconds
2017-07-05 10:50:32,415 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0220229625702 seconds
2017-07-05 10:50:32,439 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0203268527985 seconds
2017-07-05 10:50:32,452 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0109679698944 seconds
2017-07-05 10:50:32,483 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0265369415283 seconds
2017-07-05 10:50:32,515 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0290310382843 seconds
2017-07-05 10:50:32,532 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0139980316162 seconds
2017-07-05 10:50:32,586 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0357730388641 seconds
2017-07-05 10:50:32,610 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0208888053894 seconds
2017-07-05 10:50:32,626 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0128359794617 seconds
2017-07-05 10:50:32,653 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0221309661865 seconds
2017-07-05 10:50:32,676 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0205059051514 seconds
2017-07-05 10:50:32,691 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0120198726654 seconds
2017-07-05 10:50:32,719 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0237760543823 seconds
2017-07-05 10:50:32,744 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0216979980469 seconds
2017-07-05 10:50:32,758 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0114939212799 seconds
2017-07-05 10:50:32,785 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0223948955536 seconds
2017-07-05 10:50:32,809 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0209410190582 seconds
2017-07-05 10:50:32,823 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116741657257 seconds
2017-07-05 10:50:32,851 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0219972133636 seconds
2017-07-05 10:50:32,875 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0209610462189 seconds
2017-07-05 10:50:32,890 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118010044098 seconds
2017-07-05 10:50:32,922 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0268220901489 seconds
2017-07-05 10:50:32,948 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0232901573181 seconds
2017-07-05 10:50:32,964 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0129580497742 seconds
2017-07-05 10:50:32,992 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0233759880066 seconds
2017-07-05 10:50:33,016 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211539268494 seconds
2017-07-05 10:50:33,031 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.011412858963 seconds
2017-07-05 10:50:33,057 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0223009586334 seconds
2017-07-05 10:50:33,081 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0207860469818 seconds
2017-07-05 10:50:33,096 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119140148163 seconds
2017-07-05 10:50:33,143 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0429599285126 seconds
2017-07-05 10:50:33,168 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0217750072479 seconds
2017-07-05 10:50:33,184 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.012139081955 seconds
2017-07-05 10:50:33,213 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0236749649048 seconds
2017-07-05 10:50:33,238 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0210990905762 seconds
2017-07-05 10:50:33,253 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.011666059494 seconds
2017-07-05 10:50:33,283 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0258750915527 seconds
2017-07-05 10:50:33,308 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0213179588318 seconds
2017-07-05 10:50:33,322 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118389129639 seconds
2017-07-05 10:50:33,351 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0248620510101 seconds
2017-07-05 10:50:33,379 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.024276971817 seconds
2017-07-05 10:50:33,395 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0131528377533 seconds
2017-07-05 10:50:33,423 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0236659049988 seconds
2017-07-05 10:50:33,447 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0215170383453 seconds
2017-07-05 10:50:33,462 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116770267487 seconds
2017-07-05 10:50:33,489 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.022537946701 seconds
2017-07-05 10:50:33,512 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0206701755524 seconds
2017-07-05 10:50:33,527 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118339061737 seconds
2017-07-05 10:50:33,556 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0249018669128 seconds
2017-07-05 10:50:33,615 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0558350086212 seconds
2017-07-05 10:50:33,635 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0164308547974 seconds
2017-07-05 10:50:33,663 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0232219696045 seconds
2017-07-05 10:50:33,688 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0215518474579 seconds
2017-07-05 10:50:33,702 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0115931034088 seconds
2017-07-05 10:50:33,730 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225629806519 seconds
2017-07-05 10:50:33,753 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0204570293427 seconds
2017-07-05 10:50:33,769 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0127949714661 seconds
2017-07-05 10:50:33,799 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0256040096283 seconds
2017-07-05 10:50:33,825 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.022696018219 seconds
2017-07-05 10:50:33,839 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0115969181061 seconds
2017-07-05 10:50:33,868 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0241858959198 seconds
2017-07-05 10:50:33,893 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0212318897247 seconds
2017-07-05 10:50:33,907 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118069648743 seconds
2017-07-05 10:50:33,934 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225019454956 seconds
2017-07-05 10:50:33,958 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0209729671478 seconds
2017-07-05 10:50:33,972 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116641521454 seconds
2017-07-05 10:50:34,003 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0256810188293 seconds
2017-07-05 10:50:34,028 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0221729278564 seconds
2017-07-05 10:50:34,043 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118980407715 seconds
2017-07-05 10:50:34,071 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0236988067627 seconds
2017-07-05 10:50:34,095 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211799144745 seconds
2017-07-05 10:50:34,111 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0126781463623 seconds
2017-07-05 10:50:34,140 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225050449371 seconds
2017-07-05 10:50:34,164 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0210249423981 seconds
2017-07-05 10:50:34,178 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0119621753693 seconds
2017-07-05 10:50:34,208 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0246820449829 seconds
2017-07-05 10:50:34,232 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0212008953094 seconds
2017-07-05 10:50:34,246 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0113699436188 seconds
2017-07-05 10:50:34,273 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0225970745087 seconds
2017-07-05 10:50:34,296 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0208928585052 seconds
2017-07-05 10:50:34,311 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0114080905914 seconds
2017-07-05 10:50:34,337 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.02237200737 seconds
2017-07-05 10:50:34,361 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211350917816 seconds
2017-07-05 10:50:34,375 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0110540390015 seconds
2017-07-05 10:50:34,403 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0238609313965 seconds
2017-07-05 10:50:34,427 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0212471485138 seconds
2017-07-05 10:50:34,442 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0123798847198 seconds
2017-07-05 10:50:34,469 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0218849182129 seconds
2017-07-05 10:50:34,492 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0207579135895 seconds
2017-07-05 10:50:34,506 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0112588405609 seconds
2017-07-05 10:50:34,533 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0218350887299 seconds
2017-07-05 10:50:34,557 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0201771259308 seconds
2017-07-05 10:50:34,571 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0113620758057 seconds
2017-07-05 10:50:34,599 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0236439704895 seconds
2017-07-05 10:50:34,623 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211889743805 seconds
2017-07-05 10:50:34,637 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0114269256592 seconds
2017-07-05 10:50:34,664 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0218410491943 seconds
2017-07-05 10:50:34,688 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0211110115051 seconds
2017-07-05 10:50:34,702 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0113041400909 seconds
2017-07-05 10:50:34,729 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0220630168915 seconds
2017-07-05 10:50:34,753 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0200839042664 seconds
2017-07-05 10:50:34,767 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0108678340912 seconds
2017-07-05 10:50:34,797 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0255451202393 seconds
2017-07-05 10:50:34,825 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0246181488037 seconds
2017-07-05 10:50:34,841 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0133330821991 seconds
2017-07-05 10:50:34,868 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0227110385895 seconds
2017-07-05 10:50:34,892 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.021096944809 seconds
2017-07-05 10:50:34,906 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116171836853 seconds
2017-07-05 10:50:34,934 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0222270488739 seconds
2017-07-05 10:50:34,957 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0208129882812 seconds
2017-07-05 10:50:34,972 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0116300582886 seconds
2017-07-05 10:50:35,001 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0244710445404 seconds
2017-07-05 10:50:35,027 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0227980613708 seconds
2017-07-05 10:50:35,042 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.0118288993835 seconds
2017-07-05 10:50:35,070 [salt.master      ][TRACE   ][28593] Master function call _file_hash took 0.0237390995026 seconds
2017-07-05 10:50:35,095 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0215368270874 seconds
2017-07-05 10:50:35,110 [salt.master      ][TRACE   ][28594] Master function call _file_find took 0.01127409935 seconds
2017-07-05 10:50:35,130 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.0142228603363 seconds
2017-07-05 10:50:35,151 [salt.master      ][TRACE   ][28595] Master function call _dir_list took 0.0117740631104 seconds
2017-07-05 10:50:35,169 [salt.master      ][TRACE   ][28594] Master function call _dir_list took 0.0137331485748 seconds
2017-07-05 10:50:35,188 [salt.master      ][TRACE   ][28593] Master function call _file_list took 0.0142621994019 seconds
2017-07-05 10:50:35,215 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0232570171356 seconds
2017-07-05 10:50:35,240 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0218639373779 seconds
2017-07-05 10:50:35,255 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0130550861359 seconds
2017-07-05 10:50:35,283 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0231890678406 seconds
2017-07-05 10:50:35,311 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0242559909821 seconds
2017-07-05 10:50:35,327 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0131189823151 seconds
2017-07-05 10:50:35,355 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0240941047668 seconds
2017-07-05 10:50:35,386 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.026859998703 seconds
2017-07-05 10:50:35,401 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0121171474457 seconds
2017-07-05 10:50:35,428 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0234081745148 seconds
2017-07-05 10:50:35,455 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0236480236053 seconds
2017-07-05 10:50:35,470 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0119528770447 seconds
2017-07-05 10:50:35,496 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0222749710083 seconds
2017-07-05 10:50:35,526 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0264480113983 seconds
2017-07-05 10:50:35,540 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0115058422089 seconds
2017-07-05 10:50:35,597 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0229649543762 seconds
2017-07-05 10:50:35,677 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0771551132202 seconds
2017-07-05 10:50:35,697 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.016095161438 seconds
2017-07-05 10:50:35,730 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0235760211945 seconds
2017-07-05 10:50:35,754 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0212149620056 seconds
2017-07-05 10:50:35,769 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0116758346558 seconds
2017-07-05 10:50:35,798 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0227980613708 seconds
2017-07-05 10:50:35,823 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0213198661804 seconds
2017-07-05 10:50:35,838 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0116908550262 seconds
2017-07-05 10:50:35,874 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223531723022 seconds
2017-07-05 10:50:35,900 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0229878425598 seconds
2017-07-05 10:50:35,915 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0118169784546 seconds
2017-07-05 10:50:35,944 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0228569507599 seconds
2017-07-05 10:50:35,968 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0205280780792 seconds
2017-07-05 10:50:35,983 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0119590759277 seconds
2017-07-05 10:50:36,011 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0216281414032 seconds
2017-07-05 10:50:36,035 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0207190513611 seconds
2017-07-05 10:50:36,103 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0656929016113 seconds
2017-07-05 10:50:36,404 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0387079715729 seconds
2017-07-05 10:50:36,430 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0222129821777 seconds
2017-07-05 10:50:36,445 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0114541053772 seconds
2017-07-05 10:50:36,474 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0242099761963 seconds
2017-07-05 10:50:36,499 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0207269191742 seconds
2017-07-05 10:50:36,513 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.010773897171 seconds
2017-07-05 10:50:36,573 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0267028808594 seconds
2017-07-05 10:50:36,597 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0208990573883 seconds
2017-07-05 10:50:36,611 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0107228755951 seconds
2017-07-05 10:50:36,642 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0210371017456 seconds
2017-07-05 10:50:36,668 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0222411155701 seconds
2017-07-05 10:50:36,681 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0108668804169 seconds
2017-07-05 10:50:36,734 [salt.master      ][TRACE   ][28595] Master function call _file_hash took 0.0223228931427 seconds
2017-07-05 10:50:36,758 [salt.master      ][TRACE   ][28594] Master function call _file_hash took 0.0209469795227 seconds
2017-07-05 10:50:36,773 [salt.master      ][TRACE   ][28593] Master function call _file_find took 0.0127780437469 seconds
2017-07-05 10:50:36,951 [salt.master      ][TRACE   ][28595] Master function call _return took 0.0126090049744 seconds

and here is the same trace when using file_roots:

2017-07-05 10:54:35,493 [salt.master      ][TRACE   ][21289] Master function call _pillar took 4.75189995766 seconds
2017-07-05 10:54:35,560 [salt.master      ][TRACE   ][21285] Master function call _master_opts took 0.00108003616333 seconds
2017-07-05 10:54:35,562 [salt.master      ][TRACE   ][21288] Master function call _file_envs took 0.000118970870972 seconds
2017-07-05 10:54:35,933 [salt.master      ][TRACE   ][21290] Master function call _file_list took 0.369281053543 seconds
2017-07-05 10:54:40,680 [salt.master      ][TRACE   ][21287] Master function call _pillar took 4.74016308784 seconds
2017-07-05 10:54:40,720 [salt.master      ][TRACE   ][21289] Master function call _file_envs took 0.00013279914856 seconds
2017-07-05 10:54:40,724 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000849008560181 seconds
2017-07-05 10:54:40,728 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.00121402740479 seconds
2017-07-05 10:54:40,791 [salt.master      ][TRACE   ][21290] Master function call _ext_nodes took 0.00403881072998 seconds
2017-07-05 10:54:40,803 [salt.master      ][TRACE   ][21287] Master function call _file_list took 0.00173497200012 seconds
2017-07-05 10:54:40,816 [salt.master      ][TRACE   ][21289] Master function call _file_list took 0.00174808502197 seconds
2017-07-05 10:54:40,829 [salt.master      ][TRACE   ][21285] Master function call _file_list took 0.00171494483948 seconds
2017-07-05 10:54:40,843 [salt.master      ][TRACE   ][21288] Master function call _file_list took 0.00203108787537 seconds
2017-07-05 10:54:40,856 [salt.master      ][TRACE   ][21290] Master function call _file_list took 0.00161004066467 seconds
2017-07-05 10:54:40,862 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000635862350464 seconds
2017-07-05 10:54:40,865 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000171184539795 seconds
2017-07-05 10:54:40,881 [salt.master      ][TRACE   ][21285] Master function call _file_list took 0.00313591957092 seconds
2017-07-05 10:54:40,897 [salt.master      ][TRACE   ][21288] Master function call _file_list took 0.00237393379211 seconds
2017-07-05 10:54:40,910 [salt.master      ][TRACE   ][21290] Master function call _file_list took 0.00171613693237 seconds
2017-07-05 10:54:40,924 [salt.master      ][TRACE   ][21287] Master function call _file_list took 0.00165390968323 seconds
2017-07-05 10:54:40,937 [salt.master      ][TRACE   ][21289] Master function call _file_list took 0.0016610622406 seconds
2017-07-05 10:54:40,948 [salt.master      ][TRACE   ][21285] Master function call _file_list took 0.00159001350403 seconds
2017-07-05 10:54:40,961 [salt.master      ][TRACE   ][21288] Master function call _file_list took 0.00158786773682 seconds
2017-07-05 10:54:40,999 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000393867492676 seconds
2017-07-05 10:54:41,002 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000256061553955 seconds
2017-07-05 10:54:41,005 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000571012496948 seconds
2017-07-05 10:54:41,008 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.00016188621521 seconds
2017-07-05 10:54:41,031 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000414848327637 seconds
2017-07-05 10:54:41,034 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000333070755005 seconds
2017-07-05 10:54:41,037 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000616073608398 seconds
2017-07-05 10:54:41,039 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000153064727783 seconds
2017-07-05 10:54:41,062 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000417947769165 seconds
2017-07-05 10:54:41,064 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000248908996582 seconds
2017-07-05 10:54:41,068 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000631809234619 seconds
2017-07-05 10:54:41,070 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000153064727783 seconds
2017-07-05 10:54:41,077 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000324964523315 seconds
2017-07-05 10:54:41,084 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000355005264282 seconds
2017-07-05 10:54:41,088 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000734090805054 seconds
2017-07-05 10:54:41,090 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000159978866577 seconds
2017-07-05 10:54:41,113 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00067400932312 seconds
2017-07-05 10:54:41,116 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.00019097328186 seconds
2017-07-05 10:54:41,129 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000661134719849 seconds
2017-07-05 10:54:41,132 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000211954116821 seconds
2017-07-05 10:54:41,151 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00068998336792 seconds
2017-07-05 10:54:41,155 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000209093093872 seconds
2017-07-05 10:54:41,198 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000648021697998 seconds
2017-07-05 10:54:41,202 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000510215759277 seconds
2017-07-05 10:54:41,210 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00058913230896 seconds
2017-07-05 10:54:41,213 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000181913375854 seconds
2017-07-05 10:54:41,293 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000684976577759 seconds
2017-07-05 10:54:41,303 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000202178955078 seconds
2017-07-05 10:54:41,335 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000670194625854 seconds
2017-07-05 10:54:41,338 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000198125839233 seconds
2017-07-05 10:54:41,363 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000665903091431 seconds
2017-07-05 10:54:41,365 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000190019607544 seconds
2017-07-05 10:54:41,384 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000652074813843 seconds
2017-07-05 10:54:41,388 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000197887420654 seconds
2017-07-05 10:54:41,411 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000652074813843 seconds
2017-07-05 10:54:41,415 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000257968902588 seconds
2017-07-05 10:54:41,424 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000582933425903 seconds
2017-07-05 10:54:41,433 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000197887420654 seconds
2017-07-05 10:54:41,563 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000323057174683 seconds
2017-07-05 10:54:41,566 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000318050384521 seconds
2017-07-05 10:54:41,569 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000659942626953 seconds
2017-07-05 10:54:41,572 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000211954116821 seconds
2017-07-05 10:54:41,590 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000679969787598 seconds
2017-07-05 10:54:41,593 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000197887420654 seconds
2017-07-05 10:54:41,599 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000699996948242 seconds
2017-07-05 10:54:41,601 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000182867050171 seconds
2017-07-05 10:54:41,608 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000607013702393 seconds
2017-07-05 10:54:41,610 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000170230865479 seconds
2017-07-05 10:54:41,621 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000602006912231 seconds
2017-07-05 10:54:41,623 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000163078308105 seconds
2017-07-05 10:54:41,629 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000609874725342 seconds
2017-07-05 10:54:41,632 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000180006027222 seconds
2017-07-05 10:54:41,644 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000709056854248 seconds
2017-07-05 10:54:41,646 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000177145004272 seconds
2017-07-05 10:54:41,663 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000620126724243 seconds
2017-07-05 10:54:41,665 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000189065933228 seconds
2017-07-05 10:54:41,671 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00058913230896 seconds
2017-07-05 10:54:41,673 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000188112258911 seconds
2017-07-05 10:54:41,690 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00059986114502 seconds
2017-07-05 10:54:41,692 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000163078308105 seconds
2017-07-05 10:54:41,707 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000653028488159 seconds
2017-07-05 10:54:41,709 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000166177749634 seconds
2017-07-05 10:54:41,715 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000560998916626 seconds
2017-07-05 10:54:41,718 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000169038772583 seconds
2017-07-05 10:54:41,733 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00040602684021 seconds
2017-07-05 10:54:41,735 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000245094299316 seconds
2017-07-05 10:54:41,744 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00238299369812 seconds
2017-07-05 10:54:41,747 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000197172164917 seconds
2017-07-05 10:54:41,754 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.0005202293396 seconds
2017-07-05 10:54:41,756 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000161170959473 seconds
2017-07-05 10:54:41,766 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000648021697998 seconds
2017-07-05 10:54:41,768 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000174045562744 seconds
2017-07-05 10:54:41,777 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000533103942871 seconds
2017-07-05 10:54:41,779 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000563859939575 seconds
2017-07-05 10:54:41,795 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000580072402954 seconds
2017-07-05 10:54:41,798 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000194072723389 seconds
2017-07-05 10:54:41,817 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000331163406372 seconds
2017-07-05 10:54:41,820 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000245094299316 seconds
2017-07-05 10:54:41,823 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000646114349365 seconds
2017-07-05 10:54:41,825 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000245809555054 seconds
2017-07-05 10:54:41,836 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000655174255371 seconds
2017-07-05 10:54:41,839 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000276803970337 seconds
2017-07-05 10:54:41,848 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000679016113281 seconds
2017-07-05 10:54:41,851 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000338077545166 seconds
2017-07-05 10:54:41,861 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000694990158081 seconds
2017-07-05 10:54:41,864 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000270128250122 seconds
2017-07-05 10:54:41,877 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000699996948242 seconds
2017-07-05 10:54:41,880 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000309944152832 seconds
2017-07-05 10:54:41,886 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000668048858643 seconds
2017-07-05 10:54:41,888 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000266075134277 seconds
2017-07-05 10:54:41,902 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000671863555908 seconds
2017-07-05 10:54:41,905 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000288009643555 seconds
2017-07-05 10:54:41,921 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000705003738403 seconds
2017-07-05 10:54:41,924 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000324010848999 seconds
2017-07-05 10:54:41,930 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000693082809448 seconds
2017-07-05 10:54:41,933 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000295877456665 seconds
2017-07-05 10:54:41,946 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000734090805054 seconds
2017-07-05 10:54:41,949 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000285863876343 seconds
2017-07-05 10:54:41,963 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000786066055298 seconds
2017-07-05 10:54:41,966 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000253915786743 seconds
2017-07-05 10:54:41,972 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000686883926392 seconds
2017-07-05 10:54:41,975 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000229835510254 seconds
2017-07-05 10:54:41,988 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00070595741272 seconds
2017-07-05 10:54:41,990 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000147104263306 seconds
2017-07-05 10:54:42,001 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000432968139648 seconds
2017-07-05 10:54:42,004 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000248193740845 seconds
2017-07-05 10:54:42,007 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000789165496826 seconds
2017-07-05 10:54:42,010 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000281095504761 seconds
2017-07-05 10:54:42,119 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000732898712158 seconds
2017-07-05 10:54:42,122 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000284910202026 seconds
2017-07-05 10:54:42,165 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000758171081543 seconds
2017-07-05 10:54:42,168 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000310182571411 seconds
2017-07-05 10:54:42,179 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000750064849854 seconds
2017-07-05 10:54:42,181 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.00025200843811 seconds
2017-07-05 10:54:42,198 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000367879867554 seconds
2017-07-05 10:54:42,201 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000277042388916 seconds
2017-07-05 10:54:42,203 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000484943389893 seconds
2017-07-05 10:54:42,206 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000128984451294 seconds
2017-07-05 10:54:42,211 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000338077545166 seconds
2017-07-05 10:54:42,214 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.00031304359436 seconds
2017-07-05 10:54:42,216 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000614166259766 seconds
2017-07-05 10:54:42,219 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000241994857788 seconds
2017-07-05 10:54:42,228 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000881910324097 seconds
2017-07-05 10:54:42,231 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000319004058838 seconds
2017-07-05 10:54:42,237 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000696897506714 seconds
2017-07-05 10:54:42,239 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000292062759399 seconds
2017-07-05 10:54:42,248 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000672817230225 seconds
2017-07-05 10:54:42,251 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000234127044678 seconds
2017-07-05 10:54:42,261 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000883817672729 seconds
2017-07-05 10:54:42,264 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000277996063232 seconds
2017-07-05 10:54:42,270 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00075888633728 seconds
2017-07-05 10:54:42,272 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000335931777954 seconds
2017-07-05 10:54:42,285 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000891923904419 seconds
2017-07-05 10:54:42,287 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000249862670898 seconds
2017-07-05 10:54:42,298 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000698089599609 seconds
2017-07-05 10:54:42,300 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000328063964844 seconds
2017-07-05 10:54:42,306 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000687122344971 seconds
2017-07-05 10:54:42,308 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.00032901763916 seconds
2017-07-05 10:54:42,324 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00068211555481 seconds
2017-07-05 10:54:42,326 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000251054763794 seconds
2017-07-05 10:54:42,336 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000856161117554 seconds
2017-07-05 10:54:42,339 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000298976898193 seconds
2017-07-05 10:54:42,344 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000643014907837 seconds
2017-07-05 10:54:42,346 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000247001647949 seconds
2017-07-05 10:54:42,356 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000545978546143 seconds
2017-07-05 10:54:42,359 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000152111053467 seconds
2017-07-05 10:54:42,370 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000336885452271 seconds
2017-07-05 10:54:42,372 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000248908996582 seconds
2017-07-05 10:54:42,375 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000509023666382 seconds
2017-07-05 10:54:42,377 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000157117843628 seconds
2017-07-05 10:54:42,387 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000655889511108 seconds
2017-07-05 10:54:42,389 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000266075134277 seconds
2017-07-05 10:54:42,395 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000728845596313 seconds
2017-07-05 10:54:42,397 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.00021505355835 seconds
2017-07-05 10:54:42,413 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00055718421936 seconds
2017-07-05 10:54:42,415 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000139951705933 seconds
2017-07-05 10:54:42,424 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000545024871826 seconds
2017-07-05 10:54:42,426 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000165224075317 seconds
2017-07-05 10:54:42,447 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000610828399658 seconds
2017-07-05 10:54:42,449 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000159978866577 seconds
2017-07-05 10:54:42,459 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000734806060791 seconds
2017-07-05 10:54:42,462 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000267028808594 seconds
2017-07-05 10:54:42,468 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000775098800659 seconds
2017-07-05 10:54:42,470 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000218152999878 seconds
2017-07-05 10:54:42,484 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000575065612793 seconds
2017-07-05 10:54:42,487 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000205993652344 seconds
2017-07-05 10:54:42,561 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000647068023682 seconds
2017-07-05 10:54:42,564 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000147104263306 seconds
2017-07-05 10:54:42,576 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000722885131836 seconds
2017-07-05 10:54:42,578 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000205039978027 seconds
2017-07-05 10:54:42,602 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000686883926392 seconds
2017-07-05 10:54:42,605 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000217914581299 seconds
2017-07-05 10:54:42,619 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000558137893677 seconds
2017-07-05 10:54:42,622 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000153064727783 seconds
2017-07-05 10:54:42,629 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000581026077271 seconds
2017-07-05 10:54:42,631 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000152826309204 seconds
2017-07-05 10:54:42,639 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000566005706787 seconds
2017-07-05 10:54:42,642 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000188112258911 seconds
2017-07-05 10:54:42,655 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000370979309082 seconds
2017-07-05 10:54:42,657 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000246047973633 seconds
2017-07-05 10:54:42,659 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000638008117676 seconds
2017-07-05 10:54:42,662 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000223159790039 seconds
2017-07-05 10:54:42,670 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000667810440063 seconds
2017-07-05 10:54:42,673 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000231981277466 seconds
2017-07-05 10:54:42,678 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000591993331909 seconds
2017-07-05 10:54:42,681 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000232934951782 seconds
2017-07-05 10:54:42,689 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000654935836792 seconds
2017-07-05 10:54:42,692 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000249862670898 seconds
2017-07-05 10:54:42,707 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00066614151001 seconds
2017-07-05 10:54:42,710 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000231027603149 seconds
2017-07-05 10:54:42,715 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000629901885986 seconds
2017-07-05 10:54:42,718 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000252962112427 seconds
2017-07-05 10:54:42,738 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000690937042236 seconds
2017-07-05 10:54:42,740 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000226974487305 seconds
2017-07-05 10:54:42,750 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000674962997437 seconds
2017-07-05 10:54:42,753 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000241994857788 seconds
2017-07-05 10:54:42,758 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000604867935181 seconds
2017-07-05 10:54:42,760 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000336885452271 seconds
2017-07-05 10:54:42,803 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000761985778809 seconds
2017-07-05 10:54:42,805 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000241041183472 seconds
2017-07-05 10:54:42,814 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000664949417114 seconds
2017-07-05 10:54:42,817 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000195980072021 seconds
2017-07-05 10:54:42,823 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000726938247681 seconds
2017-07-05 10:54:42,825 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000261068344116 seconds
2017-07-05 10:54:42,839 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000663995742798 seconds
2017-07-05 10:54:42,841 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000272989273071 seconds
2017-07-05 10:54:42,854 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000695943832397 seconds
2017-07-05 10:54:42,856 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000269889831543 seconds
2017-07-05 10:54:42,862 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000635862350464 seconds
2017-07-05 10:54:42,864 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000252962112427 seconds
2017-07-05 10:54:43,693 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000328063964844 seconds
2017-07-05 10:54:43,698 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000301122665405 seconds
2017-07-05 10:54:43,701 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000647068023682 seconds
2017-07-05 10:54:43,707 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000355958938599 seconds
2017-07-05 10:54:43,716 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000654935836792 seconds
2017-07-05 10:54:43,719 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000232934951782 seconds
2017-07-05 10:54:43,724 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000671148300171 seconds
2017-07-05 10:54:43,727 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000257968902588 seconds
2017-07-05 10:54:43,733 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000678062438965 seconds
2017-07-05 10:54:43,735 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000181198120117 seconds
2017-07-05 10:54:43,745 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000704050064087 seconds
2017-07-05 10:54:43,747 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000195980072021 seconds
2017-07-05 10:54:43,753 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000594854354858 seconds
2017-07-05 10:54:43,755 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000229120254517 seconds
2017-07-05 10:54:43,761 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000715017318726 seconds
2017-07-05 10:54:43,764 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000274896621704 seconds
2017-07-05 10:54:43,778 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000680208206177 seconds
2017-07-05 10:54:43,781 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000241041183472 seconds
2017-07-05 10:54:43,786 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00057578086853 seconds
2017-07-05 10:54:43,789 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.00025200843811 seconds
2017-07-05 10:54:43,808 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000589847564697 seconds
2017-07-05 10:54:43,811 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.00019097328186 seconds
2017-07-05 10:54:43,840 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000611066818237 seconds
2017-07-05 10:54:43,842 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000158071517944 seconds
2017-07-05 10:54:43,852 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000355005264282 seconds
2017-07-05 10:54:43,855 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000285863876343 seconds
2017-07-05 10:54:43,858 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000532865524292 seconds
2017-07-05 10:54:43,860 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000166177749634 seconds
2017-07-05 10:54:43,866 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000535011291504 seconds
2017-07-05 10:54:43,869 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000195980072021 seconds
2017-07-05 10:54:43,880 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000513076782227 seconds
2017-07-05 10:54:43,882 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000179052352905 seconds
2017-07-05 10:54:43,891 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000701904296875 seconds
2017-07-05 10:54:43,893 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.0001540184021 seconds
2017-07-05 10:54:43,909 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000571966171265 seconds
2017-07-05 10:54:43,912 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000153064727783 seconds
2017-07-05 10:54:45,338 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00054407119751 seconds
2017-07-05 10:54:45,340 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000455856323242 seconds
2017-07-05 10:54:45,343 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000160932540894 seconds
2017-07-05 10:54:45,351 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000555992126465 seconds
2017-07-05 10:54:45,354 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000540018081665 seconds
2017-07-05 10:54:45,356 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000162839889526 seconds
2017-07-05 10:54:46,452 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000563859939575 seconds
2017-07-05 10:54:46,454 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00045108795166 seconds
2017-07-05 10:54:46,456 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000149965286255 seconds
2017-07-05 10:54:46,466 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000540971755981 seconds
2017-07-05 10:54:46,468 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000155925750732 seconds
2017-07-05 10:54:46,475 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000712156295776 seconds
2017-07-05 10:54:46,477 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000152111053467 seconds
2017-07-05 10:54:46,486 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000592947006226 seconds
2017-07-05 10:54:46,489 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000471830368042 seconds
2017-07-05 10:54:46,491 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000144958496094 seconds
2017-07-05 10:54:46,500 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000516891479492 seconds
2017-07-05 10:54:46,502 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000158071517944 seconds
2017-07-05 10:54:46,508 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000545978546143 seconds
2017-07-05 10:54:46,510 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000177145004272 seconds
2017-07-05 10:54:46,903 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000639915466309 seconds
2017-07-05 10:54:46,906 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000499963760376 seconds
2017-07-05 10:54:46,908 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000187873840332 seconds
2017-07-05 10:54:47,021 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000625848770142 seconds
2017-07-05 10:54:47,024 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000476837158203 seconds
2017-07-05 10:54:47,026 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000159025192261 seconds
2017-07-05 10:54:47,119 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000710964202881 seconds
2017-07-05 10:54:47,122 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000510931015015 seconds
2017-07-05 10:54:47,125 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000232934951782 seconds
2017-07-05 10:54:53,599 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000530004501343 seconds
2017-07-05 10:54:53,602 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000444173812866 seconds
2017-07-05 10:54:53,604 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:54:53,670 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00058388710022 seconds
2017-07-05 10:54:53,672 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000402927398682 seconds
2017-07-05 10:54:53,674 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000113010406494 seconds
2017-07-05 10:54:53,712 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000641107559204 seconds
2017-07-05 10:54:53,715 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000520944595337 seconds
2017-07-05 10:54:53,717 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000182867050171 seconds
2017-07-05 10:54:53,727 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000600099563599 seconds
2017-07-05 10:54:53,729 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00021505355835 seconds
2017-07-05 10:54:53,769 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000602960586548 seconds
2017-07-05 10:54:53,772 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00049614906311 seconds
2017-07-05 10:54:53,774 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000182151794434 seconds
2017-07-05 10:54:53,792 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000638961791992 seconds
2017-07-05 10:54:53,795 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000576972961426 seconds
2017-07-05 10:54:53,797 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000223159790039 seconds
2017-07-05 10:54:53,831 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000607013702393 seconds
2017-07-05 10:54:53,834 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000527858734131 seconds
2017-07-05 10:54:53,836 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000171899795532 seconds
2017-07-05 10:54:53,854 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000560998916626 seconds
2017-07-05 10:54:53,857 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000498056411743 seconds
2017-07-05 10:54:53,860 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000225067138672 seconds
2017-07-05 10:54:53,879 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000672817230225 seconds
2017-07-05 10:54:53,882 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000547885894775 seconds
2017-07-05 10:54:53,884 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000310897827148 seconds
2017-07-05 10:54:53,928 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00075888633728 seconds
2017-07-05 10:54:53,931 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000591039657593 seconds
2017-07-05 10:54:53,934 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000236988067627 seconds
2017-07-05 10:54:53,952 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000629901885986 seconds
2017-07-05 10:54:53,955 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000469923019409 seconds
2017-07-05 10:54:53,957 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000178813934326 seconds
2017-07-05 10:54:53,975 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000590085983276 seconds
2017-07-05 10:54:53,978 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000480890274048 seconds
2017-07-05 10:54:53,980 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000166893005371 seconds
2017-07-05 10:54:53,999 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00081205368042 seconds
2017-07-05 10:54:54,002 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000675916671753 seconds
2017-07-05 10:54:54,005 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000227928161621 seconds
2017-07-05 10:54:54,024 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000627994537354 seconds
2017-07-05 10:54:54,028 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00052285194397 seconds
2017-07-05 10:54:54,030 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000160932540894 seconds
2017-07-05 10:54:54,052 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000573873519897 seconds
2017-07-05 10:54:54,055 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000521183013916 seconds
2017-07-05 10:54:54,058 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000174045562744 seconds
2017-07-05 10:54:54,077 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00062894821167 seconds
2017-07-05 10:54:54,080 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000489950180054 seconds
2017-07-05 10:54:54,083 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000176906585693 seconds
2017-07-05 10:54:54,101 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000572919845581 seconds
2017-07-05 10:54:54,105 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000511884689331 seconds
2017-07-05 10:54:54,107 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000170946121216 seconds
2017-07-05 10:54:54,125 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000610113143921 seconds
2017-07-05 10:54:54,128 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000594854354858 seconds
2017-07-05 10:54:54,130 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000166893005371 seconds
2017-07-05 10:54:54,149 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00070595741272 seconds
2017-07-05 10:54:54,152 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000523090362549 seconds
2017-07-05 10:54:54,155 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00018310546875 seconds
2017-07-05 10:54:54,175 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000646829605103 seconds
2017-07-05 10:54:54,179 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000516891479492 seconds
2017-07-05 10:54:54,181 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000189065933228 seconds
2017-07-05 10:54:54,217 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000643014907837 seconds
2017-07-05 10:54:54,220 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000588178634644 seconds
2017-07-05 10:54:54,222 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000176191329956 seconds
2017-07-05 10:54:54,340 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000603914260864 seconds
2017-07-05 10:54:54,343 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000552177429199 seconds
2017-07-05 10:54:54,345 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000181198120117 seconds
2017-07-05 10:54:54,362 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000576019287109 seconds
2017-07-05 10:54:54,365 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000648975372314 seconds
2017-07-05 10:54:54,367 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000212907791138 seconds
2017-07-05 10:54:54,384 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00065803527832 seconds
2017-07-05 10:54:54,387 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000648975372314 seconds
2017-07-05 10:54:54,389 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00025486946106 seconds
2017-07-05 10:54:54,405 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000725984573364 seconds
2017-07-05 10:54:54,408 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000604152679443 seconds
2017-07-05 10:54:54,410 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000204801559448 seconds
2017-07-05 10:54:54,426 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00064492225647 seconds
2017-07-05 10:54:54,429 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000434875488281 seconds
2017-07-05 10:54:54,431 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000211000442505 seconds
2017-07-05 10:54:54,448 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000636100769043 seconds
2017-07-05 10:54:54,452 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000773906707764 seconds
2017-07-05 10:54:54,455 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000272989273071 seconds
2017-07-05 10:54:54,471 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000653982162476 seconds
2017-07-05 10:54:54,474 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000679016113281 seconds
2017-07-05 10:54:54,476 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.00016713142395 seconds
2017-07-05 10:54:54,492 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000615119934082 seconds
2017-07-05 10:54:54,495 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000519037246704 seconds
2017-07-05 10:54:54,497 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000238180160522 seconds
2017-07-05 10:54:54,514 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000900030136108 seconds
2017-07-05 10:54:54,518 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000597953796387 seconds
2017-07-05 10:54:54,520 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000235080718994 seconds
2017-07-05 10:54:54,538 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000745058059692 seconds
2017-07-05 10:54:54,541 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000554084777832 seconds
2017-07-05 10:54:54,544 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000200986862183 seconds
2017-07-05 10:54:54,559 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000669956207275 seconds
2017-07-05 10:54:54,562 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000525951385498 seconds
2017-07-05 10:54:54,564 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000195026397705 seconds
2017-07-05 10:54:57,302 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00186491012573 seconds
2017-07-05 10:54:57,305 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000576972961426 seconds
2017-07-05 10:54:57,308 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000134944915771 seconds
2017-07-05 10:54:58,306 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.0010199546814 seconds
2017-07-05 10:54:58,309 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00065803527832 seconds
2017-07-05 10:54:58,311 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000201940536499 seconds
2017-07-05 10:54:58,322 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000649929046631 seconds
2017-07-05 10:54:58,324 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000244140625 seconds
2017-07-05 10:54:58,330 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000860214233398 seconds
2017-07-05 10:54:58,333 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000280141830444 seconds
2017-07-05 10:54:58,339 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000761985778809 seconds
2017-07-05 10:54:58,342 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000245809555054 seconds
2017-07-05 10:54:58,366 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000813007354736 seconds
2017-07-05 10:54:58,369 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00077486038208 seconds
2017-07-05 10:54:58,371 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000266075134277 seconds
2017-07-05 10:54:58,380 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000655889511108 seconds
2017-07-05 10:54:58,382 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000200986862183 seconds
2017-07-05 10:54:58,387 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000695943832397 seconds
2017-07-05 10:54:58,390 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.00019907951355 seconds
2017-07-05 10:54:58,415 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000519037246704 seconds
2017-07-05 10:54:58,418 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000447988510132 seconds
2017-07-05 10:54:58,421 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000123023986816 seconds
2017-07-05 10:54:58,700 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000717163085938 seconds
2017-07-05 10:54:58,703 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000720977783203 seconds
2017-07-05 10:54:58,705 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000207185745239 seconds
2017-07-05 10:54:58,716 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000741004943848 seconds
2017-07-05 10:54:58,719 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000240087509155 seconds
2017-07-05 10:54:58,725 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000815153121948 seconds
2017-07-05 10:54:58,727 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000418901443481 seconds
2017-07-05 10:54:58,736 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000852823257446 seconds
2017-07-05 10:54:58,739 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00031304359436 seconds
2017-07-05 10:54:58,761 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000600099563599 seconds
2017-07-05 10:54:58,765 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000666856765747 seconds
2017-07-05 10:54:58,768 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000162839889526 seconds
2017-07-05 10:54:59,002 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000543117523193 seconds
2017-07-05 10:54:59,006 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000531911849976 seconds
2017-07-05 10:54:59,008 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000123977661133 seconds
2017-07-05 10:54:59,420 [salt.master      ][TRACE   ][21289] Master function call _file_list took 0.275437116623 seconds
2017-07-05 10:54:59,425 [salt.master      ][TRACE   ][21285] Master function call _dir_list took 0.000802040100098 seconds
2017-07-05 10:54:59,429 [salt.master      ][TRACE   ][21288] Master function call _dir_list took 0.000735998153687 seconds
2017-07-05 10:54:59,432 [salt.master      ][TRACE   ][21290] Master function call _file_list took 0.00150299072266 seconds
2017-07-05 10:54:59,436 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000520944595337 seconds
2017-07-05 10:54:59,438 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00041389465332 seconds
2017-07-05 10:54:59,440 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000118970870972 seconds
2017-07-05 10:54:59,444 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000453948974609 seconds
2017-07-05 10:54:59,446 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00043797492981 seconds
2017-07-05 10:54:59,448 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000114917755127 seconds
2017-07-05 10:54:59,451 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000454902648926 seconds
2017-07-05 10:54:59,454 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000426054000854 seconds
2017-07-05 10:54:59,456 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000173091888428 seconds
2017-07-05 10:54:59,460 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000508785247803 seconds
2017-07-05 10:54:59,462 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000455856323242 seconds
2017-07-05 10:54:59,464 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000118017196655 seconds
2017-07-05 10:54:59,468 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000635147094727 seconds
2017-07-05 10:54:59,471 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000577926635742 seconds
2017-07-05 10:54:59,473 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000113010406494 seconds
2017-07-05 10:54:59,476 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000517129898071 seconds
2017-07-05 10:54:59,479 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000590801239014 seconds
2017-07-05 10:54:59,481 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000126838684082 seconds
2017-07-05 10:54:59,485 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000612020492554 seconds
2017-07-05 10:54:59,487 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000452995300293 seconds
2017-07-05 10:54:59,489 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000121116638184 seconds
2017-07-05 10:54:59,493 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00055193901062 seconds
2017-07-05 10:54:59,495 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000493049621582 seconds
2017-07-05 10:54:59,497 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000120162963867 seconds
2017-07-05 10:54:59,501 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000782012939453 seconds
2017-07-05 10:54:59,504 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.0010199546814 seconds
2017-07-05 10:54:59,506 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000127077102661 seconds
2017-07-05 10:54:59,511 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000607013702393 seconds
2017-07-05 10:54:59,513 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000498056411743 seconds
2017-07-05 10:54:59,515 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000127077102661 seconds
2017-07-05 10:54:59,519 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000546216964722 seconds
2017-07-05 10:54:59,522 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000479936599731 seconds
2017-07-05 10:54:59,524 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000132083892822 seconds
2017-07-05 10:54:59,528 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000678062438965 seconds
2017-07-05 10:54:59,531 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000547885894775 seconds
2017-07-05 10:54:59,533 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000130176544189 seconds
2017-07-05 10:54:59,536 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00056791305542 seconds
2017-07-05 10:54:59,539 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000538110733032 seconds
2017-07-05 10:54:59,541 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000121831893921 seconds
2017-07-05 10:54:59,544 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00055718421936 seconds
2017-07-05 10:54:59,547 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000494956970215 seconds
2017-07-05 10:54:59,549 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000137090682983 seconds
2017-07-05 10:54:59,553 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000880002975464 seconds
2017-07-05 10:54:59,556 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000460863113403 seconds
2017-07-05 10:54:59,558 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000144958496094 seconds
2017-07-05 10:54:59,562 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000643968582153 seconds
2017-07-05 10:54:59,565 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000626087188721 seconds
2017-07-05 10:54:59,567 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000146150588989 seconds
2017-07-05 10:54:59,571 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000680923461914 seconds
2017-07-05 10:54:59,574 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000516891479492 seconds
2017-07-05 10:54:59,576 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000133991241455 seconds
2017-07-05 10:54:59,581 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000661134719849 seconds
2017-07-05 10:54:59,583 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00061297416687 seconds
2017-07-05 10:54:59,585 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000118017196655 seconds
2017-07-05 10:54:59,589 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000555038452148 seconds
2017-07-05 10:54:59,592 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00053882598877 seconds
2017-07-05 10:54:59,594 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000136137008667 seconds
2017-07-05 10:54:59,603 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00603604316711 seconds
2017-07-05 10:54:59,611 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00561690330505 seconds
2017-07-05 10:54:59,613 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000138998031616 seconds
2017-07-05 10:54:59,632 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000799179077148 seconds
2017-07-05 10:54:59,635 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000715017318726 seconds
2017-07-05 10:54:59,637 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000149965286255 seconds
2017-07-05 10:54:59,641 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000667095184326 seconds
2017-07-05 10:54:59,644 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000765085220337 seconds
2017-07-05 10:54:59,646 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000119209289551 seconds
2017-07-05 10:54:59,650 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000646114349365 seconds
2017-07-05 10:54:59,653 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000625133514404 seconds
2017-07-05 10:54:59,654 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000113964080811 seconds
2017-07-05 10:54:59,659 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000801086425781 seconds
2017-07-05 10:54:59,661 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000735998153687 seconds
2017-07-05 10:54:59,663 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000104904174805 seconds
2017-07-05 10:54:59,667 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000483989715576 seconds
2017-07-05 10:54:59,670 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000393152236938 seconds
2017-07-05 10:54:59,671 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000107049942017 seconds
2017-07-05 10:54:59,675 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000447988510132 seconds
2017-07-05 10:54:59,677 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000396013259888 seconds
2017-07-05 10:54:59,679 [salt.master      ][TRACE   ][21285] Master function call _file_find took 9.89437103271e-05 seconds
2017-07-05 10:54:59,682 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000442981719971 seconds
2017-07-05 10:54:59,685 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000417947769165 seconds
2017-07-05 10:54:59,686 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000116109848022 seconds
2017-07-05 10:54:59,690 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000491857528687 seconds
2017-07-05 10:54:59,692 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000434875488281 seconds
2017-07-05 10:54:59,694 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000104904174805 seconds
2017-07-05 10:54:59,697 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000408887863159 seconds
2017-07-05 10:54:59,699 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00042200088501 seconds
2017-07-05 10:54:59,701 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:54:59,704 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000439882278442 seconds
2017-07-05 10:54:59,707 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000470876693726 seconds
2017-07-05 10:54:59,709 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000148057937622 seconds
2017-07-05 10:54:59,714 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000607967376709 seconds
2017-07-05 10:54:59,716 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000452995300293 seconds
2017-07-05 10:54:59,718 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000110864639282 seconds
2017-07-05 10:54:59,721 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000473022460938 seconds
2017-07-05 10:54:59,724 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00039005279541 seconds
2017-07-05 10:54:59,726 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000111818313599 seconds
2017-07-05 10:54:59,729 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000418901443481 seconds
2017-07-05 10:54:59,731 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000398874282837 seconds
2017-07-05 10:54:59,733 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000105857849121 seconds
2017-07-05 10:54:59,736 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000419139862061 seconds
2017-07-05 10:54:59,738 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000414133071899 seconds
2017-07-05 10:54:59,740 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:54:59,743 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000437021255493 seconds
2017-07-05 10:54:59,745 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000425100326538 seconds
2017-07-05 10:54:59,747 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000113964080811 seconds
2017-07-05 10:54:59,750 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000486850738525 seconds
2017-07-05 10:54:59,753 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000443935394287 seconds
2017-07-05 10:54:59,755 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000118970870972 seconds
2017-07-05 10:54:59,758 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000434875488281 seconds
2017-07-05 10:54:59,760 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000412940979004 seconds
2017-07-05 10:54:59,762 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000108957290649 seconds
2017-07-05 10:54:59,765 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000397920608521 seconds
2017-07-05 10:54:59,767 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000391960144043 seconds
2017-07-05 10:54:59,769 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000104188919067 seconds
2017-07-05 10:54:59,772 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000395059585571 seconds
2017-07-05 10:54:59,774 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000427007675171 seconds
2017-07-05 10:54:59,776 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000107049942017 seconds
2017-07-05 10:54:59,779 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000431060791016 seconds
2017-07-05 10:54:59,781 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000414133071899 seconds
2017-07-05 10:54:59,783 [salt.master      ][TRACE   ][21290] Master function call _file_find took 9.91821289062e-05 seconds
2017-07-05 10:54:59,786 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000440120697021 seconds
2017-07-05 10:54:59,788 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000401973724365 seconds
2017-07-05 10:54:59,790 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000100135803223 seconds
2017-07-05 10:54:59,793 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000410079956055 seconds
2017-07-05 10:54:59,795 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000425100326538 seconds
2017-07-05 10:54:59,797 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000106811523438 seconds
2017-07-05 10:54:59,800 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000419855117798 seconds
2017-07-05 10:54:59,802 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000408172607422 seconds
2017-07-05 10:54:59,804 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000102996826172 seconds
2017-07-05 10:54:59,807 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000410079956055 seconds
2017-07-05 10:54:59,809 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00040602684021 seconds
2017-07-05 10:54:59,811 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000102996826172 seconds
2017-07-05 10:54:59,814 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000401020050049 seconds
2017-07-05 10:54:59,816 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00036096572876 seconds
2017-07-05 10:54:59,818 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000103950500488 seconds
2017-07-05 10:54:59,821 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000411033630371 seconds
2017-07-05 10:54:59,823 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.00041389465332 seconds
2017-07-05 10:54:59,825 [salt.master      ][TRACE   ][21285] Master function call _file_find took 9.60826873779e-05 seconds
2017-07-05 10:54:59,828 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000403881072998 seconds
2017-07-05 10:54:59,830 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000380039215088 seconds
2017-07-05 10:54:59,831 [salt.master      ][TRACE   ][21287] Master function call _file_find took 9.91821289062e-05 seconds
2017-07-05 10:54:59,834 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000351190567017 seconds
2017-07-05 10:54:59,836 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00040602684021 seconds
2017-07-05 10:54:59,837 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000105857849121 seconds
2017-07-05 10:54:59,840 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000385999679565 seconds
2017-07-05 10:54:59,843 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00039005279541 seconds
2017-07-05 10:54:59,845 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000113964080811 seconds
2017-07-05 10:54:59,847 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000482082366943 seconds
2017-07-05 10:54:59,850 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000427007675171 seconds
2017-07-05 10:54:59,851 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00011682510376 seconds
2017-07-05 10:54:59,854 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000447034835815 seconds
2017-07-05 10:54:59,856 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000423908233643 seconds
2017-07-05 10:54:59,858 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000107049942017 seconds
2017-07-05 10:54:59,861 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000471830368042 seconds
2017-07-05 10:54:59,864 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000550031661987 seconds
2017-07-05 10:54:59,865 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000103950500488 seconds
2017-07-05 10:54:59,868 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000398874282837 seconds
2017-07-05 10:54:59,871 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000419139862061 seconds
2017-07-05 10:54:59,872 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:54:59,875 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00037693977356 seconds
2017-07-05 10:54:59,878 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000396966934204 seconds
2017-07-05 10:54:59,879 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000102043151855 seconds
2017-07-05 10:54:59,883 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000452995300293 seconds
2017-07-05 10:54:59,885 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000397920608521 seconds
2017-07-05 10:54:59,887 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.00010085105896 seconds
2017-07-05 10:54:59,890 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000481843948364 seconds
2017-07-05 10:54:59,893 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000462055206299 seconds
2017-07-05 10:54:59,895 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000104904174805 seconds
2017-07-05 10:54:59,899 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.00103306770325 seconds
2017-07-05 10:54:59,902 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000885009765625 seconds
2017-07-05 10:54:59,903 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000120162963867 seconds
2017-07-05 10:54:59,910 [salt.master      ][TRACE   ][21289] Master function call _file_list took 0.00150513648987 seconds
2017-07-05 10:54:59,914 [salt.master      ][TRACE   ][21285] Master function call _dir_list took 0.00067400932312 seconds
2017-07-05 10:54:59,916 [salt.master      ][TRACE   ][21288] Master function call _dir_list took 0.000714063644409 seconds
2017-07-05 10:54:59,920 [salt.master      ][TRACE   ][21290] Master function call _file_list took 0.00145888328552 seconds
2017-07-05 10:54:59,923 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00048303604126 seconds
2017-07-05 10:54:59,925 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000403165817261 seconds
2017-07-05 10:54:59,926 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.00010085105896 seconds
2017-07-05 10:54:59,929 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000442981719971 seconds
2017-07-05 10:54:59,931 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000358819961548 seconds
2017-07-05 10:54:59,933 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000123977661133 seconds
2017-07-05 10:54:59,936 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000379085540771 seconds
2017-07-05 10:54:59,938 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00036096572876 seconds
2017-07-05 10:54:59,940 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000102043151855 seconds
2017-07-05 10:54:59,943 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000406980514526 seconds
2017-07-05 10:54:59,945 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000378847122192 seconds
2017-07-05 10:54:59,947 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000121116638184 seconds
2017-07-05 10:54:59,949 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.00040602684021 seconds
2017-07-05 10:54:59,951 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000383853912354 seconds
2017-07-05 10:54:59,953 [salt.master      ][TRACE   ][21290] Master function call _file_find took 9.79900360107e-05 seconds
2017-07-05 10:54:59,979 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.00057578086853 seconds
2017-07-05 10:54:59,982 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000433921813965 seconds
2017-07-05 10:54:59,984 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000111103057861 seconds
2017-07-05 10:54:59,989 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000411987304688 seconds
2017-07-05 10:54:59,991 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00040602684021 seconds
2017-07-05 10:54:59,993 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000105857849121 seconds
2017-07-05 10:55:00,004 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000591039657593 seconds
2017-07-05 10:55:00,007 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000462055206299 seconds
2017-07-05 10:55:00,008 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000109910964966 seconds
2017-07-05 10:55:00,013 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.000553846359253 seconds
2017-07-05 10:55:00,016 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000412940979004 seconds
2017-07-05 10:55:00,018 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:55:00,022 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000424146652222 seconds
2017-07-05 10:55:00,024 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000400066375732 seconds
2017-07-05 10:55:00,026 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000111103057861 seconds
2017-07-05 10:55:00,211 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000483989715576 seconds
2017-07-05 10:55:00,213 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000385999679565 seconds
2017-07-05 10:55:00,215 [salt.master      ][TRACE   ][21285] Master function call _file_find took 0.000106811523438 seconds
2017-07-05 10:55:00,219 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000447034835815 seconds
2017-07-05 10:55:00,222 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00042986869812 seconds
2017-07-05 10:55:00,223 [salt.master      ][TRACE   ][21287] Master function call _file_find took 0.000108003616333 seconds
2017-07-05 10:55:00,245 [salt.master      ][TRACE   ][21289] Master function call _file_hash took 0.000514984130859 seconds
2017-07-05 10:55:00,248 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000380992889404 seconds
2017-07-05 10:55:00,250 [salt.master      ][TRACE   ][21288] Master function call _file_find took 0.000123977661133 seconds
2017-07-05 10:55:00,257 [salt.master      ][TRACE   ][21290] Master function call _file_hash took 0.00045108795166 seconds
2017-07-05 10:55:00,259 [salt.master      ][TRACE   ][21287] Master function call _file_hash took 0.000427007675171 seconds
2017-07-05 10:55:00,261 [salt.master      ][TRACE   ][21289] Master function call _file_find took 0.000111103057861 seconds
2017-07-05 10:55:00,284 [salt.master      ][TRACE   ][21285] Master function call _file_hash took 0.000470876693726 seconds
2017-07-05 10:55:00,286 [salt.master      ][TRACE   ][21288] Master function call _file_hash took 0.000377893447876 seconds
2017-07-05 10:55:00,288 [salt.master      ][TRACE   ][21290] Master function call _file_find took 0.000102043151855 seconds
2017-07-05 10:55:00,399 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00830221176147 seconds
2017-07-05 10:55:03,000 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00218987464905 seconds
2017-07-05 10:55:03,008 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00569987297058 seconds
2017-07-05 10:55:03,012 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00123715400696 seconds
2017-07-05 10:55:03,032 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00129199028015 seconds
2017-07-05 10:55:03,041 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00344085693359 seconds
2017-07-05 10:55:03,044 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00106978416443 seconds
2017-07-05 10:55:03,046 [salt.master      ][TRACE   ][21288] Master function call _return took 0.000937938690186 seconds
2017-07-05 10:55:03,049 [salt.master      ][TRACE   ][21288] Master function call _return took 0.000891923904419 seconds
2017-07-05 10:55:03,054 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00106620788574 seconds
2017-07-05 10:55:03,060 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00101804733276 seconds
2017-07-05 10:55:03,068 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00113892555237 seconds
2017-07-05 10:55:03,079 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00115203857422 seconds
2017-07-05 10:55:03,116 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00116682052612 seconds
2017-07-05 10:55:03,131 [salt.master      ][TRACE   ][21290] Master function call _return took 0.0122621059418 seconds
2017-07-05 10:55:03,134 [salt.master      ][TRACE   ][21285] Master function call _return took 0.00370502471924 seconds
2017-07-05 10:55:03,136 [salt.master      ][TRACE   ][21285] Master function call _return took 0.000995874404907 seconds
2017-07-05 10:55:03,140 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00581812858582 seconds
2017-07-05 10:55:03,142 [salt.master      ][TRACE   ][21285] Master function call _return took 0.00601816177368 seconds
2017-07-05 10:55:03,145 [salt.master      ][TRACE   ][21285] Master function call _return took 0.00141000747681 seconds
2017-07-05 10:55:03,146 [salt.master      ][TRACE   ][21285] Master function call _return took 0.00109791755676 seconds
2017-07-05 10:55:03,148 [salt.master      ][TRACE   ][21285] Master function call _return took 0.000952959060669 seconds
2017-07-05 10:55:03,152 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00738096237183 seconds
2017-07-05 10:55:03,153 [salt.master      ][TRACE   ][21289] Master function call _return took 0.0158648490906 seconds
2017-07-05 10:55:03,154 [salt.master      ][TRACE   ][21285] Master function call _return took 0.000977993011475 seconds
2017-07-05 10:55:03,160 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00105404853821 seconds
2017-07-05 10:55:03,160 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00371098518372 seconds
2017-07-05 10:55:03,163 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00743889808655 seconds
2017-07-05 10:55:03,166 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00166606903076 seconds
2017-07-05 10:55:03,168 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00152206420898 seconds
2017-07-05 10:55:03,171 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00969099998474 seconds
2017-07-05 10:55:03,173 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00344777107239 seconds
2017-07-05 10:55:03,178 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00399279594421 seconds
2017-07-05 10:55:03,178 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00563597679138 seconds
2017-07-05 10:55:03,180 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00127506256104 seconds
2017-07-05 10:55:03,183 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00422811508179 seconds
2017-07-05 10:55:03,186 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00173807144165 seconds
2017-07-05 10:55:03,190 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00113892555237 seconds
2017-07-05 10:55:03,192 [salt.master      ][TRACE   ][21289] Master function call _return took 0.00117802619934 seconds
2017-07-05 10:55:03,196 [salt.master      ][TRACE   ][21285] Master function call _return took 0.00141286849976 seconds
2017-07-05 10:55:03,199 [salt.master      ][TRACE   ][21288] Master function call _return took 0.00183606147766 seconds
2017-07-05 10:55:03,220 [salt.master      ][TRACE   ][21290] Master function call _return took 0.00178718566895 seconds
2017-07-05 10:55:03,232 [salt.master      ][TRACE   ][21287] Master function call _return took 0.00176095962524 seconds

ghost on 5 Jul 2017

Once thing that is noticeable is that when using GitFS file_envs takes almost 10 seconds:

2017-07-05 10:49:44,229 [salt.master      ][TRACE   ][28595] Master function call _file_envs took 8.79764556885e-05 seconds

compared to file_roots which takes fractions of a second:

2017-07-05 10:54:35,562 [salt.master      ][TRACE   ][21288] Master function call _file_envs took 0.000118970870972 seconds

ghost on 5 Jul 2017

No, that's exponent notation, so 8.79764556885e-05 is 8.79764556885 * 10^-5, or .0000879764556885. This makes it actually a bit faster than file_roots. The reason for this is that we use caching, since compiling a list of environments from all the branches/tags is a computationally expensive action which needs to be done often, and environments rarely change so we update that cache whenever the fileserver updates and realizes that it has fetched new content.

One thing that is taking substantially longer than expected is pillar generation. You said that you're using git_pillar, can you comment out your git_pillar (or temporarily replace it with pillar_roots) and see if this affects the CPU usage?

terminalmage on 5 Jul 2017

@terminalmage we are seeing similar issues. Also running salt 2016.11.6. We have 46 gitfs roots and 720 or so minions running a scheduled highstate every 30 minutes.

With gitfs enabled the salt master load average is through the roof. With 15 worker threads the load is pegged at 15.00 100% of the time. We copy the files locally and remove gitfs the box is 90% idle.

You can see here where I made the changes to remove gitfs, but then our salt formula reverted the changes I made to our salt master on the next highstate run:

06:10:01 PM     all      1.51     74.12     24.37      0.00      0.00      0.00
06:20:02 PM     all      1.03     76.02     22.95      0.00      0.00      0.00
06:30:01 PM     all      1.28     75.61     23.11      0.00      0.00      0.00
06:40:01 PM     all      1.45     75.35     23.20      0.00      0.00      0.00
06:50:02 PM     all      0.96     76.13     22.91      0.00      0.00      0.00
07:00:01 PM     all     19.16     11.54     10.53      0.13      0.00     58.65

07:00:01 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:10:01 PM     all     16.54      0.45      6.22      0.09      0.00     76.70
07:20:01 PM     all     15.81      0.37      2.54      0.14      0.00     81.14
07:30:01 PM     all     15.53      0.38      2.76      0.11      0.00     81.21
07:40:01 PM     all      1.31     68.72     19.51      0.02      0.00     10.44
07:50:01 PM     all      0.92     77.49     20.35      0.00      0.00      1.24
08:00:02 PM     all      0.92     78.99     20.01      0.00      0.00      0.08
Average:        all      1.85     73.21     22.37      0.00      0.00      2.56

Looking at top, it's definitely salt-master processes that are eating the CPU. Looking at strace on any of the busy processes shows a ton of file activity on /var/cache/salt/master/gitfs/

lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/cache", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/cache/salt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/cache/salt/master", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var/cache/salt/master/gitfs", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0
lstat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/objects/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/HEAD", {st_mode=S_IFREG|0644, st_size=23, ...}) = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/refs/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
access("/etc/gitconfig", F_OK)          = -1 ENOENT (No such file or directory)
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/config", {st_mode=S_IFREG|0644, st_size=281, ...}) = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/config", {st_mode=S_IFREG|0644, st_size=281, ...}) = 0
open("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/.git/config", O_RDONLY|O_CLOEXEC) = 45
read(45, "[core]\n\tbare = false\n\trepository"..., 281) = 281
close(45)                               = 0
stat("/var/cache/salt/master/gitfs/3e98b8f06b91e927db7d046d6706db9194ab6fadb1c75153a45a8871b0451f3634b9008866074413a0319f1f479c613b9b1bb069e69cba7bfda6edd5a2e3c9c7/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/root/.ssh/id_rsa.pub", {st_mode=S_IFREG|0644, st_size=410, ...}) = 0
stat("/root/.ssh/id_rsa", {st_mode=S_IFREG|0600, st_size=1675, ...}) = 0
stat("/var/cache/salt/master/gitfs/66abe8be310fc0169a439a6b54e48945f454309c9269eba61f8771f29b6e88a441293ff622f906846deb3c248edba498feaea5d80df885356cc5831902cc6dbd", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/cache/salt/master/gitfs/66abe8be310fc0169a439a6b54e48945f454309c9269eba61f8771f29b6e88a441293ff622f906846deb3c248edba498feaea5d80df885356cc5831902cc6dbd", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 45
getdents(45, /* 3 entries */, 32768)    = 72
getdents(45, /* 0 entries */, 32768)    = 0
close(45)                               = 0

This just spams over and over and over again thousands of times per second.

mikeadamz on 11 Oct 2017

@mikeadamz OK, but I still don't really know anything about your setup. Are you using GitPython or Pygit2? Where is the versions report?

terminalmage on 12 Oct 2017

@terminalmage we're using pygit2 on Centos 6.9

Salt Version:
           Salt: 2016.11.6

Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 1.4.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.20.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: 0.20.3
         Python: 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5

System Versions:
           dist: centos 6.9 Final
        machine: x86_64
        release: 2.6.32-696.10.1.el6.x86_64
         system: Linux
        version: CentOS 6.9 Final

mikeadamz on 12 Oct 2017

It appears my environment is impacted by this issue as well. I am also able to reproduce the problem by enabling gitfs and eliminate it by disabling. @grobinson-blockchain are you still using a git pull cron job with local roots to get around this issue?

We employ state scheduling in my environment so load is pegged at 14 while gitfs is enabled on a system with 4 CPUs and 32 salt-master processes. However, general system performance is not impacted outside of salt state application slowness and gitfs errors in the salt master log.

dustinmcnabb on 13 Oct 2017

@mikeadamz 46 is a lot of remotes. Are there some repos that are used much more than others? If so, where are they in your gitfs_remotes list? Keep in mind that when gitfs looks for files, it doesn't know which repo contains the file, so it tries each repo in turn until it finds a match.

terminalmage on 13 Oct 2017

@terminalmage unfortunately no, most states are applied to all hosts. We just have everything broken up into separate formulas and each formula has its own git repository. To make matters worse, we're not done. We're adding 5-6 formulas per week based on code we're migrating away from $oldtool

mikeadamz on 13 Oct 2017

@terminalmage Since salt's own gitfs documentation gives examples of formula specific remotes I wouldn't think that a few dozen would be considered a lot of remotes for any serious user of this application. Assuming users want to manage all aspects of their system and application configuration with this tool I would expect hundreds, possibly thousands of remotes in a fully baked out environment. I suppose there might be some clue I am missing, but this definitely seems like a limitation worth fixing right? https://docs.saltstack.com/en/latest/topics/tutorials/gitfs.html#simple-configuration

dustinmcnabb on 13 Oct 2017

Still running into this behavior in our setup with 23 remotes (gitfs and git pillar). Anytime we run operations on our master (or syndic) we see a huge spike in CPU and the logs are spammed with: Re-using gitfs object for process <pid>

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.11.5
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.27.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.1
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.27.0
         Python: 3.4.8 (default, Mar 23 2018, 10:04:27)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.5.1804 Core
         locale: UTF-8
        machine: x86_64
        release: 4.4.135-1.el7.elrepo.x86_64
         system: Linux
        version: CentOS Linux 7.5.1804 Core

fooka03 on 10 Jun 2018

Yes, this is a known issue for the reasons I've described above.

The fact is that gitfs wasn't designed to be used with dozens of remotes, so it needs to adapt to this kind of usage. One way that we plan to do so is to use our pluggable cache subsystem to drasically reduce seeking for files. If we locate a file once, we can then pull its location out of cache instead of seeking again the next time the file is requested. The cache would then be flushed when new files are fetched. Ideally we'd just be able to flush cache entries from a given gitfs remote (i.e. the one which had updates fetched).

terminalmage on 11 Jun 2018

Is that rework in progress or on the development roadmap? I can find a way to work around it for the time being but would like to communicate to my leadership how long we have to maintain that system.

fooka03 on 11 Jun 2018

The current plan is to have this functionality present in the next feature release (this fall).

terminalmage on 13 Jun 2018

👍4

Hi, do you have update about this issue?
Each salt call wait 40s with Re-using gitfs object for process in logs
after clean gitfs directory: 40s => 9s
Maybe old branch remove on origin still present ?