Since I upgraded to 2018.3.1 from 2017.7.6 I have a minion in a different datacenter that is timing out when I run salt from the master (i.e. with salt vpn\* state.apply -l debug):
vpn01.*****:
Minion did not return. [No response]
If I use salt-call and start the highstate from the minion, it completes correctly.
timeout value was the default one (5). Now to avoid the timeout I have to put a value higher than the highstate run
There are no issues with all the other minions in the same datacenter as the master. The only one that fails is the one in a different datacenter (that was working fine till 2017.7.6)
those are the latest lines before failing:
[DEBUG ] Checking whether jid 20180614080706375333 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] return event: {'vpn01.*****': {u'failed': True}}
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded no_return.output
if I run salt-run jobs.list_job 20180614080706375333 right after it fails, I can still see the job running
salt master:
# salt --versions-report
Salt Version:
Salt: 2018.3.1
Dependency Versions:
cffi: 1.5.2
cherrypy: 3.5.0
dateutil: 2.4.2
docker-py: Not Installed
gitdb: 0.6.4
gitpython: 1.0.1
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.3.7
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.0
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: 0.9.0
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-128-generic
system: Linux
version: Ubuntu 16.04 xenial
salt minion:
# salt-call --versions-report
Salt Version:
Salt: 2018.3.1
Dependency Versions:
cffi: 1.5.2
cherrypy: Not Installed
dateutil: 2.4.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.8
libgit2: 0.24.0
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 1.0.3
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.3.7
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.24.0
Python: 2.7.12 (default, Dec 4 2017, 14:50:18)
python-gnupg: 0.3.8
PyYAML: 3.11
PyZMQ: 15.2.0
RAET: Not Installed
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: Ubuntu 16.04 xenial
locale: UTF-8
machine: x86_64
release: 4.4.0-128-generic
system: Linux
version: Ubuntu 16.04 xenial
@rmarchei Thanks for the report. Are other salt commands also timing out, for example is test.ping timing out when run against the minion in question?
Having the same problem after upgrade.
I'm getting timeouts when running long jobs like state.highstate from minions that are 'far' from the master.
Adding -t 600 to salt command helps.
test.ping is ok. Running a single state fails if it lasts more than the
timeout value.
On Thu, Jun 14, 2018, 21:14 Gareth J. Greenaway notifications@github.com
wrote:
@rmarchei https://github.com/rmarchei Thanks for the report. Are other
salt commands also timing out, for example is test.ping timing out when
run against the minion in question?—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/48130#issuecomment-397407476,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABO_pUnFSDFwQr8vrk7omlG5SHZh9FCZks5t8rYogaJpZM4UoKDe
.
Can you give us any more information?
If I put a time.sleep 600 in a state, i am still not running into this issue.
include:
- common.packages
test long run:
module.run:
- name: test.sleep
- length: 600
Can you provide us with some states that replicate this issue?
Thanks,
Daniel
I can easily replicate the issue even this way:
# salt -t 5 vpn01\* test.ping
vpn01.*****:
True
# salt -t 5 -l debug vpn01* test.sleep 1000
[INFO ] Loading Saltfile from '/root/.salt/Saltfile'
[DEBUG ] Reading configuration from /root/.salt/Saltfile
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/github.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/github.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt01.*****
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/github.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/github.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt01.*****
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/c1/add81d2a217fe84f8de1e5e8b8afb0957fbd198e5a99cb7d7afa6de30f9d1d/.minions.p
[DEBUG ] get_iter_returns for jid 20180615090548852476 sent to set(['vpn01.*****']) will timeout at 09:05:53.858118
[DEBUG ] Checking whether jid 20180615090548852476 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] return event: {'vpn01.*****': {u'failed': True}}
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] Cache file "/var/cache/salt/master/minions/s.*****.p" does not exist
[DEBUG ] Cache file "/var/cache/salt/master/minions/dd.*****.p" does not exist
[DEBUG ] Cache file "/var/cache/salt/master/minions/repl01.*****.p" does not exist
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded no_return.output
vpn01.*****:
Minion did not return. [No response]
I have many minions and this is the only one that fails after upgrading to 2018.3 (and downgrading both master and minion to 2017.7.6 solves the issue).
Salt master is in a datacenter in US and the minion is in UK, so they are far away from each other and this seems to be what triggers the issue. Other than that, there's really no particular setup.
also, if I run salt-call state.apply directly from the minion, everything works as usual
EDIT:
Not sure if testing with the module was the best approach. So here's the same test running as state:
# salt -t 5 php01* state.apply test.sleep
php01.*****:
----------
ID: test long run
Function: module.run
Name: test.sleep
Result: True
Comment: Module function test.sleep executed
Started: 09:18:21.070052
Duration: 60962.98 ms
Changes:
----------
ret:
True
# salt -t 5 vpn01* state.apply test.sleep
vpn01.*****:
Minion did not return. [No response]
# salt -t 90 vpn01* state.apply test.sleep
vpn01.*****:
----------
ID: test long run
Function: module.run
Name: test.sleep
Result: True
Comment: Module function test.sleep executed
Started: 09:23:08.110758
Duration: 60030.165 ms
Changes:
----------
ret:
True
# cat /srv/salt/test/sleep.sls
test long run:
module.run:
- name: test.sleep
- length: 60
php01 is a minion in the same datacenter as the master
Can we see debug logs on the minion for one of these failed jobs please?
minion debug log:
[INFO ] User root Executing command state.apply with jid 20180615093622296339
[DEBUG ] Command details {u'tgt_type': u'glob', u'jid': u'20180615093622296339', u'tgt': u'vpn01*', u'ret': u'', u'user': u'root', u'arg': [u'test.sleep'], u'fun': u'state.apply'}
[INFO ] Starting a new job with PID 336
[DEBUG ] LazyLoaded state.apply
[DEBUG ] LazyLoaded direct_call.execute
[DEBUG ] LazyLoaded saltutil.is_running
[DEBUG ] LazyLoaded grains.get
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.140.108.77:4506
[DEBUG ] Trying to connect to: tcp://10.140.108.77:4506
[DEBUG ] Determining pillar cache
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.140.108.77:4506
[DEBUG ] Trying to connect to: tcp://10.140.108.77:4506
[INFO ] User root Executing command saltutil.find_job with jid 20180615093627360321
[DEBUG ] Command details {u'tgt_type': u'list', u'jid': u'20180615093627360321', u'tgt': [u'vpn01.*****'], u'ret': u'', u'user': u'root', u'arg': [u'20180615093622296339'], u'fun': u'saltutil.find_job'}
[INFO ] Starting a new job with PID 343
[DEBUG ] LazyLoaded saltutil.find_job
[DEBUG ] LazyLoaded direct_call.execute
[DEBUG ] Minion return retry timer set to 5 seconds (randomized)
[INFO ] Returning information for job: 20180615093627360321
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.140.108.77:4506
[DEBUG ] Trying to connect to: tcp://10.140.108.77:4506
[DEBUG ] minion return: {u'fun_args': [u'20180615093622296339'], u'jid': u'20180615093627360321', u'return': {'tgt_type': 'glob', 'jid': '20180615093622296339', 'tgt': 'vpn01*', 'pid': 336, 'ret': '', 'user': 'root', 'arg': ['test.sleep'], 'fun': 'state.apply'}, u'retcode': 0, u'success': True, u'fun': u'saltutil.find_job'}
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO ] Loading fresh modules for state activity
[DEBUG ] LazyLoaded jinja.render
[DEBUG ] LazyLoaded yaml.render
[DEBUG ] In saltenv 'base', looking at rel_path 'test/sleep.sls' to resolve 'salt://test/sleep.sls'
[DEBUG ] In saltenv 'base', ** considering ** path '/var/cache/salt/minion/files/base/test/sleep.sls' to resolve 'salt://test/sleep.sls'
[DEBUG ] compile template: /var/cache/salt/minion/files/base/test/sleep.sls
[DEBUG ] Jinja search path: [u'/var/cache/salt/minion/files/base']
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.140.108.77:4506
[DEBUG ] Trying to connect to: tcp://10.140.108.77:4506
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/test/sleep.sls' using 'jinja' renderer: 0.00687289237976
[DEBUG ] Rendered data from file: /var/cache/salt/minion/files/base/test/sleep.sls:
test long run:
module.run:
- name: test.sleep
- length: 60
[DEBUG ] Results of YAML rendering:
OrderedDict([(u'test long run', OrderedDict([(u'module.run', [OrderedDict([(u'name', u'test.sleep')]), OrderedDict([(u'length', 60)])])]))])
[PROFILE ] Time (in seconds) to render '/var/cache/salt/minion/files/base/test/sleep.sls' using 'yaml' renderer: 0.0010.140.108.772686882019
[DEBUG ] LazyLoaded config.option
[DEBUG ] LazyLoaded module.run
[INFO ] Running state [test.sleep] at time 09:36:28.475372
[INFO ] Executing state module.run for [test.sleep]
[WARNING ] The function "module.run" is using its deprecated version and will expire in version "Sodium".
[DEBUG ] LazyLoaded test.sleep
[INFO ] {u'ret': True}
[INFO ] Completed state [test.sleep] at time 09:37:28.545466 (duration_in_ms=60070.094)
[DEBUG ] File /var/cache/salt/minion/accumulator/140594546870032 does not exist, no need to cleanup
[DEBUG ] LazyLoaded state.check_result
[DEBUG ] Minion return retry timer set to 6 seconds (randomized)
[INFO ] Returning information for job: 20180615093622296339
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506', u'aes')
[DEBUG ] Initializing new AsyncAuth for (u'/etc/salt/pki/minion', u'vpn01.*****', u'tcp://10.140.108.77:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://10.140.108.77:4506
[DEBUG ] Trying to connect to: tcp://10.140.108.77:4506
[DEBUG ] minion return: {u'fun_args': [u'test.sleep'], u'jid': u'20180615093622296339', u'return': {u'module_|-test long run_|-test.sleep_|-run': {u'comment': u'Module function test.sleep executed', u'name': u'test.sleep', u'start_time': '09:36:28.475372', u'result': True, u'duration': 60070.094, u'__run_num__': 0, u'__sls__': u'test.sleep', u'changes': {u'ret': True}, u'__id__': u'test long run'}}, u'retcode': 0, u'success': True, u'fun': u'state.apply'}
Master:
# salt -t 5 -l debug vpn01* state.apply test.sleep
[INFO ] Loading Saltfile from '/root/.salt/Saltfile'
[DEBUG ] Reading configuration from /root/.salt/Saltfile
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/github.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/github.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt01.*****
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/engines.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/engines.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/external_auth.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/external_auth.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/github.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/github.conf
[DEBUG ] Including configuration from '/etc/salt/master.d/gpg.conf'
[DEBUG ] Reading configuration from /etc/salt/master.d/gpg.conf
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: salt01.*****
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] Initializing new IPCClient for path: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] LazyLoaded local_cache.get_load
[DEBUG ] Reading minion list from /var/cache/salt/master/jobs/d1/a09d929e4997dc951b04c1ce326f4cb2efa7729a9f495a9ed48b277053d938/.minions.p
[DEBUG ] get_iter_returns for jid 20180615093622296339 sent to set(['vpn01.*****']) will timeout at 09:36:27.301968
[DEBUG ] Checking whether jid 20180615093622296339 is still running
[DEBUG ] Initializing new AsyncZeroMQReqChannel for (u'/etc/salt/pki/master', u'salt01.*****_master', u'tcp://127.0.0.1:4506', u'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://127.0.0.1:4506
[DEBUG ] Trying to connect to: tcp://127.0.0.1:4506
[DEBUG ] return event: {'vpn01.*****': {u'failed': True}}
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] Cache file "/var/cache/salt/master/minions/s.*****.p" does not exist
[DEBUG ] Cache file "/var/cache/salt/master/minions/dd.*****.p" does not exist
[DEBUG ] Cache file "/var/cache/salt/master/minions/repl01.*****.p" does not exist
[DEBUG ] LazyLoaded localfs.init_kwargs
[DEBUG ] LazyLoaded no_return.output
vpn01.*****:
Minion did not return. [No response]
Can you run this in a second session on the master?
salt-run state.event pretty=true tagmatch=salt/job/*/ret/*
And then run
salt 'vpn01.*' -t 5 state.apply test.sleep
in a different terminal, and see if you get a saltutil.find_job return back in the output of salt-run before the timeout happens from the state.apply?
Thanks,
Daniel
I got this:
# salt-run state.event pretty=true tagmatch=salt/job/*/ret/*
salt/job/20180615095930868030/ret/vpn01.***** {
"_stamp": "2018-06-15T16:59:31.196333",
"cmd": "_return",
"fun": "saltutil.find_job",
"fun_args": [
"20180615095925797804"
],
"id": "vpn01.*****",
"jid": "20180615095930868030",
"retcode": 0,
"return": {
"arg": [
"test.sleep"
],
"fun": "state.apply",
"jid": "20180615095925797804",
"pid": 1186,
"ret": "",
"tgt": "vpn01.*",
"tgt_type": "glob",
"user": "root"
},
"success": true
}
at this point the master failed. But then I also got (I guess when the job finished on the minion):
salt/job/20180615095925797804/ret/vpn01.***** {
"_stamp": "2018-06-15T17:00:32.407790",
"cmd": "_return",
"fun": "state.apply",
"fun_args": [
"test.sleep"
],
"id": "vpn01.*****",
"jid": "20180615095925797804",
"out": "highstate",
"retcode": 0,
"return": {
"module_|-test long run_|-test.sleep_|-run": {
"__id__": "test long run",
"__run_num__": 0,
"__sls__": "test.sleep",
"changes": {
"ret": true
},
"comment": "Module function test.sleep executed",
"duration": 60054.687,
"name": "test.sleep",
"result": true,
"start_time": "09:59:32.079876"
}
},
"success": true
}
Cool, that is what I am seeing on one of our internal setups.
We are looking into this, to see if we can simplify how we are able to replicate it, because I am not seeing this error with test.sleep, i have to do a large staterun.
Thanks,
Daniel
I just submitted a PR that fixed this for us, can you try it out?
patch -Np1 -d /usr/lib/python2.7/dist-packages/ < <(curl -sL https://github.com/saltstack/salt/pull/48158.diff)
Thanks,
Daniel
Worked! Thanks for the quick fix!
root@salt01:~# patch -Np1 -d /usr/lib/python2.7/dist-packages/ < <(curl -sL https://github.com/saltstack/salt/pull/48158.diff)
patching file salt/client/__init__.py
root@salt01:~# salt 'vpn01.*' -t 5 state.apply test.sleep
vpn01.*****:
----------
ID: test long run
Function: module.run
Name: test.sleep
Result: True
Comment: Module function test.sleep executed
Started: 13:48:04.616433
Duration: 60069.986 ms
Changes:
----------
ret:
True
Summary for vpn01.*****
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 60.070 s
Awesome! thanks for verifying!
Daniel
I have see this issue too.
This will be fixed in 2018.3.2 and 2017.7.7 which is tagged and will be released soon.
I backported the patch to 2018.3.1, and works for me.