Salt: IPV6 Warning when ipv6 set to False

Created on 27 Apr 2017 · 17Comments · Source: saltstack/salt

Description of Issue/Question

Migrating from 2015.5.3 to 2016.11.4. Copied /etc/salt/master, restarted daemon, and looking for configuration warnings in log.

Setup

Configuration file entries:

interface: 0.0.0.0
ipv6: False

Steps to Reproduce Issue

Many warnings such as this:

Apr 26 23:20:34 ip-172-31-23-247 salt-master[6228]: [WARNING ] Unable to find IPv6 record for "ip-172-31-23-247.us-west-2.compute.internal" causing a 10 second timeout when rendering grains. Set the dns or /etc/hosts for IPv6 to clear this.

Versions Report

$ salt --versions-report
Salt Version:
           Salt: 2016.11.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: 3.5.0
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
        machine: x86_64
        release: 4.4.0-1013-aws
         system: Linux
        version: Ubuntu 16.04 xenial

Pending Discussion

Source

razed11

Most helpful comment

Why is salt-call trying to render IPv6 info when IPv6 is disabled on the host? And why should I add an IPv6 entry in /etc/hosts to solve this error? I disabled IPv6 for a reason.

jsaathof on 26 Mar 2018

👍6

All 17 comments

I'm seeing the same thing on RHEL 7.

Salt Version:
           Salt: 2016.11.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.3
         pygit2: Not Installed
         Python: 2.7.5 (default, Aug  2 2016, 04:20:16)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.3 Maipo
        machine: x86_64
        release: 3.10.0-514.16.1.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.3 Maipo

amendlik on 27 Apr 2017

Someone on the Google Group posted the following. Though it is unclear to me why this would affect the master or show in the master log:

IPv6 is just for connecting to the salt master.

The minion grains around ipv6 are still rendered. There are some
issues where getent does not return a valid ipv6 address when looking
up the ipv6 address for the hostname to provide in grains. If this is
the case, there is a 10 second timeout that happens while the grains
are rendered (specifically on salt-run)

If you set the hostname in the ::1 in /etc/hosts this warning will go away

However my /etc/hosts seems to contain this entry:

127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

razed11 on 27 Apr 2017

The trick is to add the hostname to the ::1 list. I'm not familiar with IPV6 changes in this regard.

::1 ip6-localhost ip6-loopback ip-172-31-23-247

That last entry is the hostname Amazon assigned to my Ubuntu machine.

razed11 on 27 Apr 2017

Perhaps this can be a request to improve the warning message:

Set the dns or /etc/hosts for IPv6 to clear this.

I'm not sure what to change it to but set for IPv6 wasn't helpful to me. Also unclear why I am seeing this message at all.

razed11 on 27 Apr 2017

I'm not sure what "rendering grains" is exactly (does it mean updating grains, or rendering templates which use grains, or specifically ip_fqdn() itself?), but the commands that cause this to be printed (e.g. salt-call state.apply) are not experiencing a ten second timeout themselves. The DNS lookup fails more or less instantly.

Adding the name to /etc/hosts would cause some of our machines to have a public IP when resolved over IPv4 and a loopback over IPv6, because we don't really have an IPv6 connection for them. That smells bad to me - I'd prefer the IPv6 lookups to fail as they currently do, because I don't want anything trying to connect to them over IPv6 at all.

joewreschnig on 27 Apr 2017

👍1

Salt-call starts up a separate minion process which does all of the maintenance tasks that are usually done on a schedule when running a salt-minion.

This will render the grains, that is run everything in salt.grains and render pillars and do a few other tasks if it is a masterless minion.

Another option i believe is to make sure you have myhostname as the last option for hosts in /etc/nsswitch.conf

Thanks,
Daniel

gtmanfred on 27 Apr 2017

I have submitted the PR #40934 to strip out the generation of ipv6 grains, as well as only log the warning in salt-run/master which is where we saw the slow down.

gtmanfred on 27 Apr 2017

This has been fixed in 2016.11.6

gtmanfred on 14 Jun 2017

@gtmanfred this is not fixed.

[salt.loaded.int.grains.core][WARNING ][2395] Unable to find IPv6 record for "my_hostname" causing a 10 second timeout when rendering grains. Set the dns or /etc/hosts for IPv6 to clear this.

sudo salt --versions-report
Salt Version:
Salt: 2016.11.6

Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.6.0
docker-py: Not Installed
gitdb: 0.5.4
gitpython: 0.3.2 RC1
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: 0.9.1
msgpack-pure: Not Installed
msgpack-python: 0.4.6
mysql-python: 1.2.3
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
python-gnupg: Not Installed
PyYAML: 3.10
PyZMQ: 14.0.1
RAET: Not Installed
smmap: 0.8.2
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.0.5

System Versions:
dist: Ubuntu 14.04 trusty
machine: x86_64
release: 3.13.0-92-generic
system: Linux
version: Ubuntu 14.04 trusty

mf-bmihaescu on 7 Jul 2017

Yes, we ran into problems removing the ipv6 nameservers from grains when ipv6 was set to false, because most machines are dual stack anyway.

We have moved this warning to only show when the host name actually takes more than 5 seconds to render. But that wont be out until 2016.11.7 or 2017.7.0

gtmanfred on 7 Jul 2017

What should I do in order to not have master log file full with this warnings ?

mf-bmihaescu on 7 Jul 2017

make sure that the hostname of your master is set correctly for lookups.
You can set it to the ip address in /etc/hosts.

getent ahostv6

This probably returns an invalid ip address like ::::127.0.0.1.

You could also set the log_level: critical in /etc/salt/master

On Fri, Jul 7, 2017 at 7:16 AM, mf-bmihaescu notifications@github.com
wrote:

What should I do in order to not have master log file full with this
warnings ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/saltstack/salt/issues/40912#issuecomment-313677751,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAssocmEvbfDF0z51TAZprmaJ-LTjVk8ks5sLi-dgaJpZM4NJjF4
.

gtmanfred on 7 Jul 2017

We don't run ipv6. I get:

[root@myhostname ~]# getent ahostv6 myhostname
Unknown database: ahostv6

salt-master --version
salt-master 2016.11.7 (Carbon)

how can I avoid the spam in logging?

davidkarlsen on 26 Oct 2017

Is this fixed at all? By the looks of it, it is not.

ZAM- on 8 Jan 2018

unfortunately it has not been fixed.

walisonmorales on 31 Jan 2018

Why is salt-call trying to render IPv6 info when IPv6 is disabled on the host? And why should I add an IPv6 entry in /etc/hosts to solve this error? I disabled IPv6 for a reason.

jsaathof on 26 Mar 2018

👍6

A year and a half later, I'm hitting this trying to upgrade us to 2018.3:

$ time sudo salt-run manage.not_present
[WARNING ] Unable to find IPv6 record for "my.host.name" causing a 0:00:10.007213 second timeout when rendering grains. Set the dns or /etc/hosts for IPv6 to clear this.
[INFO    ] Runner completed: 20190828205654237254

real    0m11.752s
user    0m1.444s
sys     0m0.342s

However, due to helpful comments in #32719 I was able to discover the root cause... IPv6 was still enabled on loopback:

$ sudo sysctl net.ipv6.conf|grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 0

Disable it on loopback:

$ sudo sysctl net.ipv6.conf.lo.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6 = 1

All good now:

$ sudo sysctl net.ipv6.conf 2> /dev/null |grep disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.eth0.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

$ time sudo salt-run manage.not_present
[INFO    ] Runner completed: 20190828210202085345

real    0m1.814s
user    0m1.524s
sys     0m0.331s