Habitat: Habitat services reporting sys IP as 127.0.0.1

[wduncanfraser]

Ran into a situation yesterday where in one of our gossip rings some services had started reporting their sys.ip as 127.0.0.1.

For example:
"sys": { "ip": "127.0.0.1", "hostname": "<hostname>", "gossip_ip": "192.168.100.20", "gossip_port": 9638, "http_gateway_ip": "0.0.0.0", "http_gateway_port": 9631 },

This caused bound services to render the wrong IP in configurations/hooks, breaking interaction. In one case, a single node had both PostgreSQL and a backend API. PostgreSQL was showing the correct IP, while the backend API was showing 127.0.0.1.

I was able to correct the issue by unloading and loading the effected services. This is a long running ring that has updates for services pushed into it via CI/Chef. Host OS is Centos 7.

I have some debug logs I can send on request (not wanting to post directly here as they are live servers).

[baumanj]

@wduncanfraser: I'm trying to confirm whether this is still an issue on v0.57. Are you still seeing it, or do you have any steps for reproducing it I might try?

[gscho]

@baumanj Also seeing this behavior after I ran a reboot on my Jenkins slaves.

My setup is

• 2 jenkins slaves --bind to 1 jenkins master
• Ran a reboot on both jenkins slaves
• Both slaves are reporting their {{sys.ip}} as 127.0.0.1
• OS is SLES 12 SP3

[baumanj]

Thanks for the additional context, @gscho.

I really wish Jenkins would update their terminology 😖

[gscho]

:bondage_emojii:

[gscho]

@baumanj any updates on this one?

[baumanj]

This hasn't had active work yet, @gscho. Is this causing you headaches currently?

[gscho]

I'd say a minor headache. Since I can't rely on {{sys.ip}} and no {{sys.fqdn}} is available, I'm currently working around this using {{sys.hostname}}.{{cfg.domain}} in all of my packages that need this info.

[baumanj]

Thanks for the input. I can't make any particular promises at this point, but we'll factor it into our prioritization for sure.

[y-me-y]

This is still causing an issue in our environments and one of my coworkers seems to experience this issue continuously. Since the the hab team has previously decided to not embrace {{sys.fqdn}} is there any chance to get some actual movement on this.

[y-me-y]

This issue was closed in November based on the sys.ip is really what’s needed and I feel at this point both issues still loom https://github.com/habitat-sh/habitat/issues/5731

[gscho]

I'm actually going to change my previous comment and say this is a big headache. Just had a production vault cluster outage due to the fact that the consul nodes were reporting their sys.ip as 127.0.0.1.

[kmott]

A little bit more information, it seems like the metadata for sys is different/incorrect in some cases, though I have not been able to figure out why or under what conditions (though it most often seems to occur with EC2 instances).

          "sys": {
            "ip": "127.0.0.1",
            "hostname": "ip-10-5-2-59.ec2.internal",
            "gossip_ip": "10.5.2.59",
            "gossip_port": 9638,
            "http_gateway_ip": "0.0.0.0",
            "http_gateway_port": 9631,
            "ctl_gateway_ip": "0.0.0.0",
            "ctl_gateway_port": 9632
          },

In the above output, I would expect sys.gossip_ip to always be the same as sys.ip (based on my limited understanding of Habitat's internals).

• ip string The IP address of the running service.
• gossip_ip string Listening address for Supervisor's gossip connection.

[christophermaier]

Doing a bit more research into this... Are the cases where the IP address is being reported as 127.0.0.1 being run in environments where Google's DNS (i.e. 8.8.8.8) is not routable?

It looks like the current code is using the kernel's routing tables to return the IP address of the interface that can route to 8.8.8.8.

https://github.com/habitat-sh/habitat/blob/4527c32e25d60d77ece99cb7ac8cd1182cb85279/components/core/src/util/sys.rs#L9-L16

(For reference, this technique is detailed in Section 8.14, "Determining Outgoing Interface with UDP", of _UNIX Network Programming_, Volume 1, 3rd Edition by Stevens et al., the relevant text to which is available here.)

In an environment in which 8.8.8.8 is not routable, this function will return an error. However, we recover from that here:

https://github.com/habitat-sh/habitat/blob/4527c32e25d60d77ece99cb7ac8cd1182cb85279/components/sup/src/manager/sys.rs#L38-L45

It appears, however, that the only use of this IP address is actually to be exposed in templating contexts, through the top-level sys.ip key, as well as the sys.ip key of any svc_member structs (detailed at https://www.habitat.sh/docs/reference/#template-data).

Provided that hostnames of machines resolve via DNS, I think we can modify the code to return the IP address associated with the hostname / canonical name of the host. Apart from that, I'm not sure of a deterministic way to get a single IP address without also having to specify a particular network interface, which gets into a bunch of additional complexity.

[christophermaier]

Ping @wduncanfraser @gscho @y-me-y @kmott ☝️

[kmott]

Doing a bit more research into this... Are the cases where the IP address is being reported as 127.0.0.1 being run in environments where Google's DNS (i.e. 8.8.8.8) is not routable?

I believe that is the case, although I'd need @gscho to confirm (team is travelling back from ChefConf, so may not get a response until next week).

[dmccown]

@kmott Can you confirm this is the behavior so we can look at getting the fix prioritized?

[y-me-y]

@dmccown @christophermaier - not currently on this project any longer but based on the discovery you did and familiarity of the environment this goes back to my original ask of returning the member.hostname.fqdn. https://github.com/habitat-sh/habitat/issues/5731.

but as long as you use the full hostname hostname -f or command substitute for the fqdn the returning the ip address from the dns should work as a better back up vs defaulting 127.0.0.1.

However, I would a make suggestion as a long term plan to allow advertising a specific interface.

[kmott]

@dmccown, I will try to get a repro this week, and let you know what I find.

[kmott]

@dmccown, I confirmed in a local Vagrant setup that sys.ip does report 127.0.0.1 if you null/reject route the Google DNS servers. Here's what I did:

• Created a Vagrantfile from some base system of your choice (I used Debian 9).
• Add the following for config.vm.provision:

  config.vm.provision "shell", inline: <<-SHELL
route add 8.8.8.8/32 reject
route add 8.8.4.4/32 reject
[ ... bootstrap habitat to your liking ... ]
curl --silent -X GET http://localhost:9631/census | hab pkg exec core/jq-static jq -r ''
  SHELL

• Run vagrant up, inspect resulting output to see sys.ip is set to 127.0.0.1.

I think this should allow you to repro locally as needed. Let me know if you need anything else.

[nellshamrell]

Thanks for the repro steps, @kmott - looking into this now

[nellshamrell]

I was able to replicate this with the help of @kmott's steps.

Successful IP address identification

Using this Vagrant file:

Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "generic/debian9"
  config.vm.provision "shell", inline: <<-SHELL
      echo "export HAB_LICENSE=accept-no-persist" >> /etc/profile.d/myvar.sh
      curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | sudo bash
      sudo groupadd hab
      sudo useradd -g hab hab
      hab license accept
      sudo -E hab pkg install core/jq-static
  SHELL
end

I can run:

vagrant ssh
sudo hab sup run &
sudo hab svc load core/jenkins
curl --silent -X GET http://localhost:9631/census | hab pkg exec core/jq-static jq -r '.'

And see the correct IP address in the census data.

    "sys": {
            "ip": "10.0.2.15",
            "hostname": "debian9.localdomain",
            "gossip_ip": "0.0.0.0",
            "gossip_port": 9638,
            "http_gateway_ip": "0.0.0.0",
            "http_gateway_port": 9631,
            "ctl_gateway_ip": "127.0.0.1",
            "ctl_gateway_port": 9632
          },

Unsuccessful IP address identification

If, however, I reject the Google DNS servers with this Vagrant file:

Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "generic/debian9"
  config.vm.provision "shell", inline: <<-SHELL
      route add 8.8.8.8/32 reject
      route add 8.8.4.4/32 reject
      echo "export HAB_LICENSE=accept-no-persist" >> /etc/profile.d/myvar.sh
      curl https://raw.githubusercontent.com/habitat-sh/habitat/master/components/hab/install.sh | sudo bash
      sudo groupadd hab
      sudo useradd -g hab hab
      hab license accept
      sudo -E hab pkg install core/jq-static
  SHELL

end

And run the same commands as above, I will see 127.0.0.1 as the sys.ip

          "sys": {
            "ip": "127.0.0.1",
            "hostname": "debian9.localdomain",
            "gossip_ip": "0.0.0.0",
            "gossip_port": 9638,
            "http_gateway_ip": "0.0.0.0",
            "http_gateway_port": 9631,
            "ctl_gateway_ip": "127.0.0.1",
            "ctl_gateway_port": 9632
          },

Got to head out for the weekend, but will dig deeper into this on Monday!

[nellshamrell]

Back on this - experimenting with modifying the code to return the IP address associated with the hostname / canonical name of the host (as @christophermaier suggested above), doing a build, and seeing what it does.

[nellshamrell]

Debating whether I should modify this method:

core/src/util/sys.rs

static GOOGLE_DNS: &'static str = "8.8.8.8:53";

pub fn ip() -> Result<IpAddr> {
    let socket = UdpSocket::bind("0.0.0.0:0")?;
    socket.connect(GOOGLE_DNS)?;
    let addr = socket.local_addr()?;
    Ok(addr.ip())
}

to always look up the IP address based on the hostname/canonical name of the host (thereby decoupling us from Google DNS).

Or to leave that method as it is, and lookup up the IP address based on hostname when this method returns an error:

components/sup/src/manager/sys.rs

impl Sys {
    pub fn new(permanent: bool,
               gossip: GossipListenAddr,
               ctl: ListenCtlAddr,
               http: http_gateway::ListenAddr)
               -> Sys {
        let ip = match lookup_ip() {
            Ok(ip) => ip,
            Err(e) => {
                let ip = IpAddr::V4(Ipv4Addr::new(127, 0, 0, 1));
                outputln!("IP Address lookup failed; using fallback of {} ({})", ip, e);
                ip
            }
        };
(...)

[raskchanky]

I might be thinking about this wrong, as I haven't actually experienced this problem myself, but I'm guessing if your service cluster is running on an internal network, either via VMs or containers, it's entirely possible that none of the nodes in question will have externally resolvable IP addresses. If that's the case, it seems like we'd need to be able to specify an interface directly, no?

[stevendanna]

My suspicion is that we might have two distinct failure cases. The first case, of which you've reproduced an instance of above, is where the method of looking up a local address fails because the "udp connect" trick we are using doesn't work.

For that case, we might consider using the netlink and rtnetlink APIs on linux to get more complete address and route table information. Or as @raskchanky suggests, perhaps provide a way for the user to hint to us how to find the IP address (or perhaps provide a way to configure it directly).

However, I think a second potential case might be where the lookup method we are using _would work_ but hab-sup is coming up before any network interfaces and routing information has been configured. Consider that our example systemd configuration

https://www.habitat.sh/docs/best-practices/#running-under-systemd

does not instruct the user to declare a dependency on any of the well-known network targets:

https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

thus, I _believe_ it is possible that users who are seeing this problem _intermittently_ may be experiencing a race condition between the network being up and hab-sup coming up. In this case, it isn't clear to me that _any_ method of looking up the IP is going to be reliable since it may be that no network interfaces or routes have been added yet.

[raskchanky]

It seems like at the very least, we might want to update our instructions to configure systemd in such a way that the network is up and available before our service starts. That doesn't help in instances where nodes aren't using systemd, but it doesn't seem like there's any downside to doing that, is there? And that's independent of any solution in the code itself.

[nellshamrell]

Thank you for the comments :)

I agree that there are two potential failure scenarios here - the one where the UDP connect trick doesn't work and the one where there is a race condition between the network being up and hab-sup coming up.

I'm inclined to focus on the one that I can replicate first - the former - and get a PR in which fixes it. Then I will turn my attention to the other scenario (including updating the documentation around configuring systemd).

Super appreciate the insight and input!

[stevendanna]

@nellshamrell Another thing that occurred to me is that ohai might have some wisdom here. Ohai parses command line output, which I don't think we want to do, but at the end of the day it ends up with a list of route information and has to choose a single IP address for its "ipaddress" attribute. It does that with some heuristics that are here:

https://github.com/chef/ohai/blob/master/lib/ohai/plugins/linux/network.rb#L437

[stevendanna]

Final comment of the day I swear:

Or to leave that method as it is, and lookup up the IP address based on hostname when this method returns an error:

To your original question, among the options you outlined I would definitely say that a multi-method approach is probably better. My experience with our user base is that properly configured name resolution and hostnames is hit or miss so replacing the existing method completely might be a cure worse than the disease without more data.

[nellshamrell]

The comments are immensely helpful and appreciated, @stevendanna! Feel free to leave as many on this issue as you like!

[christophermaier]

@nellshamrell We use dns_lookup already in other parts of the code; it may be useful in the solution to this issue.

[nellshamrell]

Did some experimentation with a few different ways to get the host's IP address - captured the experiments in this crate.

Based on these experiments - I'm inclined to go with the pnet crate as it returns the external IP of the host without needing to access the Google DNS servers. What I need to figure out next is to parse the list of network interfaces on the box for the correct one with the external IP address. It's late on a Friday, so I will pick this back up on Mon.

[sirajrauff]

Just wanted to drop my experience here with some of this in that I think there may be an argument for being able to specify the network adapter for sys.ip distinctly from the one with internet access be a valid use case; I've had a scenario pretty much as @raskchanky described:

I'm guessing if your service cluster is running on an internal network, either via VMs or containers,

Vagrant + Virtualbox multi-machine environment using NAT with the host for internet access, and a private network to communicate with other VMs. In this case the first network adapter (NAT) is what the Supervisor will resolve as sys.ip. This IP address, being proxied, does not allow ingress and in fact is the same IP address across all the machines. This breaks bindings in this scenario, unless I modify the package to use sys.gossip_ip which will resolve to the IP associated with the second network adapter attached to the private network.

[nellshamrell]

This is definitely something worthy of a design proposal and discussion. Putting that together today or tomorrow.

[raskchanky]

https://github.com/habitat-sh/habitat/pull/6960 should fix this