Rkt: Sets up /etc/hosts file

[yifan-gu]

Ref https://github.com/kubernetes/kubernetes/issues/21923
We probably wants rkt to setup the /etc/hosts with the given hostname.

cc @steveeJ @sjpotter @jonboulle @vishh

[krnowak]

Re: Sets up /etcd/hosts file title - systemd syndrome strikes again!

[iaguis]

Re: Sets up /etcd/hosts file title - systemd syndrome strikes again!

:joy:

[jellonek]

@yifan-gu which /etc/hosts you have in mind? Inside pod? On the host?
Which problem this tries to resolve? Could you describe situation in few more words?

This smells like someone try to resolve some problem with something looking hmmm... not so good...

[yifan-gu]

@jellonek Sorry for not putting the context here.

I mean the /etc/hosts inside the pod. The problem is currently the /etc/hosts inside each rkt pod does not have contents expected by k8s, i.e. [1].

However this is not the final version of how /etc/hosts should look like. There are some ongoing work here [2] . I think what we should do is to have rkt the ability to do what @thockin suggested here (setting correct hostname, domain name) [3]

[1] https://github.com/kubernetes/kubernetes/blob/v1.2.0-alpha.8/pkg/kubelet/kubelet.go#L1214-L1229
[2] https://github.com/kubernetes/kubernetes/pull/20688/files#diff-bf28da68f62a8df6e99e447c4351122dR1267
[3] https://github.com/kubernetes/kubernetes/issues/21923#issuecomment-189000162

[yifan-gu]

@jellonek To summarize, the /etc/hosts inside the rkt pod only contains
127.0.0.1 rkt-c5fd8794-0e1b-4d8e-9969-dda7828a localhost localhost.localdomain

But we would like it to contains, e.g. 1.2.3.4 hostname.domain.com hostname when user provides a hostname and domain name.

[jellonek]

So from higher perspective we need to provide method of ... providing file with such information and this is not connected to some /etc/hosts bind mounting with it's host counterpart which i seen somewhere around (if same information are needed cross pods, still my opinion would be to create for this purpose some temporary file, shared by pods, but without access to potentially sensitive host data).

For me this looks like job for CNI plugins which should have common method to operate on entries which could be expected by applications in theirs /etcs - this could be hosts, but also resolv.conf, nssswitch.conf, networks, hostname, protocols, services, timezone and probably at minimum passwd with group (with entries for root and optional separate user/grp for particular application). This list covers also other than network connected subjects - so this common method should be common between CNI and rkt.

This requires architectonic decisions ( @jonboulle - hello there ;) ), because my proposal would require also adding knowledge about rootfs place into CNI.

Btw. mentioned timezone - it would be nice to provide method of setting timezone for pod from command line...

[jellonek]

Sry for post under post - it's only for information that previous one have update, probably not seen in mails...

[steveeJ]

For me this looks like job for CNI plugins which should have common method to operate on entries which could be expected by applications in theirs /etcs

No way CNI should be abused as a generic configuration utility ;-)
I agree that it would be nice to declare these things in a specified way, but CNI is just not the place for all of it. There is always the possibility of defining annotations well-known by rkt to handle such things. I would however stick to the issue of the hostname for now within this issue because it seems to be rather urgent in regard to k8s integration.

[jellonek]

IMO CNI only should use same method which could be reused in rkt for this settings - I do not propose that CNI plugins should do all this things ;)
For me it's obvious that protocols, services, timezone if are needed - should be added in prepare step of stage[01], but hosts, resolv.conf, nsswitch.conf, networks - maybe at least a possibility to tune content of these files could be added to CNI?

[euank]

One of the e2e DNS tests incidentally depends on this issue; hostname -i works only if your hostname is in the hosts file, and here they use that.

[yifan-gu]

@euank What's the expected result? The pod IP of the pod itself? But we can't populate that today.

[euank]

Yeah, the pod IP. That's how it works in docker and k8+docker, so that's what the test expects. I know there's discussion elsewhere about this as well since K8 assumes it for multiple things (e.g. downward api), but this is another instance of that.

[f0]

hi, also stumbled over this, have some java tests failing, they try to resolve the local hostname , e.g (rkt-3454353-34534-53452) and this is not in DNS so it fails
Please implement this in the next version

[s-urbaniak]

@yifan-gu any ideas here?

[euank]

I think the resolution we're going to go with for rktnetes (assuming I can get it working :) is to create a network namespace and invoke CNI outside of rkt, nab the ip from said namespace, and then launch rkt into that namespace (nsenter ... rkt --net=host essentially) with appropriately configured /etc/hosts and so on.

It's possible the right longterm solution is to allow something closer to that in rkt: creation of a pod sandbox, including network namespace and CNI stuff, before the bindmounts, environment variables, and other configs are finalized. Some code between the sandbox and rkt run could then implement this.

Alternately, stage1 plugins or init containers or such could imitate something similar to this if they could edit other container's mounts within the pod prior to any other containers launching.

[jellonek]

@euank this would not work with kvm flavor.
btw. this issue is probably connected with #2573

[euank]

@jellonek I think this issue doesn't have to be connected. What kubernetes (and possibly by extension rktnetes, see comment above) does is populate an /etc/hosts file with content and bindmount it through, which dodges our stage1 doing the same (what that issue appears to be).

And yeah, that approach won't work with the kvm flavor. As I understand it, there are broader reasons why rktnetes (really kubernetes as a whole) doesn't work with kvm-stuff right now, so I don't think we need to make this solution suitably generic yet.

When kubernetes makes changes to support kvm, we can improve this.

[jonboulle]

@euank is this still relevant?

[euank]

@jonboulle It's not immediately relevant.
It's not blocking rktnetes unless we move managing the network namespace back into rkt.

It's still something that could be helpful for doing such a thing less hackily and in a more generic way and I'm sure other people (such as f0 above) would find this useful for their own reasons, so vfuture sounds reasonable to me.

[pskrzyns]

@euank what do you mean by "there are broader reasons why rktnetes (really kubernetes as a whole) doesn't work with kvm-stuff right now" ?

[jonboulle]

@pskrzyns I think all of the stuff showing up in the bare-metal conformance tests.. which is you right?

[pskrzyns]

@jonboulle the only problems that are not common for coreos and kvm are:
port forwarding
kubectl logs doesnt work (also not working for special cases in coreos, but different reason)
3 SchedulerPredicates tests - still under investigation

so its not so big number of problems

other issues at this point of time looks like common for both flavors

[euank]

@pskrzyns I was thinking of places it shells out to nsenter, cadvisor-related features, and networking, but it seems like I was being overly pessimistic. Apologies for my misunderstanding, I was wrong on that point.

However, it does seem like for networking there are remaining issues. Per the above, the solution of setting the namespace up via kubelet and using --net=host doesn't work with kvm flavor... on the other hand, that was done to solve numerous problems related to k8s wanting the IP prior to a pod starting.
Perhaps there should be a way to detect if the stage1 doesn't operate along with a network namespace and have it fall back the previous behaviour where ip-stuff isn't available but at least it works?
Perhaps all of this should be in rkt and kubelet should lean on that?
I'm not sure this issue is the right place to discuss it, but I'm happy to discuss further. If you'd like I can open an umbrella issue for the discussion of rkt networking + k8s.

[pskrzyns]

@euank thats great idea ;)

[derekmahar]

For what it's worth, here's an example that illustrates this behaviour. I was a bit surprised that ping postgres inside the container (pod) didn't produce the same result as ping localhost.

core@core-01 ~ $ sudo systemd-run /usr/bin/rkt run --hostname=postgres --net=host quay.io/coreos/postgres
core@core-01 ~ $ sudo rkt enter af7a88d0 /bin/sh
/ # cat /etc/hosts
127.0.0.1       localhost
127.0.1.1       buildroot
/ # uname -a
Linux postgres 4.6.4-coreos #1 SMP Thu Jul 14 20:36:35 UTC 2016 x86_64 GNU/Linux
/ # ping localhost -c 4
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: seq=0 ttl=64 time=0.028 ms
64 bytes from 127.0.0.1: seq=1 ttl=64 time=0.038 ms
64 bytes from 127.0.0.1: seq=2 ttl=64 time=0.036 ms
64 bytes from 127.0.0.1: seq=3 ttl=64 time=0.034 ms
/ # ping postgres
ping: bad address 'postgres'

I first documented this example in https://groups.google.com/d/msg/rkt-dev/pjJ6knYiB-Q/pdZyzL6TAgAJ.

[lucab]

@derekmahar's problem seems to be a bit different, as the quay.io/coreos/postgres image already comes with an /etc/hosts file that we don't touch, thus at runtime it doesn't match the custom hostname.

[mgoltzsche]

I stumbled accross this issue in a different context...

When /etc/hosts is generated the fully qualified domain name (FQDN) should not appear as last but as first name followed by the short hostname.
The hosts man page documents the syntax for hosts entries as follows:

IP_address canonical_hostname [aliases...]

But when rkt parameters --hostname=hostname --dns-domain=example.org are passed and /etc/hosts generated currently it looks like this:
127.0.0.1 localhost localhost.domain localhost4 localhost4.localdomain4 hostname.example.org
At least on centos and fedora the pod cannot resolve its domain that way and resolves its machine name to localhost. Unfortunately one cannot even work around this by adding --hosts-entry=127.0.0.1=hostname.example.org hostname since the 1st entry is still used as domainname.
To make it also resolve properly on these distributions /etc/hosts should be generated with this line instead:
127.0.0.1 hostname.example.org hostname localhost localhost.domain localhost4 localhost4.localdomain4
Additionally the --dns-domain parameter value could be derived from the --hostname parameter when the latter is provided with a FQDN.

Alternatively the FQDN could be mapped to the pod's public IP in /etc/hosts like this:
172.16.28.2 hostname.example.org hostname
Since the provided FQDN is supposed to be accessed from outside the pod the name should also resolve to the public interface's IP within the pod by default.
To do this the user/caller would have to choose one of the pod's public IPs and pass it to rkt's --hosts-entry parameter. Unfortunately in most cases the caller doesn't know the pod IP before the pod has been started. This wouldn't be a problem if rkt supports the following parameter format:
--hosts-entry="default=hostname.example.org hostname"
where default is a network's name bound to the pod. rkt could resolve this name to the pod's actual IP within the referenced network. This way the user/caller could configure proper FQDN resolution herself.
This approach is consistent with the current CLI since the /etc/hosts file is only generated when a --hosts-entry parameter is provided. It would also support mapping app names within the pod to the public pod IP (which would also be very helpful to me).

The current workaround to this issue is to generate a temporary hosts file, mount it into the pod and do not pass --hosts-entry parameters to not override it. This workaround still doesn't support resolution of the hostname to the pod's public IP since it is unknown at generation time. Hence the hostname is resolved to 127.0.0.1.