Kubeadm: "kubeadm join" Hostname pre-flight check fails to recognize hostname

[seh]

What keywords did you search in kubeadm issues before filing this one?

• _join_
• _hostname_
• _lookup_
• _"no such host"_

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

• _1.10.2_
  > kubeadm version: &version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
• _v1.11.0-alpha.2_ .
  > kubeadm version: &version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.2", GitCommit:"ed9b25c90241b2b8a1fa10b96381c57f99ca952a", GitTreeState:"clean", BuildDate:"2018-05-02T12:02:56Z", GoVersion:"go1.10.1", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"81753b10df112992bf51bbc2c2f85208aad78335", GitTreeState:"clean", BuildDate:"2018-04-27T09:10:24Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

AWS EC2 instance, but no Kubernetes cloud provider installed or activated.

• OS (e.g. from /etc/os-release):

NAME="Container Linux by CoreOS"
ID=coreos
VERSION=1688.5.3
VERSION_ID=1688.5.3
BUILD_ID=2018-04-03-0547
PRETTY_NAME="Container Linux by CoreOS 1688.5.3 (Rhyolite)"
ANSI_COLOR="38;5;75"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://issues.coreos.com"
COREOS_BOARD="amd64-usr"

• Kernel (e.g. uname -a):

Linux ip-10-103-0-183 4.14.32-coreos #1 SMP Tue Apr 3 05:21:26 UTC 2018 x86_64 Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz GenuineIntel GNU/Linux

• Others:

My _/etc/hosts_ file has only the following two entries:

127.0.0.1       localhost
::1             localhost

What happened?

When running _kubeadm join_ (with the --config flag, at least), it fails to detect my machine's hostname, and the "Hostname" pre-flight check fails:

$ sudo kubeadm-alpha join --config ./node-config.yaml 
W0517 17:47:19.298833    2556 flags.go:39] log_backtrace_at is DEPRECATED and will be removed in a future version. Use log-backtrace-at instead.
W0517 17:47:19.299530    2556 flags.go:39] log_backtrace_at is DEPRECATED and will be removed in a future version. Use log-backtrace-at instead.
W0517 17:47:19.299535    2556 flags.go:39] log_dir is DEPRECATED and will be removed in a future version. Use log-dir instead.
W0517 17:47:19.299541    2556 flags.go:39] log_dir is DEPRECATED and will be removed in a future version. Use log-dir instead.
I0517 17:47:19.300854    2556 join.go:228] [preflight] running pre-flight checks
    [WARNING FileExisting-crictl]: crictl not found in system path
Suggestion: go get github.com/kubernetes-incubator/cri-tools/cmd/crictl
I0517 17:47:19.321199    2556 kernel_validator.go:81] Validating kernel version
I0517 17:47:19.321257    2556 kernel_validator.go:96] Validating kernel config
    [WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.12.1-ce. Max validated version: 17.03
    [WARNING Hostname]: hostname "" could not be reached
    [WARNING Hostname]: hostname "" lookup : no such host
[preflight] Some fatal errors occurred:
    [ERROR Hostname]: hostname "" a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

Note, though, that my machine has a hostname, though it's not registered with DNS:

$ hostname
ip-10-103-0-183

$ nslookup ip-10-103-0-183
Server:     10.103.0.130
Address:    10.103.0.130#53

Non-authoritative answer:
Name:   ip-10-103-0-183.ec2.internal
Address: 10.103.0.183

What you expected to happen?

_kubeadm join_ should have detected my machine's hostname and carried on without complaint.

How to reproduce it (as minimally and precisely as possible)?

To match my scenario,

• Run an AWS EC2 instance that's not yet part of this Kubernetes cluster.
• Create files _node-config.yaml_:
  _(This may be immaterial.)_

apiVersion: kubeadm.k8s.io/v1alpha1
kind: NodeConfiguration
tlsBootstrapToken: 4d4c4k.6u9tj98fukd68f8c
discoveryFile: /etc/kubernetes/kubeadm/discovery.yaml

• Create file _discovery.yaml_:
  _(This may be immaterial.)_

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64-encoded CA certificate bundle>
    server: <API server endpoint>
  name: ""
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

• Run the following command:
  _kubeadm join --config node-config.yaml_
• Observe that the "Hostname" pre-flight check complains about the hostname being empty.

Anything else we need to know?

It's possible to work around this problem by skipping the "Hostname" pre-flight check.

I first mentioned this problem in the long-closed kubernetes/kubernetes#49065. @erikh has ideas for what still be wrong here.

[luxas]

(sorry didn't have time to read through yet)
@timothysc I think we should try to fix in time for v1.11

[dixudx]

/assign

[luxas]

I believe https://github.com/kubernetes/kubernetes/pull/64974 will help kinda-solve this.
We _should not_ do validation of the node name in our preflight checks, but instead in the normal config API validation that is happening _before_ the preflight checks run.

So https://github.com/kubernetes/kubernetes/pull/64974 contains an UX improvement related to this, instead of failing with weird preflight errors, kubeadm will earlier say "you need to set --node-name", as os.Hostname() doesn't seem to work in your env :(. Why os.Hostname() doesn't work for you I can't say, but I don't know enough about your env I guess to say _how_ kubeadm could detect the hostname that os.Hostname() can't (maybe reading from /proc directly as a backup?). In any case, such a thing can only merge in time for v1.12.

@seh what do you think about this?

[seh]

I think that helps, as it tells the user what to do to work around the problem.

I haven't tried yet to compile my own Go program calling on os.Hostname() to see if it behaves the same way as _kubeadm_ does here (failing to find a nonempty name), but I'll report back if I learn more on that front.

[luxas]

Thanks! Then I'm gonna keep this in the v1.11 milestone, close this as partially-fixed in v1.11 after the PR merges, and we can then open a new issue targeting v1.12 for better auto-detection of the hostname if we can come up with such a thing.

[luxas]

https://github.com/kubernetes/kubernetes/pull/64974 is merged :tada:
Thanks @seh for posting this issue, very helpful to get at least this part fixed!