Rkt: Improve vague error: "run: discovery failed"

[waisbrot]

I'm interested in a better error message that would have saved me some debugging headache.

Environment

rkt Version: 1.11.0
appc Version: 0.8.5
Go Version: go1.5.4
Go OS/Arch: linux/amd64
Features: -TPM +SDJOURNAL

--

Linux 4.7.0-coreos x86_64

--

NAME=CoreOS
ID=coreos
VERSION=1122.1.0
VERSION_ID=1122.1.0
BUILD_ID=2016-08-11-0252
PRETTY_NAME="CoreOS 1122.1.0 (MoreOS)"
ANSI_COLOR="1;32"
HOME_URL="https://coreos.com/"
BUG_REPORT_URL="https://github.com/coreos/bugs/issues"

--

systemd 229
+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS -ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN

What did you do?

Attempted to launch flanneld (via cloud-init) on a machine with no access to the Internet (an AWS EC2 instance without a NAT or public IP).

What did you expect to see?

An error message that told me either that the problem was occurring in rkt (and not in flannel) or an error message that clearly indicated some network failure (e.g. "unable to connect to 1.2.3.4")

What did you see instead?

run: discovery failed

[lucab]

Hi, thanks for the report and sorry for the pain :(

Do you mind sharing the whole execution output and possibly also a --debug run to better pinpoint where that is coming from?

[waisbrot]

The machine in question is no longer broken, so I tried to reproduce to reproduce on a VM with the network interfaces turned off. Running rkt run quay.io/coreos/flannel in the shell, I see the output

image: using image from file /usr/lib64/rkt/stage1-images/stage1-coreos.aci
image: searching for app image quay.io/coreos/flannel
run: discovery failed

That's not as bad as when it was bothering me -- I was only spotting the last line in journalctl. I'd still like for any output from rkt to be clear on a line-by-line basis whether it's coming from rkt or from the code that rkt is launching.

With the --debug flag, I see

image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.13.0
image: searching for app image quay.io/coreos/flannel
image: meta tag 'ac-discovery' not found on quay.io/coreos/flannel: Get https://quay.io/coreos/flannel?ac-discovery=1: dial tcp: lookup quay.io: Temporary failure in name resolution
image: meta tag 'ac-discovery' not found on quay.io/coreos: Get https://quay.io/coreos?ac-discovery=1: dial tcp: lookup quay.io: Temporary failure in name resolution
image: meta tag 'ac-discovery' not found on quay.io: Get https://quay.io?ac-discovery=1: dial tcp: lookup quay.io: Temporary failure in name resolution
run:
  └─discovery failed for "quay.io/coreos/flannel"
    └─discovery failed

[antoineco]

On a broader level I also noticed that flannel itself does provide information about failures when ran on the command line, while journalctl seems to trim the interesting part of the output.

[discordianfish]

Running into this too, maybe in my case it's because I try to use the gcr.io hyperkube image? (Because quay is unusable slow right now)

$ sudo /usr/bin/rkt run --uuid-file-save=/var/run/kubelet-pod.uuid \
   --volume coreos-etc-kubernetes,kind=host,source=/etc/kubernetes,readOnly=false --volume coreos-etc-ssl-certs,kind=host,source=/etc/ssl/certs,readOnly=true --volume coreos-usr-share-certs,kind=host,source=/usr/share/ca-certificates,readOnly=true --volume coreos-var-lib-docker,kind=host,source=/var/lib/docker,readOnly=false --volume coreos-var-lib-kubelet,kind=host,source=/var/lib/kubelet,readOnly=false,recursive=true --volume coreos-var-log,kind=host,source=/var/log,readOnly=false --volume coreos-os-release,kind=host,source=/usr/lib/os-release,readOnly=true --volume coreos-run,kind=host,source=/run,readOnly=false --volume coreos-lib-modules,kind=host,source=/lib/modules,readOnly=true --mount volume=coreos-etc-kubernetes,target=/etc/kubernetes --mount volume=coreos-etc-ssl-certs,target=/etc/ssl/certs --mount volume=coreos-usr-share-certs,target=/usr/share/ca-certificates --mount volume=coreos-var-lib-docker,target=/var/lib/docker --mount volume=coreos-var-lib-kubelet,target=/var/lib/kubelet --mount volume=coreos-var-log,target=/var/log --mount volume=coreos-os-release,target=/etc/os-release --mount volume=coreos-run,target=/run --mount volume=coreos-lib-modules,target=/lib/modules --stage1-from-dir=stage1-fly.aci \
   gcr.io/google_containers/hyperkube-amd64:v1.7.6 \
   --exec=/kubelet -- --kubeconfig /etc/kubernetes/kubeconfig --pod-manifest-path=/etc/kubernetes/manifests

run: discovery failed

[discordianfish]

Looks like I needed to use docker://. I suspect without it, it uses the rkt protocol and since it can reach the registry, I don't get a connection error but this 'discovery failed'.

[unixfox]

Sorry for bumping this issue, I had the same error message and it was caused by the fact that iptables wasn't installed (I installed rkt from the official ubuntu repo, so I expected iptables to be installed with rkt).