Kind: [probably already fixed] cmder: stderr can end up in stdout even if using cmd.SetStdout()

[neolit123]

What happened:
running a command in a node-image container that has klog stderr output and using SetStdout on the command can result in the stderr output present in the io.Writer/buffer set for stdout.

How to reproduce it (as minimally and precisely as possible):

kubeadm needs to be 1.14.x

var etcdVersionBuf bytes.Buffer
cmder := docker.ContainerCmder(containerName)
cmd := cmder.Command("/bin/sh", "-c", "kubeadm config images list | grep etcd")
cmd.SetStdout(&etcdVersionBuf)
cmd.Run()

if the kubeadm command outputs a klog warning it will end up in etcdVersionBuf.
contents of etcdVersionBuf:

I0626 02:54:47.931193    3673 version.go:240] remote version
  is much newer: v1.15.0; falling back to: stable-1.14
k8s.gcr.io/etcd:3.3.10

current fix is to do this:

cmd := cmder.Command("/bin/sh", "-c", "kubeadm config images list 2> /dev/null | grep etcd")

i don't see issues in the kind code, so maybe a Go thing? please close if this is expected.
locally if i do:

kubeadm config images list | grep etcd > test.log

test.log only contains k8s.gcr.io/etcd:3.3.10

Anything else we need to know?:

xref https://github.com/kubernetes/kubeadm/pull/1636

Environment:
kinder

• kind version: (use kind version):
• Kubernetes version: (use kubectl version): ci/latest-1.14
• Docker version: (use docker info): 18.06.3-ce
• OS (e.g. from /etc/os-release):

NAME="Ubuntu"
VERSION="17.10 (Artful Aardvark)"

[neolit123]

/priority awaiting-more-evidence

[BenTheElder]

is it possible it's using some old kind version? I can't find any logical errors in the current code, stdout and stderr are managed distinctly.

[BenTheElder]

I think this bug was (mostly) fixed 2 months ago, it seems the pseudo tty from docker exec -t includes both stdout and stderr. we will only choose this if the cmder stdout or stderr are wired to a terminal currently. we should probably not support that and never allocate one.

[BenTheElder]

specifically: https://github.com/kubernetes-sigs/kind/commit/97d0968a8e505382088587afeb6f8814a9fb5e6f

[neolit123]

thanks, seems like https://github.com/kubernetes-sigs/kind/commit/97d0968a8e505382088587afeb6f8814a9fb5e6f might be a fix indeed.
kinder is still at https://github.com/kubernetes-sigs/kind/commit/161151a26faf

will confirm once we move to latest - hopefully early next month.
/assign

[bibz0r]

I was experiencing this issue when following a tutorial on how to create a kubernetes cluster with ansible. (https://www.digitalocean.com/community/tutorials/how-to-create-a-kubernetes-cluster-using-kubeadm-on-ubuntu-18-04) In the ansible playbook there is this:

shell: kubeadm init --pod-network-cidr=10.244.0.0/16 >> cluster_initialized.txt

But the ansible recipe fails. When executing the command directly it produces the same error neolit123 experienced:

 remote version
  is much newer: v1.15.0; falling back to: stable-1.14

Now why I'm even posting this, here is the simple fix for it to work:

shell: kubeadm init --pod-network-cidr=10.244.0.0/16

Sorry for hijacking this, its the only thread I've found through google :)

[neolit123]

@bibz0r

hi,

it produces the same error neolit123 experienced:

it's technically a warning on stderr. it does not exit(1).
does ansible treat stderr output as errors and exits immediately?

shell: kubeadm init --pod-network-cidr=10.244.0.0/16

hm, i don't see how passing --pod-network-cidr will silence the kubeadm version checks. did you missing adding --kubernetes-version in your example? basically if you don't pass --kubernetes-version or kubernetesVersion (in the config) kubeadm tries to get the latest stable version e.g. 1.15.1, but falls back if the latest stable is too new. i don't exactly agree with this logic, and it might get removed soon.

[neolit123]

/close
let's close this and re-open if it manifests again.

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

/close
let's close this and re-open if it manifests again.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.