Toolbox: Connection refused when using kubectl from toolbox 33

[stove-panini]

Describe the bug
Running a kubectl command from within toolbox yields a connection refused error, no matter the target cluster. However, if kubectl is run from outside toolbox, or using another container such as bitnami/kubectl it works as expected.

Steps how to reproduce the behaviour

1. Add the Google Cloud SDK repo for el8 to your toolbox 33 container
2. Install kubectl
3. Run any kubectl command (using any kubeconfig)
4. See error

Expected behaviour
The kubectl command works without any connectivity error, as it does from other containers or on the host OS.

Actual behaviour
The command returns the following:

The connection to the server <my server name> was refused - did you specify the right host or port?

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.96

Toolbox package info (rpm -q toolbox)
toolbox-0.0.96-1.fc33.x86_64

Output of podman version

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Wed Oct  7 12:21:20 2020
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-2.1.1-12.fc33.x86_64

Info about your OS
Fedora Silverblue 33

Additional context
This used to work properly in Silverblue 32 using the Fedora 32 image.

Also note that I can ping and ssh to the target machine from within toolbox. Nmap from within toolbox shows the port to be open.

[fedgiac]

I had the same problem after updating Silverblue from version 32 to version 33.

A workaround is replacing the symlink present in /etc/resolv.conf in the toolbox container with the host resolv.conf:

sudo rm /etc/resolv.conf && sudo cp /run/host/etc/resolv.conf /etc

After that I could run kubectl get pods without any issue.

Before this change /etc/resolv.conf pointed to ../run/systemd/resolve/stub-resolv.conf (which did not exist).
After rebooting the custom /etc/resolv.conf was replaced by a symlink to the host resolv.conf file: /run/host/etc/resolv.conf.
kubectl also appears to work with the new symlink.

[stove-panini]

Many thanks @fedgiac! That was it, indeed.

[stove-panini]

I wonder why this occurred in the first place. The correct symlink should've been set up here:
https://github.com/containers/toolbox/blob/18e3955fc0f7bedb7b95f4bb4d6f19be65011147/toolbox#L1206

[fedgiac]

The original /etc/resolv.conf was also a symlink, so ! readlink /etc/resolv.conf always succeeded and the link wasn't updated at this point. Replacing this file with any other symlink should make this file stable across reboots.

Part of the issue is that the original /etc/resolv.conf was a _relative_ symlink, so it didn't undergo proper host path replacement, you can see the comments in the code about the topic.

[debarshiray]

I am glad that you managed to figure this one out!

In case you want to pursue this further, then here are some pointers that might be of help.

First of all, make sure that you are using the Go implementation of /usr/bin/toolbox, and not the POSIX shell implementation. :)

The container's /etc/resolv.conf should generally be a symbolic link to /run/host/etc/resolv.conf, regardless of whether the host operating system is using systemd-resolved or some other mechanism to manage /etc/resolv.conf.

If the host operating system is using systemd-resolved, then the host's /etc/resolv.conf should be a relative symbolic link to something like ../run/systemd/resolve/stub-resolv.conf.

Lastly, the host's /etc should be available at /run/host/etc inside the container.

[debarshiray]

Closing this. Feel free to re-open or leave a comment, if you have something new to add.