Toolbox: /etc/resolv.conf is broken when it's an absolute symbolic link on the host

[refi64]

systemd-resolved has you symlink /etc/resolv.conf to some other location (under /run/systemd/resolve). When the host volume /etc is mounted, this location of course does not exist, so all hostname access within the container fails.

[debarshiray]

Yeah, that wouldn't work. We could bind mount /run too or switch to using the flatpak-spawn maintained copies.

What does ls -l /etc/resolv.conf show on the host? Could you please paste the output here?

[refi64]

lrwxrwxrwx. 1 root root 32 Jun  6 13:23 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

like the resolved documentation mentions.

[debarshiray]

So, Ubuntu uses systemd-resolved since 16.04.

[debarshiray]

I think the best way forward is to switch the /etc/resolv.conf symbolic link to the copy maintained by flatpak-session-helper at /run/host/monitor/resolv.conf.

However, I'd like to wait a bit longer for https://github.com/flatpak/flatpak/pull/2916 to be more widely installed, to avoid having some unintended fallout. Ultimately, Toolbox's primary use-case is Silverblue, and it doesn't use systemd-resolved, so it's better to lean in favour of it.

[debarshiray]

I'd like to wait a bit longer for flatpak/flatpak#2916
to be more widely installed, to avoid having some unintended fallout.
Ultimately, Toolbox's primary use-case is Silverblue, and it doesn't use
systemd-resolved, so it's better to lean in favour of it.

https://github.com/flatpak/flatpak/pull/2916 was introduced in Flatpak 1.4.0, which is too new for Fedora 29. So we need to wait until Fedora 29 is end-of-life:d before we make this change.

[TingPing]

So we need to wait until Fedora 29 is end-of-life:d before we make this change.

Since this date is approaching is there any work on this? Its a bit of a problem to simply not work with many configurations...

EDIT: Seems like you can modify toolbox to use /run/host/monitor/resolv.conf instead of /run/host/etc/resolv.conf.

[fansari]

Seems to work with /run/host/monitor/resolv.conf. I set the link stopped the box and started again. The link remained intact.

[sigmavirus24]

I can confirm @fansari's results with @TingPing's work around. I'm running Fedora 31 Workstation, toolbox-0.0.17-1.fc31.src.rpm, podman-1.6.2-2.fc31.src.rpm, and registry.fedoraproject.org/f31/fedora-toolbox 31 a198bc8c3cda 4 weeks ago 448 MB and having this issue. Let me know if I can provide any other details to help get this resolved (no pun intended).

[fansari]

When will this issue been fixed? I have now again created directory /run/host/monitor (which does not exist on Fedora 31) removed the NetworkManager link and place the resolv.conf there.

toolbox should work with NetworkManager without this issue.

[abitrolly]

Can't add custom DNS servers to /etc/resolv.conf from toolbox. I had to copy it, unlink and copy back.

[imcsk8]

The problem is that in Fedora 31 /etc/resolv.conf is a symlink to /run/host/etc/resolv.conf which in turn is a symlink to /var/run/NetworkManager/resolv.conf, this file does not exist in the container since it does not use NetworkManager.

The registry.fedoraproject.org/f31/fedora-toolbox:31 image has to be fixed to avoid this problem.

I found this Workaround:

⬢[ichavero@toolbox ~]$ sudo -i
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
⬢[root@toolbox ~]# ping google.com
ping: google.com: Name or service not known
⬢[root@toolbox ~]# mkdir /var/run/NetworkManager
⬢[root@toolbox ~]# echo "nameserver 192.168.1.254" > /var/run/NetworkManager/resolv.conf
⬢[root@toolbox ~]# ping -c2 google.com
PING google.com (216.58.217.14) 56(84) bytes of data.
64 bytes from qro02s15-in-f14.1e100.net (216.58.217.14): icmp_seq=1 ttl=57 time=6.74 ms
64 bytes from qro02s15-in-f14.1e100.net (216.58.217.14): icmp_seq=2 ttl=57 time=5.63 ms

--- google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 5.632/6.187/6.742/0.555 ms
⬢[root@toolbox ~]# 

[martinpitt]

I sent a fix for that in PR #380 a while ago.

[Liambeguin]

I tested #380 and it works great for me with systemd-resolved.

Thanks

[fansari]

I have created a toolbox for Fedora 32 Silverblue and again ran into the same issue.
Only quick workaround was again to remove the link /etc/resolv.conf and set it to /run/host/monitor.resolv.conf.

[DanMan]

I'm running into this issue right now with Fedora 31 WS that has been upgraded since 24 I think. So it'd be nice, if someone could merge Martin's PR 380. The best software is worthless, if it only creates more problems.

[Jmennius]

Just making sure we stay on top of it :)
Fedora is aiming to enable systemd-resolved by default in Fedora 33 (link).

[martinpitt]

This is tracked in https://bugzilla.redhat.com/show_bug.cgi?id=1785244 as well.

[debarshiray]

The problem is that in Fedora 31 /etc/resolv.conf is a symlink to
/run/host/etc/resolv.conf which in turn is a symlink to
/var/run/NetworkManager/resolv.conf, this file does not exist in
the container since it does not use NetworkManager.

Umm... not quite.

I suppose, what you meant is that on Fedora 31 hosts, /etc/resolv.conf is a symbolic link to /run/NetworkManager/resolv.conf (because /var/run is a symbolic link to /run).

/etc/resolv.conf isn't meant to be a symbolic link on any Fedora hosts prior to Fedora 33. From Fedora 33 onwards, it's going to be managed by systemd-resolved and will indeed be a symbolic link. You can verify that by installing a Fedora < 33 system from scratch in a virtual machine or something.

However, due to an old (or passing?) NetworkManager bug, Fedora systems that have undergone a series of upgrades from one release to another for a good few years can end up with /etc/resolv.conf being a symbolic link as you mention.

[debarshiray]

An update about the impending systemd-resolved change in Fedora 33.

By default, Fedora will ship /etc/resolv.conf as a relative symbolic link to ../run/systemd/resolve/stub-resolv.conf, and so does Ubuntu. This already works since Toolbox 0.0.14 because of commit d63b0a9c0f1cd8a137ebc818b297f3b9303b5c32

This situation is an illustration of the benefits of a relative symbolic link. It ensures that the link continues to resolve properly even when the prefix or the root is changed.

So, if you need a quick solution that works with Toolbox as it's shipping today, I'd suggest switching to a relative symbolic link on your systems.

What doesn't work, and we need to fix, are absolute symbolic links. ie., a /etc/resolv.conf that links to /run/systemd/resolve/stub-resolv.conf.