Nextcloud-snap: Trouble with enable-https, letsencrypt "Name or service not known"

[lindhe]

So I can't seem to create a certificate. I installed a server a month ago to try it out, and then it worked perfectly. Now, it doesn't.

I'm running nextcloud 11.0.0snap3 on Unbuntu Server 16.10.

This is what I get:

~> sudo nextcloud.enable-https -d
In order for Let's Encrypt to verify that you actually own the domain for
which you're requesting a certificate, there are a number of requirements
of which you need to be aware:

1. In order to register with the Let's Encrypt ACME server, you must agree
   to the currently-in-effect Subscriber Agreement located here:

       https://letsencrypt.org/repository/

   By continuing to use this tool you agree to these terms. Please cancel
   now if otherwise.

2. You must have the domain name(s) for which you want certificates
   pointing at the external IP address of this machine.

3. Both ports 80 and 443 on the external IP address of this machine must
   point to this machine (e.g. port forwarding might need to be setup on
   your router).

Have you met these requirements? (y/n) y  
Please enter an email address (for urgent notices or key recovery): [email protected]
Please enter your domain name(s) (space-separated): example.com
Attempting to obtain certificates... error running certbot:

Saving debug log to /var/snap/nextcloud/current/certs/certbot/logs/letsencrypt.log
An unexpected error occurred:
ConnectionError: HTTPSConnectionPool(host='acme-staging.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f81b8bd6910>: Failed to establish a new connection: [Errno -2] Name or service not known',))
Please see the logfiles in /var/snap/nextcloud/current/certs/certbot/logs for more details.

(i obviously didn't use example.com when I ran it)

This is what the log says:

2017-01-17 14:39:50,777:DEBUG:certbot.main:Root logging level set at 20
2017-01-17 14:39:50,780:INFO:certbot.main:Saving debug log to /var/snap/nextcloud/current/certs/certbot/logs/letsencrypt.log
2017-01-17 14:39:50,781:DEBUG:certbot.main:certbot version: 0.9.3
2017-01-17 14:39:50,781:DEBUG:certbot.main:Arguments: ['--text', '--config-dir', '/var/snap/nextcloud/current/certs/certbot/config', '--work-dir', '/var/snap/nextcloud/current/certs/certbot/work', '--logs-dir', '/var/snap/nextcloud/current/certs/certbot/logs', '--dry-run', '--authenticator', 'nextcloud:webroot', '--nextcloud:webroot-path', '/var/snap/nextcloud/current/certs/certbot', '--rsa-key-size', '4096', '--email', '[email protected]', '--non-interactive', '--agree-tos', '--force-renewal', '-d', 'example.com']
2017-01-17 14:39:50,783:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone,PluginEntryPoint#nextcloud:webroot)
2017-01-17 14:39:50,784:DEBUG:certbot.plugins.selection:Requested authenticator nextcloud:webroot and installer None
2017-01-17 14:39:50,795:DEBUG:certbot.plugins.selection:Single candidate plugin: * nextcloud:webroot
Description: Place files in webroot directory without running chown
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot_nextcloud_plugin.webroot:Authenticator
Initialized: <certbot_nextcloud_plugin.webroot.Authenticator object at 0x7f81b8bdda90>
Prep: True
2017-01-17 14:39:50,797:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_nextcloud_plugin.webroot.Authenticator object at 0x7f81b8bdda90> and installer None
2017-01-17 14:39:54,419:DEBUG:root:Sending GET request to https://acme-staging.api.letsencrypt.org/directory. args: (), kwargs: {}
2017-01-17 14:39:54,580:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
2017-01-17 14:39:54,585:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/nextcloud/559/bin/certbot", line 11, in <module>
    sys.exit(main())
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/main.py", line 776, in main
    return config.func(config, plugins)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/main.py", line 558, in obtain_cert
    le_client = _init_le_client(config, auth, installer)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/main.py", line 368, in _init_le_client
    acc, acme = _determine_account(config)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/main.py", line 353, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/client.py", line 118, in register
    acme = acme_from_config_key(config, key)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/certbot/client.py", line 42, in acme_from_config_key
    return acme_client.Client(config.server, key=key, net=net)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/acme/client.py", line 63, in __init__
    self.net.get(directory).json())
  File "/snap/nextcloud/559/lib/python2.7/site-packages/acme/client.py", line 624, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/acme/client.py", line 606, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/snap/nextcloud/559/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPSConnectionPool(host='acme-staging.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f81b8bd6910>: Failed to establish a new connection: [Errno -2] Name or service not known',))

I couldn't find an issue on it, but feel free to yell at me if I'm a dummy and missed it.

[kyrofa]

Haha, you're not a dummy! It doesn't really look snap-specific though, so I did some quick research, and found this. Have you seen that? Any chance it's a proxy issue?

[lindhe]

It is a fresh install, and I did not (or at least I think so) set any proxy during installation. echo $http_proxy and echo $https_proxy returns empty string, so I guess they are unset.

I don't know where to try and run sudo -E ./letsencrypt-auto certonly --standalone -d myfqdn, as mentioned by the poster. Where is that file?

I'll try to bypass my router, to see if that gives any luck.

EDIT: Bypassing the router didn't help.

[lindhe]

Do you think I should post this somewhere else?

[kyrofa]

Yeah I think you should ask on the Let's Encrypt forums. Note that instead of running sudo -E letsencrypt-auto [...] you would just run sudo -E nextcloud.enable-https (which calls certbot for you).

[kyrofa]

@lindhe please let me know the output of the following command:

PYTHONPATH=/snap/nextcloud/current/lib/python2.7/site-packages /snap/nextcloud/current/usr/bin/python -c 'import requests; print requests.get("https://acme-staging.api.letsencrypt.org/directory").text'

[lindhe]

{
  "key-change": "https://acme-staging.api.letsencrypt.org/acme/key-change",
  "new-authz": "https://acme-staging.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-staging.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-staging.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-staging.api.letsencrypt.org/acme/revoke-cert"
}

[kyrofa]

Alright, now let's run the same thing from within confinement:

snap run --shell nextcloud.enable-https

That will drop you into a new shell. Please let me know the output of the following command from within this shell:

PYTHONPATH=$SNAP/lib/python2.7/site-packages $SNAP/usr/bin/python -c 'import requests; print requests.get("https://acme-staging.api.letsencrypt.org/directory").text'

[lindhe]

Got same result both as user andreas and root.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/snap/nextcloud/862/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/snap/nextcloud/862/lib/python2.7/site-packages/OpenSSL/crypto.py", line 13, in <module>
    from cryptography.hazmat.primitives.asymmetric import dsa, rsa
  File "/snap/nextcloud/862/lib/python2.7/site-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 14, in <module>
    from cryptography.hazmat.backends.interfaces import RSABackend
  File "/snap/nextcloud/862/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 7, in <module>
    import pkg_resources
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3030, in <module>
    @_call_aside
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3014, in _call_aside
    f(*args, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3043, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 645, in _build_master
    ws = cls()
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 638, in __init__
    self.add_entry(entry)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 694, in add_entry
    for dist in find_distributions(entry, True):
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2010, in find_on_path
    path_item_entries = _by_version_descending(os.listdir(path_item))
OSError: [Errno 13] Permission denied: '/home/andreas'

[kyrofa]

Did you use exactly the PYTHONPATH I said? It seems to be looking in your home directory for some unknown reason.

[lindhe]

Yes, I copied it.

[lindhe]

Now I tried to substitute the $SNAP variable with the explicit path, same error.

Command:
PYTHONPATH=$SNAP/lib/python2.7/site-packages /snap/nextcloud/862/usr/bin/python -c 'import requests; print requests.get("https://acme-staging.api.letsencrypt.org/directory").text'

Output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/__init__.py", line 52, in <module>
    from .packages.urllib3.contrib import pyopenssl
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 46, in <module>
    import OpenSSL.SSL
  File "/snap/nextcloud/862/lib/python2.7/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/snap/nextcloud/862/lib/python2.7/site-packages/OpenSSL/crypto.py", line 13, in <module>
    from cryptography.hazmat.primitives.asymmetric import dsa, rsa
  File "/snap/nextcloud/862/lib/python2.7/site-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 14, in <module>
    from cryptography.hazmat.backends.interfaces import RSABackend
  File "/snap/nextcloud/862/lib/python2.7/site-packages/cryptography/hazmat/backends/__init__.py", line 7, in <module>
    import pkg_resources
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3030, in <module>
    @_call_aside
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3014, in _call_aside
    f(*args, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3043, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 645, in _build_master
    ws = cls()
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 638, in __init__
    self.add_entry(entry)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 694, in add_entry
    for dist in find_distributions(entry, True):
  File "/snap/nextcloud/862/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2010, in find_on_path
    path_item_entries = _by_version_descending(os.listdir(path_item))
OSError: [Errno 13] Permission denied: '/home/andreas'

[lindhe]

When I cd $SNAP before running the command you gave, I get this instead:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named requests
andreas@server:/snap/nextcloud/862$ /snap/nextcloud/862/usr/bin/python -c 'import requests; print requests.get("https://acme-staging.api.letsencrypt.org/directory").text'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named requests
andreas@server:/snap/nextcloud/862$ PYTHONPATH=$SNAP/lib/python2.7/site-packages $SNAP/usr/bin/python -c 'import requests; print requests.get("https://acme-staging.api.letsencrypt.org/directory").text'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/adapters.py", line 487, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-staging.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9c8045f790>: Failed to establish a new connection: [Errno -2] Name or service not known',))

[kyrofa]

Alright interesting, that's a better result. Okay get back into the confined shell and cd into $SNAP again

snap run --shell nextcloud.enable-https
cd $SNAP

Now try to ping that domain:

ping acme-staging.api.letsencrypt.org

What does it say?

Also, from a normal (unconfined) shell, can I see the output of snap interfaces please?

[lindhe]

Oh, this was rather interesting.

Command:
ping acme-staging.api.letsencrypt.org

Output:
bash: /bin/ping: Permission denied

---

Command:
snap interfaces

Output:

Slot                      Plug
:alsa                     -
:avahi-observe            -
:bluetooth-control        -
:browser-support          -
:camera                   -
:cups-control             -
:dcdbas-control           -
:docker-support           -
:firewall-control         -
:fuse-support             -
:gsettings                -
:hardware-observe         -
:home                     -
:io-ports-control         -
:kernel-module-control    -
:libvirt                  -
:locale-control           -
:log-observe              -
:lxd-support              -
:modem-manager            -
:mount-observe            -
:network                  nextcloud
:network-bind             nextcloud
:network-control          -
:network-manager          -
:network-observe          -
:network-setup-observe    -
:ofono                    -
:opengl                   -
:openvswitch              -
:openvswitch-support      -
:optical-drive            -
:physical-memory-control  -
:physical-memory-observe  -
:ppp                      -
:process-control          -
:pulseaudio               -
:raw-usb                  -
:removable-media          -
:screen-inhibit-control   -
:shutdown                 -
:snapd-control            -
:system-observe           -
:system-trace             -
:time-control             -
:timeserver-control       -
:timezone-control         -
:tpm                      -
:unity7                   -
:upower-observe           -
:x11                      -
-                         nextcloud:removable-media

[kyrofa]

bash: /bin/ping: Permission denied

Whoops, I should have tested that before I asked you to. Indeed, the snap doesn't have permission to access that.

What I'm trying to drive at, though, is that there seems to be network issues, perhaps at the DNS level. Okay, try this from within that confined shell (again, cd to $SNAP):

PYTHONPATH=$SNAP/lib/python2.7/site-packages $SNAP/usr/bin/python -c 'import requests; print requests.get("https://104.126.140.214/directory").text'

Note that I simply replaced the domain name with the IP address. You should hopefully get an SSLError:

requests.exceptions.SSLError: hostname '104.126.140.214' doesn't match either of '*.api.letsencrypt.org', 'api.letsencrypt.org'

[lindhe]

Yep, that's what I get:

import requests; print requests.get("https://104.126.140.214/directory").text'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/snap/nextcloud/862/lib/python2.7/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: hostname '104.126.140.214' doesn't match either of '*.api.letsencrypt.org', 'api.letsencrypt.org'

[lindhe]

I thought I'd check dmesg if I could find some clues, an there is a bunch of these:

[  +0.004208] EXT4-fs (sdc9): VFS: Can't find ext4 filesystem
[  +0.003764] squashfs: SQUASHFS error: Can't find a SQUASHFS superblock on sdc9
[  +0.003879] FAT-fs (sdc9): bogus number of reserved sectors
[  +0.000077] FAT-fs (sdc9): Can't find a valid FAT filesystem
[  +0.016398] XFS (sdc9): Invalid superblock magic number

which I'm not very sure is because of nextcloud, but also a bunch of these:

[  +0.569803] audit: type=1400 audit(1485332837.502:34): apparmor="DENIED" operation="capable" profile="snap.nextcloud.mysql" pid=3526 comm="mysqld" capability=24  capname="sys_resource"
[  +0.462238] audit: type=1400 audit(1485332837.966:35): apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/systemctl" pid=3682 comm="mysql.server" requested_mask="x" denied_mask="x" fsui
[  +0.509604] audit: type=1400 audit(1485332838.474:36): apparmor="DENIED" operation="ptrace" profile="snap.nextcloud.mysql" pid=3811 comm="ps" requested_mask="trace" denied_mask="trace" peer="unconfined"

and

[Feb 8 21:13] audit: type=1400 audit(1486584833.119:2179): apparmor="DENIED" operation="open" profile="snap.nextcloud.renew-certs" name="/proc/27098/mounts" pid=27098 comm="python" requested_mask="r" denied_mask="
[Feb 9 12:26] audit: type=1400 audit(1486639627.411:2180): apparmor="DENIED" operation="open" profile="snap.nextcloud.enable-https" name="/proc/21546/mounts" pid=21546 comm="python" requested_mask="r" denied_mask=
[  +0.632705] audit: type=1400 audit(1486639628.043:2181): apparmor="DENIED" operation="open" profile="snap.nextcloud.enable-https" name="/home/andreas/" pid=21546 comm="python" requested_mask="r" denied_mask="r" 

which I thought might be related...

[kyrofa]

Those denials should be harmless.

Can I please see the output of cat /etc/resolv.conf from both your host system shell and the shell you obtain via snap run --shell? Do they differ?

[lindhe]

They are identical:

andreas@server ~ >cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.53

andreas@server:/home/andreas$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.0.53

[kyrofa]

May I also see /etc/hosts from both shells, please?

[lindhe]

andreas@server:/home/andreas$ cat /etc/hosts
127.0.0.1   localhost
127.0.1.1   server

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

(which btw is identical to the hosts file in a non-restricted shell)

[kyrofa]

That's interesting-- my /etc/hosts looks the same, but my /etc/resolv.conf contains nameserver 127.0.1.1 (Desktop, not Server), whereas yours contains nameserver 127.0.0.53, so you must be using systemd-resolved. Can I see the output of systemctl status systemd-resolved.service, please?

[lindhe]

Unconfined shell:

andreas@server ~ >systemctl status systemd-resolved.service
● systemd-resolved.service - Network Name Resolution
   Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/systemd-resolved.service.d
           └─resolvconf.conf
   Active: active (running) since Wed 2017-01-25 09:27:16 CET; 2 weeks 3 days ago
     Docs: man:systemd-resolved.service(8)
           http://www.freedesktop.org/wiki/Software/systemd/resolved
           http://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers
           http://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients
 Main PID: 3158 (systemd-resolve)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 3.1M
      CPU: 3.155s
   CGroup: /system.slice/systemd-resolved.service
           └─3158 /lib/systemd/systemd-resolved

Feb 09 21:32:23 server systemd-resolved[3158]: Processing query...
Feb 09 21:32:23 server systemd-resolved[3158]: Processing query...
Feb 10 05:45:44 server systemd-resolved[3158]: Processing query...
Feb 10 05:45:44 server systemd-resolved[3158]: Processing query...
Feb 10 21:33:19 server systemd-resolved[3158]: Processing query...
Feb 10 21:33:19 server systemd-resolved[3158]: Processing query...
Feb 11 06:36:44 server systemd-resolved[3158]: Processing query...
Feb 11 06:36:44 server systemd-resolved[3158]: Processing query...
Feb 11 10:15:16 server systemd-resolved[3158]: Processing query...
Feb 11 10:15:16 server systemd-resolved[3158]: Processing query...

Confined shell:

andreas@server:/home/andreas$ systemctl status systemd-resolved.service
bash: /bin/systemctl: Permission denied

Could that be it? That the snap does not have access to systemd-resolved?

[kyrofa]

Yes, it does sound like there might be some issues related to systemd-resolved and snappy. Sorry for the delay, I'm still investigating this and will report back when I've made some more progress.

[kyrofa]

Note that it's expected for the confined shell to be unable to access systemctl, that's not the issue. It's something more intrinsic to how systemd-resolved works.

[lindhe]

I understand. Time is hard to come by these days. I really appreciate your effort.

Have you been able to reproduce it?

[kyrofa]

Have you been able to reproduce it?

Yes actually, I have. This isn't an issue on Xenial as it doesn't use systemd-resolved, so I fired up a Yakkety Server VM and did indeed encounter this error. I just haven't been able to get to the bottom of why, just yet.

[kyrofa]

tl;dr this is an upstream Ubuntu bug, and a fix is being backported to Yakkety. This issue should be fixed soon.

After pounding my head on this for too long, I finally dropped a line to the snapcraft mailing list only to get literally an immediate response saying that this is actually LP: #1647031. I never would have figured that one out. I should have asked sooner :stuck_out_tongue: .

[lindhe]

Oh. Not too obvious, no. I don't feel like a dummy for missing that. Very well — at least there is a solution on the way now. That's nice to hear.

[kyrofa]

I don't feel like a dummy for missing that.

Oh heavens no. This stumped both of us. Odd issue!

[lindhe]

I'm not used to reading LP pages. Is there an ETA for my system?

[lindhe]

Is there any other fix, if I don't want to wait for months and months?

[kyrofa]

Is there any other fix, if I don't want to wait for months and months?

You can use a nameserver other than systemd-resolved (e.g. see step 5 on the mailing list post to which I referred earlier).

[lindhe]

For reference for other people: sudo systemctl stop systemd-resolved.service solved it for me, so sudo systemctl disable systemd-resolved.service to make it permanent.

The reason I asked was because I didn't know if I needed to do some extra rituals to make it work in snap.

[kyrofa]

@lindhe thanks for sharing the solution in here.

[lindhe]

The fix is apparently released now. If you test it and find it in working order, feel free to close this issue. I don't have time to test atm.

[kyrofa]

Confirmed: after getting my 16.10 server completely up-to-date, this problem appears to have gone away.