Nextcloud-snap: Unable to renew Letsencrypt certificate via SNAP ([Errno 1] Operation not permitted)

[rjsears]

I am running snap 2.38.1 on Ubuntu 18.04 LTS.

I installed nextcloud and set it up with letsencrypt and everything is running fine until it came time to renew my certificate which snap tried to do automatically (now multiple times). There seems to be some kind of OS or filesystem error and I am getting the following error in the log files each time it attempts to renew, except that it increases the privkey number (ex: privkey1.pem, privkey2.pem, etc) but the error is the same. THe LEtsEncrypt forum is saying that it is a problem with SNAP so I am looking for some help:

2019-05-15 16:19:18,454:DEBUG:acme.client:Storing nonce: LrlnuRVT0-aPA-PusulYmOvNuqdr7G8OxaAewYTF8do

2019-05-15 16:19:18,456:DEBUG:certbot.storage:Writing new private key to /var/snap/nextcloud/current/certs/certbot/config/archive/nextcloud.bob.aero/privkey4.pem.

2019-05-15 16:19:18,457:WARNING:certbot.renewal:Attempting to renew cert (nextcloud.bob.aero) from /var/snap/nextcloud/current/certs/certbot/config/renewal/nextcloud.bob.aero.conf produced an unexpected error: [Errno 1] Operation not permitted: '/var/snap/nextcloud/current/certs/certbot/config/archive/nextcloud.bob.aero/privkey4.pem'. Skipping.

2019-05-15 16:19:18,458:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/snap/nextcloud/13144/lib/python2.7/site-packages/certbot/renewal.py", line 450, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/snap/nextcloud/13144/lib/python2.7/site-packages/certbot/main.py", line 1192, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/snap/nextcloud/13144/lib/python2.7/site-packages/certbot/main.py", line 115, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/snap/nextcloud/13144/lib/python2.7/site-packages/certbot/renewal.py", line 315, in renew_cert
lineage.save_successor(prior_version, new_cert, new_key.pem, new_chain, config)
File "/snap/nextcloud/13144/lib/python2.7/site-packages/certbot/storage.py", line 1111, in save_successor
os.chown(target["privkey"], -1, os.stat(old_privkey).st_gid)
OSError: [Errno 1] Operation not permitted: '/var/snap/nextcloud/current/certs/certbot/config/archive/nextcloud.bob.aero/privkey4.pem'

[bit-space-42]

I'm having the same issue with the same output in my log.

[the-sane]

I'm also experiencing this issue. Let's Encrypt just notified me that my certificate is about to expire and I realized the auto-renew hasn't been able to complete. Same error and traceback in my logs.

Is it failing on the chown operation? I noticed that privkey1.pem has the permissions 644 while the newly created privkey2,3,etc.pem have permissions 600. All are owned by root, so it shouldn't matter, but manually changing the permissions of the newly created privkeys has no effect anyway since the next renewal attempt simply creates a new one.

Edit w/ version info:
Also Ubuntu 18.04
Nextcloud snap v15.0.7

[bit-space-42]

Ok, so I found a work-around, but that doesn't solve this bug.

If you want to get something working until this is resolved, you can start by cleaning out all your old certificates and keys (or backing them up using mv instead):

sudo rm -rf /var/snap/nextcloud/current/certs

And then running something like:

sudo nextcloud.enable-https lets-encrypt

It doesn't solve the bug but at least you get a working certificate again for now!

[kyrofa]

Hang tight folks, I can duplicate this (it's due to the certbot update we shipped in the last release) and have the fix, just need to clean it up. I'll ship it ASAP.

[kyrofa]

This has been released and is automatically rolling to you now. Run snap refresh if you want to do it sooner. Thanks for reporting this so quickly.

[the-sane]

Thank you very much for taking care of this so quickly!

Upon refreshing and updating to 15.0.8, the renewal ran again but this time it failed with a parsefail for me.

Renewal configuration file /var/snap/nextcloud/current/certs/certbot/config/renewal/---.---.---.conf (cert: ---.---.---) produced an unexpected error: 'Namespace' object has no attribute 'nextcloud:webroot_path'. Skipping.
nextcloud.renew-certs[6803]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
nextcloud.renew-certs[6803]: No renewals were attempted.
nextcloud.renew-certs[6803]: No hooks were run.
nextcloud.renew-certs[6803]: Additionally, the following renewal configurations were invalid:
nextcloud.renew-certs[6803]:   /var/snap/nextcloud/current/certs/certbot/config/renewal/---.---.---.conf (parsefail)
nextcloud.renew-certs[6803]: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
nextcloud.renew-certs[6803]: 0 renew failure(s), 1 parse failure(s)

Examining the renewal .conf file, I notice that it ends with this line, which appears to be what the above error is complaining about:
nextcloud:webroot_path = /var/snap/nextcloud/current/certs/certbot

I don't see anything else unusual in the config file. Everything seems to be in order and I've never manually touched this file. Here's the rest of the config file:

# renew_before_expiry = 30 days
version = 0.28.0
archive_dir = /var/snap/nextcloud/current/certs/certbot/config/archive/---.---.---
cert = /var/snap/nextcloud/current/certs/certbot/config/live/---.---.---/cert.pem
privkey = /var/snap/nextcloud/current/certs/certbot/config/live/---.---.---/privkey.pem
chain = /var/snap/nextcloud/current/certs/certbot/config/live/---.---.---/chain.pem
fullchain = /var/snap/nextcloud/current/certs/certbot/config/live/---.---.---/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = ---
work_dir = /var/snap/nextcloud/current/certs/certbot/work
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = nextcloud:webroot
logs_dir = /var/snap/nextcloud/current/certs/certbot/logs
rsa_key_size = 4096
config_dir = /var/snap/nextcloud/current/certs/certbot/config
nextcloud:webroot_path = /var/snap/nextcloud/current/certs/certbot

[kyrofa]

Agh darnit, I thought I could fix that before anyone noticed :wink: . It's #1012, and the fixed release is 15.0.8snap2, which is already rolling out. You're right, it's not a file you should need to touch, just refresh and things will be peachy again.

[the-sane]

Fix confirmed, everything is indeed peachy again. Thank you for the fix and for your work on this project. We appreciate it!

So the custom webroot plugin is being phased out. Is there anything I might want to change to get a head start on that and begin using the newer defaults?

[kyrofa]

My pleasure! I use it too, which is how I discovered #1012 once I updated my production instance :man_facepalming: .

You really don't need to do anything about the plugin being phased out. It's exactly certbot's plugin minus a chown that fails under confinement. Certbot updated theirs to catch exceptions from the chown, which means the custom one simply isn't required anymore. Removing it outright was a pretty naive call on my part, I totally forgot those things got saved into the config file, and my testing didn't cover that edge case (it will now, I tell ya). I doubt I'll ever remove it now unless certbot changes API enough I need to, in which case I'll automatically rewrite those configs. You should never have to know they exist.

If you really want to stop using the custom one though, you're welcome to. Just remove nextcloud: from the config file above (e.g. turn nextcloud:web_root into simply web_root). You'll void your warranty, though!

Just kidding.

[the-sane]

Oh no, my warranty! :laughing:

Thanks for the details.

[gjgd]

@kyrofa I'm encoutering the same error on Nextcloud 16, but can't rollback to 15.0.8snap2 because Nextcloud doesn't allow downgrades. Is it possible to push the certbot fix on 16/candidate or 16/edge?

[kyrofa]

Of course my friend. It should already be on 16/edge, but I'm building one for 16/candidate now (16.0.1snap1-rc2). You'll get it soon. Other than that, how is 16 behaving so far?

[gjgd]

Awesome, my cerbot works now, thanks! 🎉
I skipped 15 and upgraded straight to 16 so can't really compare but it's working well for me, didn't have any problems

[kyrofa]

Perfect, glad to hear it. As soon as the core apps support it fully we'll roll it out for everyone.

[rpacheco2019]

@kyrofa , i really aprecciate your time and effort. I followed the steps (upgrade to snap 15.0.8snap3 from v14) and run the command sudo nextcloud.enable-https lets-encrypt and everything is working fine now. Be aware that Letsencrypt only permit 5 certs creation per week with the same domain or sub-domain, (if you fail 5 times you need to wait some days, also in my case this happened to me because the cert was created but not replaced). Again @kyrofa , thanks you for all your help!

[iaandr]

Hi All
I have the same trouble with snap 2.39.3 at Ubuntu 18.04 with install cert to NextCloud 14.0.10 and i recieve error at the command sudo nextcloud.enable-https lets-encrypt:
OSError: [Errno 1] Operation not permitted: '/var/snap/nextcloud/current/certs/certbot/config/archive/cloud.ferro-estate.ru/privkey27.pem'

Could you write me what should I do to install cert onto NextCloud step by step?
I would be very grateful for the detailed information.

Best regards,
Andrey