The private keys in /etc/letsencrypt/archive/_domain_/privkey
I guess it's better to have only root to be able to read the private key, hence set them to 0600.
Hrm. What permissions do you see on the directory /etc/letsencrypt/archive
? Can you actually read a privkey as an unprivileged user?
Probably the keys should be 0640 and set the group that runs the webserver, if it isn't running as root? And the directory should be 0750, assuming such a group exists?
This is going to be hard to get right from letsencrypt-auto
. But we should have control of these variables concisely exported for OS packagers. @hlieberman @fmarier.
This is part of #420.
Hm, you're right.
/etc/letsencrypt/archive is set to 700, so a regular user shouldn't be able to reach the keys...
One point made on community.letsencrypt.org is that users may move the certs or keys out of /etc/letsencrypt. Of course they should be symlinking in rather than moving / copying keys out, but it will happen. So bumping this ticket up to nice for 1.0.
Actually, the point about not moving keys makes me think that in fact the permissions on everything in the directory should be 0440, not 0640.
A related matter would be to honor the group permissions set by the directory the certs reside in. I use a "ssl" group that's the only one allowed to touch the certs aside root, for example.
Since ~2006, Debian-based distros come along with a system group ssl-cert
which is designed for use with private key material (according to changelog).
For symlinks from /etc/ssl/..., something like this would be nice:
-rw-r--r-- 1 root root 2.1K Jan 17 17:54 cert1.pem
-rw-r--r-- 1 root root 1.1K Jan 17 17:54 chain1.pem
-rw-r--r-- 1 root root 3.2K Jan 17 17:54 fullchain1.pem
-rw-r----- 1 root ssl-cert 3.2K Jan 17 17:54 privkey1.pem
These settings should be fine for e.g. Debian 8 / Apache and many other applications.
I guess that users have different requirements (e.g. according to distro), so I think the permissions should be configurable via cli.ini
and maybe via parameters (overwriting cli.ini
settings).
I also want to note that wrong permissions (in the sense of user's requirements) cannot necessarily be healed by a later user interaction: Once the private key is readable after being written to disk with weak permissions, it could theoretically be stolen if the admin user changes permissions too late. Therefore, an _emtpy_ file with correct file permissions must be written to disk _before_ key material is written into the file.
I suggest to introduce three config values for cli.ini
(including example settings):
private-key-owner = root
private-key-group = ssl-cert
private-key-permisson = 640
Optionally, this could also be done for public key material (just for completeness).
I'm not sure for other distros than Debian, if there is a ssl-cert
group (or equivalent). But assumed, such a group is widely used, this could be a nice default for LE (when cli.ini
is unmodified or unused).
I guess the Debian package maintainers might want to go for ssl-cert
as a default anyways.
References:
Courier Mailserver is complaining when LE's public certificates are symlinked from /etc/ssl/certs
:
courieresmtpd: STARTTLS failed: couriertls: /etc/ssl/certs/xxxxxxxx.0: Permission denied
Result: Courier only receives mail from _some_ mailservers (I assume servers not supporting encryption). This even happens while I don't use LE certs for my mail server host name (yet). Courier has its own certificates stored in /etc/courier
, but it uses public certificates stored in /etc/ssl/certs
.
The reason for that is that /etc/letsencrypt/live
and /etc/letsencrypt/archive
are only readable to the root
user, which is a good idea as long as private key material is not protected by itself.
Current state:
drwx------ 7 root root 4.0K Jan 19 17:40 live
drwxr-xr-x 2 root root 4.0K Jan 19 14:30 live/www.example.com
lrwxrwxrwx 2 root root 4.0K Jan 19 14:30 live/www.example.com/privkey.pem (symlink)
lrwxrwxrwx 2 root root 4.0K Jan 19 14:30 live/www.example.com/fullchain.pem (symlink)
drwx------ 7 root root 4.0K Jan 19 17:40 archive (currently protects private certificates)
drwxr-xr-x 2 root root 4.0K Jan 19 14:30 archive/www.example.com
-rw-r--r-- 2 root root 4.0K Jan 19 14:30 archive/www.example.com/privkey3.pem
-rw-r--r-- 2 root root 4.0K Jan 19 14:30 archive/www.example.com/fullchain3.pem
To mitigate such issues and still be able to symlink the latest public certificate, the directories /etc/letsencrypt/live
and /etc/letsencrypt/live
should be world readable _AND_ private certificates should be protected separately (in a configurable manner).
Should be:
drwxr-xr-x 7 root root 4.0K Jan 19 17:40 live
drwxr-xr-x 2 root root 4.0K Jan 19 14:30 live/www.example.com
lrwxrwxrwx 2 root root 4.0K Jan 19 14:30 live/www.example.com/privkey.pem (symlink)
lrwxrwxrwx 2 root root 4.0K Jan 19 14:30 live/www.example.com/fullchain.pem (symlink)
drwxr-xr-x 7 root root 4.0K Jan 19 17:40 archive
drwxr-xr-x 2 root root 4.0K Jan 19 14:30 archive/www.example.com
-rw-r----- 2 root root 4.0K Jan 19 14:30 archive/www.example.com/privkey3.pem (group configurable per certificate directory)
-rw-r--r-- 2 root root 4.0K Jan 19 14:30 archive/www.example.com/fullchain3.pem
With these permissions, symlinks to LE's public certificates would be world readable (which is fine for public certificates) while private certificates would be readable to only root and a configurable group, e.g.
-rw-r----- 2 root root 4.0K Jan 19 14:30 archive/www.example1.com/privkey3.pem
-rw-r----- 2 root aabb 4.0K Jan 19 14:30 archive/www.example2.com/privkey3.pem
-rw-r----- 2 root ssl-cert 4.0K Jan 19 14:30 archive/www.example3.com/privkey3.pem
@hlieberman has agreed to try to look at the set of proposed PRs for this, pick the most promising, and shepherd it towards landability.
That may allow Debian to packaged with a letsencrypt gid option!
To add a bit more here: I agree that the good practice for using those files elsewhere on the server (like, webserver, MTA, MDA…) is symlinking against files in /etc/letsencrypt/live.
However, my MTA at least complains about default file permissions (on the key file):
insecure permissions: must be at most rwxr-----
And then refuse to start.
So, on each cert renewal, I change the file permission to 400 (but could live with 640 or 440 to).
So, I must agree that this should really be considered. ;)
kick
It's a pity to see this moving from milestone to milestone without visible process! What's holding it back?
This sounds good to me - I'd love to be able to share the certificates amongst a few users with an ssl
group or something.
@joostrijneveld the core development team is small and there is a lot of important work to do; this hasn't been at the top of the list because the directory's permissions protect the keys by default. We've been hoping a volunteer would show up to help with this, though it's actually a fairly subtle task to get this right without creating regressions, and the solution should probably address #2964 at the same time.
I use manuale (pip install manuale
) which is a tiny and non-automatic client which sets acceptable file permissions.
To help volunteers who want to work on this, we should probably do some design to figure out how we want this to work. Here's a first draft proposal:
--uid
flag to set the user id for all of /etc/letsencrypt/
--gid
flag to set the group id for all of /etc/letsencrypt/
Currently we have:
Directories which are world-readable (755):
/etc/letsencrypt
/etc/letsencrypt/renewal
/etc/letsencrypt/csr
Directories which are readable only by root (700):
live/
, archive/
, keys/
and accounts/
subdirectoriesAlmost all files are world-readable (644) but protected by the permissions of their directories, except:
keys/
directory has archived keys that are user-readable (400).It will be messy to make config options for all of these different modes. The easiest way to preserve full compatibility would be to mask over the top of whatever the user supplies, but we could also have "old permissions" which are the default, and switch to something totally user-specified if the user provides a --permissions
flag. Or we could take the risk of assuming that nothing outside of live/
and archive/
is really crucial from a backwards compatibility perspective.
I'd really like the ability to specify different permissions for the key material and certificates. The web server I use (H2O) reads keys as root, but performs OCSP updating as a low trust user, and with the current setup it can't access the certificates.
The assumption that one "should be symlinking in rather than moving / copying keys" is a particularly dangerous one. Not only have some pointed out that certain applications have problems with the symlinks, but also consider:
Given that management of a single instance of certbot and associated scripts appears to be easier than one in each of a dozen or more jails (and probably easier on the ACME servers), I'm of the opinion that one should _expect_ keys and certs to be _copied_ in many situations.
There are additional advantages to copying, as chattr/chflags can then be used _selectively_ to additionally lock down the keys somewhat in Linux, and much more firmly in FreeBSD.
As such, stronger, as-generated permissions on _at least_ the keys would be greatly appreciated.
Personally, I'd go with 400 on the keys and 444 on the certs, if only to prevent accidental modification or erasure. I'd be happy with 600 and 644 as defaults.
It's been mentioned about the problems with some MTAs, but I would explicitly add here.
Sendmail+tls on FreeBSD complains about the private key being group readable and refuses to use them:
Feb 11 23:47:24
And I agree with @jeffsf 's thoughtful arguments suggestions.
Here I suffered from the same problem that due to incorrect private key permissions my sendmail STARTTLS support suddenly stopped working after renewal of my certificates. Here I have now adapted my daily cronjob to do the following:
MAILTO="root"
0 */12 * * * root /usr/bin/certbot renew --text >>/var/log/certbot.log 2>&1 || (find /etc/letsencrypt/archive -type f -name "privkey*.pem" -exec chmod 600 {} \; ; /etc/init.d/nginx reload; /etc/init.d/dovecot reload; /etc/init.d/sendmail restart)
Thus, I will enforce proper 600 permissions after successful execution of certbot. This temporary solved my issues until certbot itself supports to set file permissions correctly.
The workaround I am using is post_hook
option for certbot. I think it is much more elegant.
In etc/letsencrypt/renewal/DOMAIN.conf, it looks as follows:
post_hook = chmod 400 /usr/local/etc/letsencrypt/live/DOMAIN/privkey.pem; service apache24 reload; services dovecot reload; service sendmail reload
From the command line it would be
--post_hook "chmod 400 /usr/local/etc/letsencrypt/live/DOMAIN/privkey.pem; service apache24 reload; service dovecot reload; service sendmail reload"
This works (at least on FreeBSD) since chmod
without any flags acts on the actual file to which the symlink is pointing. I am not sure if this is the same default for chmod
on all systems. (You might need to adjust the path for your OS and you'd have your own "restart" commands)
Disclaimer: I am still waiting for my actual renewal to happen automatically from the cron job.
Using a post_hook
is a good idea, but FWIR it does not work if you use certbot through a dedicated (e.g. certbot) linux user (because no way I’m giving it root rights). But I handle all those things (chmod, restarting whatever program use those certs…) in my systemd service file, which provide such kinds of hooks too.
I haven't tried this with a dedicated user, but I don't see the reason why it shouldn't work. Since the directory permissions allow that user to Write the files (certs), the same user can change the permissions too.
So, that part can be placed in post_hook
.
Restarting your daemons is a different issue; there you'd need your elevated permissions. Obviously, since you have to run something else from your cron it probably doesn't make a big difference.
Also, - just in case you missed it: you can avoid the extra script (find, etc.) if you run chmod
on the soft-links (which remain the same upon renewal).
@St-Ranger: I use chmod directly too. ;) You’re right about changing permissions, as a matter of fact my issue is with chown
, because at least opensmtpd (if I remember correctly) complains if the keys are not owned by root. And apparently you cannot chown a file to another user (or at least not to root), for security reasons I guess. So like you said, I still need to run things outside of the hook.
@ArchangeGabriel: The fact that you have to do chown
from outside of certbot is not certbot's fault (or even relevant to certbot per se). Rather, it is an obvious consequence of your conscious choice of not running certbot as root. This is different for many other users who run certbot as root (which, as far as I understand, is an intended primary _m.o._).
A quick digression: Most *nix systems do not allow ownership giveaway "for obvious security reasons"
So, a user can chown
only from and to the user (and the group) whose effective permissions you have. Thus, a non-root user can chown
only to a different group that he also belongs to.
AFAIR, all BSD's are like that, but some System V OSes, e.g. IRIX, did allow ownership giveaway and that was a problem.
@ArchangeGabriel: The fact that you have to do
chown
from outside of
certbot is not certbot's fault (or even relevant to certbot per se).
Rather, it is an obvious consequence of your conscious choice of not
running certbot as root. This is different for many other users who run
certbot as root (which, as far as I understand, is an intended primary
_m.o._).
You’re absolutely right, and I never implied this was a certbot issue. I know that whatever happens, I’ll still need chown and restarts from within my systemd service.
A quick digression: Most *nix systems do not allow ownership giveaway
"for obvious security
reasons"
So, a user canchown
only from and to the user (and the group) whose
effective permissions you have. Thus, a non-root user canchown
only
to a different group that he also belongs to.
AFAIR, all BSD's are like that, but some System V OSes, e.g. IRIX, did
allow ownership giveaway and that was a problem.
Thank you for this reference.
Hi @pde
Just trying to get my head around the logic of not setting the file to 400, is it that one wants people to be able to dink with directory permissions in order to determine access to keys? After all, in the default configuration, the grandparent is set to 700, owned by root, and so only root can access the file one way or another. At that point it's simply a minor nuisance to have to exit_hook the thing, but a nuisance never-the-less.
I ask because what you're proposing seems to me to suggest too many configuration options, with which to hang oneself and complicate your code. Better it would seem to WONTFIX this and point people with sendmail and such at a simple workaround.
Thoughts?
Nearly 2 years have gone by and this simple thing is still unfixed?
After doing updates or renewals, I just run this script from /etc/letsencrypt
/etc/letsencrypt/setprivkeyperms.sh
~~~~
find /etc/letsencrypt/archive -type f -perm 0644 -name 'privkey*.pem' -exec chmod 600 {} +
~~~~
Once done, I just rsync the dirs to other servers in the cluster.
Sendmail and apache are happy.
I re-discovered this issue when investigating an intrusion on a Debian server, as I looked for world-readable files that may have been accessible by the intruder.
Private keys and certs generated by certbot are world-readable.
Upon generation and renewal, live keys are immediately copied into the archive directory. Both key files and the archive directory itself are world readable, not 700 contrary to what @ZaiLynch claimed.
Edit + update:
I made a new clean install of cerbot 0.10.2 on Debian 9 and directories archive, accounts, keys, live all have right 700, so the keys are not world-readable with a clean install.
It may have been a human error with the previous installation, I have fixed directory rights manually and will write a script to monitor files rights and ensure they stay non-world-readable.
I discovered this problem today after wasting quite a bit of time and effort trying to figure out why my email clients stopped being able to send email. Apparently it was due to automated certbot renewal which made the key world readable, which caused sendmail to refuse to use it, which broke mail for everyone because the server is specifically restricted (as it should be) from allowing clients to authenticate on a non-secured channel.
I wouldn't consider this a low priority problem just because it only causes problems for people using services other than http. Good SSL security is important for other services too. 2+ years and multiple reports (#3108 #3754 are dupes of this issue) for this to still be unresolved seems pretty crazy to me.
This is definitely still on our roadmap, and there'll be much more work on improving mailserver support in Certbot in the coming months. In the mean time, you can use the --deploy-hook
option to run a script at renewal time to make sendmail happy.
(Having your hook script cp
the relevant files and then chmod
them, and then configuring sendmail
to use the copied versions, is probably the safest approach here).
@pde Can you clarify why running chmod
directly against the privkey might be considered unsafe? Your earlier comment here suggests that users should not be copying the keys out of their original location.
It does seem like there are scenarios where copying a world-readable private key file into a world-readable directory even temporarily could lead to a loss of access control, e.g. if a snapshot were triggered.
This issue also makes trouble for samba - our domain controller uses certbot for certificates, and the 644
permissions cause samba to refuse starting as the privkey file is too visible. Found the problem and managed to fix it by correcting the permissions, but that debugging should really not have been necessary...
I really don't see why the permissions aren't just set properly in the first place (even though the directory is "safe" with 700
)?
It's a bit strange that the code which originally generates the key, in https://github.com/certbot/certbot/blob/master/certbot/crypto_util.py#L65, already does the right thing (it uses open mode 0600, which is perfectly appropriate for private keys), and there is even a safe_open
function which supports such modes, but it is not used when the keys are actually saved to the archive directory...
In my opinion, 0600 is the right default mode for any private key material, and you should really use that. If e.g. Debian or another distro wants to introduce separate 'ssl-cert' groups with additional permissions, then they can patch their copy of certbot while packaging. I don't think it is up to upstream to cope with all the distro-specific variations.
The following diff should make >95% of people happy (and I'm applying it locally anyway):
diff --git a/certbot/storage.py b/certbot/storage.py
index 32d6771c..88b804f4 100644
--- a/certbot/storage.py
+++ b/certbot/storage.py
@@ -1123,7 +1123,7 @@ class RenewableCert(object):
logger.debug("Writing symlink to old private key, %s.", old_privkey)
os.symlink(old_privkey, target["privkey"])
else:
- with open(target["privkey"], "wb") as f:
+ with util.safe_open(target["privkey"], "wb", chmod=0o400) as f:
logger.debug("Writing new private key to %s.", target["privkey"])
f.write(new_privkey)
Did a review of all of the concerns and proposed solutions in this thread. Here's a summary, interspersed with some concerns and some steps forward for the short-term.
There's a couple different things folks on this thread are concerned with:
...and also other things, but these are the three main ones.
There are a couple of complete solutions that would be nice to knock out all of these at once, like this directory permissioning recommendation from @thomaszbz. However, if we do this, or any similar proposal, we run the risk of exposing private key material if Certbot downgrades.
Currently, it's a fairly common workflow to downgrade versions of Certbot (e.g. use certbot-auto
, but then switch to OS packages). Unfortunately, Certbot is careful to write privkeys as 644, BUT never ensures the live/
or archive/
directory remains 700. So, in the short-term, we shouldn't make live/
or archive/
more permissive without a more complex and well-thought-out solution, which might involve migrating the keys to a new directory that is properly permissioned. This would be good to talk about for 1.0!
For now, there's no reason we can't still knock out problem 1 in a simple and entirely safe way by dropping key permissions to 600 (basically @DimitryAndric's proposal), and punt this larger discussion to later.
I'd prefer some limited functionality like --key-group <gid>
in the short-term, especially since the primary use case is for a private key access group. This would make it clear that such a group should be limited (good for private key hygiene, and also good for the downgrade case), unlike a --gid
flag.
We're also talking about this more this week, so this post/thread might be updated with more developments in the very near future.
radicale needs +x to 'read' certificates (?); if certs are just +r, radicale.log yelds
WARNING: Error while reading SSL certificate '/etc/letsencrypt/live/*.me/cert.pem': [Errno 13] Permission denied: '/etc/letsencrypt/live/.me/cert.pem'
WARNING: Error while reading SSL key '/etc/letsencrypt/live/.me/privkey.pem': [Errno 13] Permission denied: '/etc/letsencrypt/live/*.me/privkey.pem'
g+rx on pem files fixes this error
If you need a custom permission set on your private key: Starting with the next release, if you alter the GID or group mode of /etc/letsencrypt/live/<domain>/privkey.pem
, Certbot will preserve that information in future renewals.
When you create a new certificate, the permission is set by default to 0600.
The primary issue (as indicated in the title) is fixed. Further discussion setting group permissions for private key material as a command-line option, an adjacent issue, should continue in this issue #2964.
If you need a custom permission set on your private key: Starting with the next release, if you alter the GID or group mode of
/etc/letsencrypt/live/<domain>/privkey.pem
, Certbot will preserve that information in future renewals.
That seems like a good solution for everyone. Just two questions:
/etc/letsencrypt/live/<domain>/privkey.pem
is a symlink on my systems so I can't set GID/permissions on that exact file. I assume certbot retains GID/group permission of the file referenced from the "live" directory?Edit: If I understand the commit correctly the change is behavior group-specific so permissions for "other" will NOT be retained.
It seems like the new version will always set the "world" permission bits to 0 (no permission). This might be a breaking change for some. Was this a deliberate decision?
Edit 2: Maybe my last question would be better placed in the pull request's comments (PR #6480)?
Thinking a bit more about this I came to the conclusion that this will be a breaking change for many users relying on the old behavior: Previously there was no way to retain file GID/permission on renewals.
So I think what many did was to change the GID of "archive"/"live" (+ subdirectories) so a non-root group could access the keys. The key itself was world-readable so no problem in that case.
Now if I understand the change correctly (and that might be a big IF) certbot will set the "world" permissions to 0. In that case the key is not accessible anymore in the scenario above.
To avoid breaking any setups I think certbot must also keep the "world" permissions on renewal.
Yes I confirm, private key world permissions will be set to 0, I have migrated the integration test and the assertion about one hour ago ^^
And the group permission also, by default, on new certificates. However, if afterward the group permission or the group owner is modified, it will be retained for any private key generated during the renewal.
So there will be a problem only for non-root processes outside of this group that relies on world readable permissions on private key to read it. Is it really a case? For what I know, most daemon processes are running as root, or fork themselves to an unprivileged user after setup as root.
And after, even if it is breaking change, it is, for that matter security, a sufficient reason to take the risk and remove the flaw in my opinion.
So there will be a problem only for non-root processes outside of this group that relies on world readable permissions on private key to read it. Is it really a case?
One notable example which comes to mind is Exim which reads the private key at runtime (under the "exim" user).
And after, even if it is breaking change, it is, for that matter security, a sufficient reason to take the risk and remove the flaw in my opinion.
I agree that the change in itself is good but the previous setup was not insecure by itself so I don't like breaking existing setups - especially because most users will have automated renewals so something will go wrong in the middle of the night and monitoring will wake up some some poor sysadmin who needs to fix this in a frenzy.
@FelixSchwarz You're right, that setup would break on renewal. I'll work on something to remedy this (the simple solution being to preserve the other-read bit, or maybe a more complicated solution) today.
EDIT: By "more complicated": maybe we can limit the scope of the change to people with group-readable live/archives. The idea here being to discourage people from relying on the world-readable bit in the future, while still making sure people's current setups don't break.
@sydneyli I like your "more complicated" solution, I am very uncomfortable with private key that are world readable ^^
But for the long term, we should quickly go to the separated directory for private keys approach, and fix definitively the issue.
Sydney and I talked about this a bit out of band. The reason why at least I am not too worried about copying over the other read permission is that the key permissions are irrelevant from a security perspective as long as the permissions on archive
and live
are unmodified.
I also think this is a clear improvement to the state of things. Currently, private keys are always made world readable. With the change Sydney is proposing, the default will be that new lineages are created with keys with 600 permissions and that will be preserved between renewals unless the user makes changes to the permissions on the keys. In that case, we'll preserve the group, group permissions, and other read bit.
I think we agree that correct long term solution requires changes to the directory structure of /etc/letsencrypt
though.
Indeed, I was thinking of this as a second wall in case something unexpected happens on the directory permissions level, but anyway, any root level process could just mess up all the security for a lot of reasons. So I probably overthinking ^^
That's true. Preserving the other permission removes a second layer of protection that could help users from shooting themselves in the foot. That layer doesn't exist in any released version of Certbot so there's no harm there, however, it existed in the initial proposal for the change here and now we're removing it.
I appreciate you thinking about the problem though! I think it'll be a bit tricky, but I'd be happy to chat or hear a proposal for how to really solve the permission problems with certs and keys in the future.
Hi, I read the long thread here already, but i am sorry can not figure out how to solve correctly, please somebody help to brief the step.
I install letsencrypt with certbot, on Linux CentOs 7 with apache webserver used for reverse proxying
certificate is generated but apache seem not able to read the required file, so the TLS handshake always fail (time out)
please help what the correct action (how) to solve this securely without exposing security file to the world?
Thank you
@bunhin, I'd encourage to make a post on https://community.letsencrypt.org where there is a large community of people familiar with the project that should be able to help with that.
@bmw thank you for the link, i will then try to find help there
Most helpful comment
Since ~2006, Debian-based distros come along with a system group
ssl-cert
which is designed for use with private key material (according to changelog).For symlinks from /etc/ssl/..., something like this would be nice:
These settings should be fine for e.g. Debian 8 / Apache and many other applications.
I guess that users have different requirements (e.g. according to distro), so I think the permissions should be configurable via
cli.ini
and maybe via parameters (overwritingcli.ini
settings).I also want to note that wrong permissions (in the sense of user's requirements) cannot necessarily be healed by a later user interaction: Once the private key is readable after being written to disk with weak permissions, it could theoretically be stolen if the admin user changes permissions too late. Therefore, an _emtpy_ file with correct file permissions must be written to disk _before_ key material is written into the file.
I suggest to introduce three config values for
cli.ini
(including example settings):Optionally, this could also be done for public key material (just for completeness).
I'm not sure for other distros than Debian, if there is a
ssl-cert
group (or equivalent). But assumed, such a group is widely used, this could be a nice default for LE (whencli.ini
is unmodified or unused).I guess the Debian package maintainers might want to go for
ssl-cert
as a default anyways.References: