Nixpkgs: Can't send mail from php-fpm via sendmail

[florianjacob]

Issue description

I'm trying to send mail by using php's mail function, executed by php-fpm, sendmail provided by postfix. I get an Assertion!(st.st_mode & S_ISGID) || (st.st_gid == getegid())' failed.` error. This does not happen when I use sendmail from the command line.

Might be related to the way php-fpm is spawning unpriviledged child processes from its root master process? UPDATE: It does so by using the setuid() and setgid() syscalls after the child process is forked.

It seems this is the assertion from https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/security/wrappers/wrapper.c#L204 , but I don't understand what this assertion actually checks for. I'm at a dead end here and don't know how to track this down further.

@ixmatus, you seem to be the author of the file, maybe you can tell me something about why those assertions are there and what case they catch? :smile:

exact journald entry:

php-fpm[16885]: [WARNING] [pool piwik] child 16887 said into stderr: "sendmail: /nix/store/3xsjm8rfpy0ysfjs1pcybj33fsigszgp-wrapper.c:204: main: Assertion `!(st.st_mode & S_ISGID) || (st.st_gid == getegid())' failed."

Steps to reproduce

Minimal php script, add a php script like this to a webroot and let it execute by php-fpm:

<?php
    $receiver = "[email protected]";
    $subject = "Test";
    $text = "Works!";
    if(mail($receiver, $subject, $text)) {
        echo 'Mail sent!';
    } else {
        echo 'Error. :(';
    }
?>

Technical details

• System: 17.09.git.258b4c1 (Hummingbird)
• Nix version: nix-env (Nix) 1.12pre5413_b4b1f452
• Nixpkgs version: 17.03.1051.090ffd5fab
• Sandboxing enabled: false

[ixmatus]

Hi @florianjacob. I was able to reproduce your issue and not just with sendmail but with any wrapper program.

Those assertions and other logic I preserved from the previous setuid-wrapper.c code. I tweaked it a bit in places while working on the rest of the code to add setcap wrapper support but, to be honest, did not deeply analyze the negative case for those two assertions.

In order to test my understanding of how one would execute a wrapper program with a different effective uid or gid, I created a separate wrapper program with a lot of debugging information then wrote a small C program that would perform similar actions to php-fpm (fork, setgid, setgroups, setuid, popen).

This small program (run as root, as if I were the privileged php-fpm process) behaved the way I would expect it to, it forks and then changes the gid, groups, and uid to something less privileged and executes a popen call. I tried it on my enhanced debugging wrapper program and on the sendmail generated by nixos. In both cases the effective gid was set to the gid of the file.

I then setup nginx and php-fpm to reproduce using your script. I not only reproduced your issue but PHP's exec function doesn't work for _any_ setuid or setgid program. I also tried to call my enhanced debugging wrapper program and I was able to see that the effective gid and effective uid of the program _were not_ being changed to that of the program. I also tested by setting the sendmail_path to the sendmail wrapper program and another wrapper program, these cases failed similarly.

I do not know what is causing this behavior. We need to figure out why setuid and setgid programs called using PHP's exec or PHP's use of C's popen are not setting the uid or gid we would expect.

The assertions, after reading a lot about uids, gids, euids, egids, ruids, and rgids (there's still more to read, too) are guarding the program from being executed with an effective uid or gid that does not match with the ones set on the program's own file. I have not been able to write a program that was able to exercise this as every time I tried to set the effective uid or gid before exec'ing a wrapped program, its uid or gid was correctly changed to the one set on the filesystem. But, for some reason, php-fpm is exercising it.

I don't know enough about what is causing this odd behavior between php-fpm and our wrappers to know if these two assertions are the most optimal way to check for this negative case. Until we understand more, I don't think this logic should change.

[florianjacob]

Thank you for that in-depth look at the problem! 😃

Could you post your C program somewhere so I can take a look? Maybe I can spot a difference to how php-fpm + php's mail handle things.

(For anybody else who wants to check the php side: php's mail function is defined as https://github.com/php/php-src/blob/master/ext/standard/mail.c , php-fpm's setgid call is at https://github.com/php/php-src/blob/master/sapi/fpm/fpm/fpm_unix.c#L382 )

So the assertion fails because for some reason the setgid bit has no effect and the group is still the php-fpm group instead of e.g. postdrop, is that right? I really wonder how that could happen…
The comment above mentions this could happen when „being executed… as some other setuid program“, but I'm not sure what's meant with that, and how that could apply to this case…
Do you know the author of the original setuid-wrapper.c?

[ixmatus]

@florianjacob git blame says @edolstra was the original author - there was an SVN import sometime after the original code was written so I'm not sure if that preserved history accurately but I think it might have: https://github.com/NixOS/nixpkgs/blame/release-16.09/nixos/modules/security/setuid-wrapper.c

Yes, I can share the simple C program - I will write up a bit more (may not be today) around how to use and repro steps; I think it may be worth talking to the php-fpm maintainers also and see if this behavior rings a bell for anyone that may understand that code better.

The assertion is enforcing the invariant that the effective uid and gid of the process match the uid and gid of the wrapper program's file (iff the SUID or SGID bit are set). This invariant is being violated because the SGID bit is being ignored (I think but am not positive). I'm still studying the semantics of the suid / sgid system in Linux because I just don't understand enough to simulate non-trivial cases in my head; I'm using these as my resources:

1. https://people.eecs.berkeley.edu/~daw/papers/setuid-usenix02.pdf
2. http://nob.cs.ucdavis.edu/bishop/secprog/

In any event, the assertion checks are behaving as intended and understanding why this behavior is happening with php-fpm (or perhaps, it's php-fpm's specific usage combined with the semantics of SGID) will help us understand what is going on. Generally speaking, since a basic C program with no execve calls with the SGID bit set and configured as the sendmail program in the php.ini was failing to setgid, I am inclined to think this is not specific to our wrappers.

To unblock you, I'd recommend configuring SMTP for PHP instead of using sendmail. It is much better to use programs that don't have SUID and SGID bits set on them in your webapps anyway to reduce your exposure to mistakes in a program with those abilities that can be exploited.

@grahamc has pretty good experience with security too so hopefully he can weigh in on this.

[ixmatus]

I did find a bug in the wrappers/default.nix code as a result of investigating this too but I don't think it has any bearing on your issue here.

[florianjacob]

To unblock you, I'd recommend configuring SMTP for PHP instead of using sendmail.

Seems like you can't configure PHP's mail function to use SMTP, it can only use sendmail. But it turned out the web app I'm running (piwik, #26073) actually has an SMTP implementation that can be enabled as an alternative to the standard mail. 👍

I will write up a bit more (may not be today) around how to use and repro steps

No problem, my NixOS-based server is still a testing machine to eventually replace my current server, so it's not urgent for me to get this fixed in the short run. 😉

[florianjacob]

@fpletz , you added the sendmail path configuration to php-pfm in 323d0fdd5a6e1a47ab9da72d15ed9872625fe8a0 , can you confirm sendmail worked back then? Maybe something has changed with a new php release.

[fpletz]

Interesting, it was definitely working back then. I'll look into it. Looks like systemd prevents the setcap wrapper from working.

[ixmatus]

Oh interesting, I hadn't thought about that (I was so focused on php-fpm), some random internet person said they were able to call a setuid program successfully from a systemd service by setting the NoNewPrivileges=false directive. I'll test that when I get home tonight.

That doesn't sound very safe though. Perhaps that is only turned on for php-fpm's systemd service if postfix is enabled and sendmail is configured for the php.ini (would require a new nixos option probably)?

[fpletz]

@ixmatus Just checked: We have postfix enabled via the NixOS module and are quite close to master. We can send email from php-fpm processes. So I can't reproduce this issue unfortunately.

[ixmatus]

@fpletz that is odd. It was very easy for me to reproduce. When I have a minute I will try to share some minimal Nix code that was reproducing @florianjacob's experience. I am on 17.03.

[ixmatus]

I think this should be reopened as my bug fix in the related PR has no bearing on the behavior described in this issue.

[vcunat]

Oh, I misunderstood you.

[florianjacob]

(@ixmatus, not to much of our surprise, I can confirm now #26657 does not fix this.)

[fpletz]

I have found the issue: We were using networking.defaultMailServer which turns sendmail into a symlink to ssmtp which doesn't need root rights. I have removed NoNewPrivileges for now.

[fpletz]

And backported to 17.03.

[ixmatus]

Nice find and thanks for circling back on this.

[florianjacob]

@fpletz Yay, thanks alot for tracking this down. 😄

@fpletz @ixmatus Can you confirm sending mail from php-fpm now works in your setups?

With my setup (IPv6 enabled), after updating to the current master, I still can't send mail from piwik or from my test script, now with a different error:

php-fpm[26577]: [WARNING] [pool piwik] child 26596 said into stderr: "sendmail: fatal: inet_addr_local[getifaddrs]: getifaddrs: Address family not supported by protocol"
postfix/sendmail[7962]: fatal: inet_addr_local[getifaddrs]: getifaddrs: Address family not supported by protocol

Could this be some other capability problem? sendmail still works just fine when used on the command line or by a python daemon.

[ixmatus]

@florianjacob I'd have to resurrect my repro setup which I don't have time to do ATM. Sorry. IPv6 enabled and "Address family not supported" are suspicious. Try disabling IPv6 and see if the problem goes away, that will help narrow down what is going on.

[fpletz]

Weird, we do have the following the phpfpm service config:

RestrictAddressFamilies = "AF_UNIX AF_INET AF_INET6";

That should actually allow IPv6 sockets. Seems we need to remove that too? :confused:

I'll try to come up with a testcase and investigate further.

[fpletz]

Now tested and fixed. ;)

[florianjacob]

Thanks alot, works for me now as well! 🎉