Phpmailer: Obfuscation of passwords from debugOutput

[pgorod]

This is half a request for information, half a suggestion for a feature (if it is deemed appropriate).

My Issue

I am working on improving SMTP handling in the world's most used free and open-source CRM, SuiteCRM. It uses phpMailer (thank you!). We're sensing a shift as many SMTP providers start to get more restrictive.

So I need to handle not just a specific case, but make the product robust enough for all sorts of SMTP situations.

I am adding ways for users to get more complete SMTP debug logs, so they know why their emails aren't sending. I want to put debug logs on screen, and possibly in logs, but I don't want clear-text password included. People will be posting them online when asking for help troubleshooting.

I will count BASE64 encoded password as clear-text, for the purposes of this issue, since anybody can decode them (encoding is not encryption, after all).

So my main concern is SMTP plain authorization, segments like these:

CLIENT -> SERVER: AUTH LOGIN
SMTP -> get_lines(): $data is ""
SMTP -> get_lines(): $str is "334 VXNlcm5hbWU6"
SERVER -> CLIENT: 334 VXNlcm5hbWU6
CLIENT -> SERVER: bWVAc2l0ZS5jb20=
SMTP -> get_lines(): $data is ""
SMTP -> get_lines(): $str is "334 UGFzc3dvcmQ6"
SERVER -> CLIENT: 334 UGFzc3dvcmQ6
CLIENT -> SERVER: MTIzNDU2
SMTP -> get_lines(): $data is ""
SMTP -> get_lines(): $str is "235 2.7.0 Accepted"

Some questions

1. Do I have to worry about passwords in SSL and TLS transcripts? Or are these well encrypted?

2. What is a precise way to locate and remove the usernames and passwords in these logs? Removing the line after SERVER -> CLIENT: 334 VXNlcm5hbWU6 and the line after SERVER -> CLIENT: 334 UGFzc3dvcmQ6, will this work in ALL cases?

3. Any other details I should obfuscate in the logs?

Feature request

If this sort of thing is useful for more people (and I think it is), could phpMailer take care of this? Give me an option to request a sanitized debugOutput in the first place?

Maybe this should have two levels: a) sanitize just usernames and passwords, or b) sanitize both passwords and URLs/domain names.

I believe a wise handling of this situation requires deep knowledge of SMTP protocols, so it should be done within the code that actually understands and specializes in SMTP.

Thanks in advance for any insights.

[Synchro]

You can already inject your own handler into Debugoutput, and that would be a straightforward solution to this, though it might be difficult to spot this info (Debugoutput doesn't get a lot of context passed to it) - the 334 response code would be the thing to look out for. It may be better to create a subclass of the SMTP class that suppresses the debug output of the auth function. Suppressing it by default isn't a great idea - it would undermine its utility for debugging, which is why it's there, not for SMTP logging, which your mail server already does for you (and you can probably extract from logs using the transaction ID passed back from the DATA command, if your mail server provides one).

When you're using TLS, you're on the inside of the encrypted tunnel, so to you it looks the same as cleartext, but of course it's all invisible to anyone on the outside. None of the commonly supported auth schemes (including OAuth) are "safe" to use without encryption. Also, you shouldn't really be using debug output in production, and definitely not at level 4, which is way too verbose to be useful for general SMTP logging - it's for diagnosing low-level issues in PHPMailer.

[pgorod]

Thanks for your clarifications. I agree level 3 debugging is more than enough. I am already using a (very simple) injected debug handler to get the debugOutput.

I started this whole thing following two hard cases I met in real-world scenarios of failing SMTP configurations:

1. From field was different from the Account email address, so the server was rejecting it
2. Warning: stream_socket_enable_crypto(): SSL operation failed with code 1. OpenSSL Error messages:
   error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed

On both these cases, SuiteCRM wasn't giving any useful information to the admin user trying to configure email accounts. I am now making sure we grab the Exception string and display it, something we weren't doing, but I still find the debug logs tell a more complete story, you can get a better sense of what worked, and what didn't.

Anyway, I'm not intending to use this for every email that gets sent from the system, that's overkill, I was planning of including this in the Email accounts configuration area, where you can click a Test Sending an email button. I'm seeing too many people getting stuck there when they aren't able to get the configuration right. It's normal for a CRM to have one or more outbound accounts, plus bounce-handling accounts, plus one account per user, plus some group accounts, so it can get complicated.

Part of these problems were caused by bugs in our new Email Module code, part is because we have many clueless sysadmins using our software, and part is because there are actual complexities in SMTP as servers get more restrictive and things like Certificates configuration come into play.

I take it the feature suggestion doesn't sound appropriate to you? If so, it's alright for you to close this Issue and I will keep working on it myself. Thanks for your input, it _was_ helpful.

[pgorod]

Thanks @Synchro!

In numeric terms, the first debug level that includes passwords would be what, 4?

[Synchro]

I had more of a think about this. I often see debug output posted on here and Stack Overflow that includes encoded credentials, and that's obviously a bad thing. The problem is that the point at which the debug info was output did not know the command context, and attempting to parse the data is unreliable, so what I've done is to pass in the command to provide context at the point data is sent. This allows us to be more selective about debug output. I've made it so that credentials are hidden at all but the most verbose level of debug output (4) - all the docs typically advise using level 2 or 3 so this will deal with the vast majority of credential leaks without preventing access to that data if it's needed.

[pgorod]

@Synchro I expect this will be coming out as PhpMailer 5.2.27 pretty soon. Do yo have any set date, or a Release Schedule I can check? I would like to synchronize our SuiteCRM release with this, so that I can put in the code to provide detailed logs at the same time that we update the included PhpMailer to have the obfuscated output.

[Synchro]

It won't be in 5.2, only 6.0. Releases are generally whenever I get around to it or something significant or urgent comes up. I'll aim to do one tomorrow - there was another thing someone else was waiting for.

[pgorod]

Hhmm ok. I'll have a look at the Upgrade Guide, it looks a bit more challenging for us, but that's the way to the future, I guess. Thanks.

[Synchro]

This was released in 6.0.2 last week.