Phpmailer: Subject Encoding

[sergeyklay]

Hi!

I want to know any special settings needed for Subject consisting of the non-Latin characters? I am interested in UTF-8.

Of course I understand that need to set $CharSet value. I do this in my application, after creating a PHPMailer object:

        // Create phpmailer object
        $this->mail = new PHPMailer(true);

        // Set some defaults
        $this->mail->setFrom('[email protected]',  'Gleez CMS');
        $this->mail->WordWrap = 70;
        $this->mail->CharSet  = 'UTF-8'; // the same as 'utf-8'
        $this->mail->XMailer  = 'Gleez CMS 0.10.5';
        $this->mail->setLanguage('ru');
        $this->mail->Debugoutput = 'error_log';

Please see screenshot

Mail body is displayed correctly (in Russian), but Subject - no.

Mail header for this screenshot

Return-Path: <[email protected]>
Received: from cerber.homedomain ([213.87.143.240])
        by mx.google.com with ESMTPSA id ac2sm829046lbc.10.2013.11.06.15.52.11
        for <[email protected]>
        (version=TLSv1.2 cipher=RC4-SHA bits=128/128);
        Wed, 06 Nov 2013 15:52:12 -0800 (PST)
Received: from www-data by cerber.homedomain with local (Exim 4.80)
    (envelope-from <[email protected]>)
    id 1VeCtq-0001ha-2l
    for [email protected]; Thu, 07 Nov 2013 03:52:38 +0400
To: Sergey <[email protected]>
Subject: =?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
 =?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05Ddz89?=
 =?UTF-8?B?CiA9P3V0Zi04P0I/MFliUXVOQyswTDNRdmRHTDBZVWcwTFRRc05DOTBMM1Jp?=
 =?UTF-8?B?OUdGSU5DMDBMdlJqeUJUWlhKblpYaz0/PQ==?=
X-PHP-Originating-Script: 1000:class.phpmailer.php
Date: Wed, 6 Nov 2013 23:52:37 +0000
From: Gleez CMS <[email protected]>
Message-ID: <[email protected]>
X-Priority: 3
X-Mailer: Gleez CMS 0.10.5
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

I sent mail using the localhost:

• Debian 7.2 (wheezy)
• Kernel 3.2.0-4-amd64
• Exim 4.80 (as /usr/sbin/sendmail)
• PHPMailer 5.2.7 (github master branch)

I use smarthost setting for Gmail

Locale:

LANG=ru_RU.utf8
LANGUAGE=
LC_CTYPE="ru_RU.utf8"
LC_NUMERIC="ru_RU.utf8"
LC_TIME="ru_RU.utf8"
LC_COLLATE="ru_RU.utf8"
LC_MONETARY="ru_RU.utf8"
LC_MESSAGES="ru_RU.utf8"
LC_PAPER="ru_RU.utf8"
LC_NAME="ru_RU.utf8"
LC_ADDRESS="ru_RU.utf8"
LC_TELEPHONE="ru_RU.utf8"
LC_MEASUREMENT="ru_RU.utf8"
LC_IDENTIFICATION="ru_RU.utf8"
LC_ALL=

Excuse me, I'm not sure what other information I should provide.

It seem to me this issue associated with the UTF-8 string which is divided into blocks of 78 bytes. Probably with short strings is no problem. I really have tried different methods of decoding/encoding the Subject (mb_encode_mimeheader, utf_encode, base64_encode, strtolower($CharSet) - any tricks that I could find on the google.) and not found solutions.

Another tip that will help you understand the cause of the issue: Subject consists of Latin and non-Latin characters. The first and last word in English (Latin), the rest - Russian words (Cyrillic).

Any ideas?

[Synchro]

I think you're getting a bit enthusiastic with the encoding. If I start with your subject line:

Subject: =?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
 =?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05Ddz89?=
 =?UTF-8?B?CiA9P3V0Zi04P0I/MFliUXVOQyswTDNRdmRHTDBZVWcwTFRRc05DOTBMM1Jp?=
 =?UTF-8?B?OUdGSU5DMDBMdlJqeUJUWlhKblpYaz0/PQ==?=

I decode that and I get:

Subject: =?utf-8?B?R2xlZXogQ01TIC0g0J/RgNC+0LLQtdGA0LrQsCDRgNC10LPQuNGB0YLRgNCw?=
 =?utf-8?B?0YbQuNC+0L3QvdGL0YUg0LTQsNC90L3Ri9GFINC00LvRjyBTZXJnZXk=?=

which is what appears in your screen shot. First thought: "it failed to decode!"; Second thought: "It's encoded twice!"!

Decode it again, and hey presto:

Subject: Gleez CMS - Проверка регистрационных данных для Sergey

Are you, by any chance, encoding the header before passing it to PHPMailer? If so, just don't. PHPMailer will encode it for you. Just say:

$this->mail->Subject = 'Gleez CMS - Проверка регистрационных данных для Sergey';

PHPMailer is actually cleverer than that too. B-encoding is quite an overhead if only a few chars actually need encoding, so it checks which of B and Q encoding (or none) will produce the shorter result.

[sergeyklay]

@Synchro For experimental purposes, I really tried use a simple, test string:

$this->mail->Subject = 'Gleez CMS - Проверка регистрационных данных для Sergey';

and received it

Return-Path: <[email protected]>
Received: from cerber.homedomain ([213.87.143.215])
        by mx.google.com with ESMTPSA id vs11sm4256571lac.3.2013.11.07.08.59.52
        for <[email protected]>
        (version=TLSv1.2 cipher=RC4-SHA bits=128/128);
        Thu, 07 Nov 2013 08:59:57 -0800 (PST)
Received: from www-data by cerber.homedomain with local (Exim 4.80)
    (envelope-from <[email protected]>)
    id 1VeSwI-0001W3-Vy
    for [email protected]; Thu, 07 Nov 2013 21:00:15 +0400
To: Sergey <[email protected]>
Subject: =?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
 =?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05Ddz89?=
 =?UTF-8?B?CiA9P3V0Zi04P0I/MFliUXVOQyswTDNRdmRHTDBZVWcwTFRRc05DOTBMM1Jp?=
 =?UTF-8?B?OUdGSU5DMDBMdlJqeUJUWlhKblpYaz0/PQ==?=
X-PHP-Originating-Script: 1000:class.phpmailer.php
Date: Thu, 7 Nov 2013 17:00:14 +0000
From: Gleez CMS <[email protected]>
Message-ID: <[email protected]>
X-Priority: 3
X-Mailer: Gleez CMS 0.10.5
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

I do not quite guess what is wrong.

[Synchro]

I can't reproduce this error. This is my complete test script. I'm sending to a local fake mail server and the subject appears correctly.

<?php
require 'PHPMailerAutoload.php';
$mail = new PHPMailer;
$mail->isSMTP();
$mail->Host = 'localhost';
$mail->Port = 2500;
$mail->setFrom('[email protected]', 'Gleez CMS');
$mail->addAddress('[email protected]');
$mail->WordWrap = 70;
$mail->CharSet  = 'UTF-8'; // the same as 'utf-8'
$mail->XMailer  = 'Gleez CMS 0.10.5';
$mail->setLanguage('ru');
$mail->Subject = 'Gleez CMS - Проверка регистрационных данных для Sergey';
$mail->Body = 'hello';
$mail->send();

When I open the resulting email, it looks like this:

Maybe try a fake mail server, or enable debug output in SMTP to see what's going out. Also try putting a breakpoint or an echo in the encodeHeader function as that's the only place that B-encoding is done, and see if it's called more than once for the subject.

[sergeyklay]

@Synchro Yes, I spent a small exeperimental

$str = '=?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
 =?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05Ddz89?=
 =?UTF-8?B?CiA9P3V0Zi04P0I/MFliUXVOQyswTDNRdmRHTDBZVWcwTFRRc05DOTBMM1Jp?=
 =?UTF-8?B?OUdGSU5DMDBMdlJqeUJUWlhKblpYaz0/PQ==?=';

$decoded =  imap_utf8(imap_utf8($str));
// Gleez CMS - Проверка регистрационных данных для Sergey

$decoded =  mb_decode_mimeheader(mb_decode_mimeheader($str));
// Gleez CMS - Проверка регистрационных данных для Sergey

It's good - at least I now understand what is happening. But frankly I'm not quite (yet) know what can I do

[Synchro]

I spotted something suspicious. The double-encoded header you are getting has a charset of UTF-8 (in upper case, as does my test), but the decoded one has utf-8 (in lower case). Given that the script is definitely set to UTF-8, and that's what PHPMailer uses when it's doing the encoding (and it doesn't call strtolower), it suggests that something else is doing the encoding before it gets to the PHPMailer encoder. Maybe you have some test code lurking?

[sergeyklay]

Full listing

First class

class Email {
    protected $mail;

    public static function factory() {
        return new Email();
    }

    public function __construct()
    {
        require 'PHPMailerAutoload.php';

        // Create phpmailer object
        $this->mail = new PHPMailer(true);

        // Set some defaults
        $this->mail->setFrom('[email protected]',  'Gleez CMS');
        $this->mail->WordWrap = 70;
        $this->mail->CharSet  = 'utf-8';
        $this->mail->XMailer  = 'Gleez CMS 0.10.5';
        $this->mail->setLanguage('ru');
        $this->mail->Debugoutput = 'error_log';
    }

    public function subject($subject)
    {
        $this->mail->Subject = $subject;

        return $this;
    }

    public function to($email, $name = NULL)
    {
        $this->mail->addAddress($email, $name);

        return $this;
    }

    public function message($body, $type = NULL)
    {
        if ( ! $type OR $type === 'text/plain')
        {
            // Set the main text/plain body
            $this->mail->Body = $body;
        }
        else
        {
            // Add a custom mime type
            $this->mail->msgHTML($body);
        }

        return $this;
    }

Another class

        // Create an email message
        $email = Email::factory()
            ->subject('Gleez CMS - Проверка регистрационных данных для Sergey')
            ->to('[email protected]', 'Sergey')
            ->message('hello'); 

        // Send the message
        $email->send();

Now result is (Source of mail in GMail)

Return-Path: <[email protected]>
Received: from cerber.homedomain ([213.87.143.215])
        by mx.google.com with ESMTPSA id jv8sm3718426lbc.2.2013.11.07.10.36.59
        for <[email protected]>
        (version=TLSv1.2 cipher=RC4-SHA bits=128/128);
        Thu, 07 Nov 2013 10:37:02 -0800 (PST)
Received: from www-data by cerber.homedomain with local (Exim 4.80)
    (envelope-from <[email protected]>)
    id 1VeUSH-0001qr-DC
    for [email protected]; Thu, 07 Nov 2013 22:37:21 +0400
To: Sergey <[email protected]>
Subject: =?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
 =?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05Ddz89?=
 =?UTF-8?B?CiA9P3V0Zi04P0I/MFliUXVOQyswTDNRdmRHTDBZVWcwTFRRc05DOTBMM1Jp?=
 =?UTF-8?B?OUdGSU5DMDBMdlJqeUJUWlhKblpYaz0/PQ==?=
X-PHP-Originating-Script: 1000:class.phpmailer.php
Date: Thu, 7 Nov 2013 18:37:20 +0000
From: Gleez CMS <[email protected]>
Message-ID: <[email protected]>
X-Priority: 3
X-Mailer: Gleez CMS 0.10.5
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

Subject in GMail

=?utf-8?B?R2xlZXogQ01TIC0g0J/RgNC+0LLQtdGA0LrQsCDRgNC10LPQuNGB0YLRgNCw?= =?utf-8?B?0YbQuNC+0L3QvdGL0YUg0LTQsNC90L3Ri9GFINC00LvRjyBTZXJnZXk=?=

It turns out indeed, something is trying to encode a 2 times Subject. But then how can I decode the string before passing it to the class?

[Synchro]

OK, I set up a test using your code (I added a send function to the class), and sending via SMTP or sendmail (using the fakesendmail script in the tests folder), just adding these lines to the constructor:

$this->mail->isSendmail();
$this->mail->Sendmail = './test/fakesendmail.sh';

both sending mechanisms produced identical, correct subject lines. Maybe your mail server is doing something strange? Can you try using the fake smtp or sendmail servers?

[sergeyklay]

Hm... Maybe mail server is doing something strange...
I have to to experience. I need some time to install and configure the appropriate software. I'll start it right now.

[sergeyklay]

Done! I manually created fake sendmail server (shell script) and used test described above.
This is mail (text file): http://vpaste.net/0coAA

[Synchro]

Hm. So it's still double-encoding. Next step is to track down how that's happening. If you have a debugger configured (e.g. xdebug in PHPStorm), set a breakpoint in the encodeHeader function around line 2269 which is $encoding = 'B';. Otherwise print the output from debug_backtrace() from there. that way we can see how its getting there, and if it really is happening twice. What version of PHP are you running? Incidentally, I'm using the current master branch code rather than the 5.2.7 release - it might be worth checking that.

[sergeyklay]

To my regret I am having difficulty with the debugger. At least further index.php I was not able to pass. Tomorrow I'll try to understand. I'll let you know results of debugging. Sorry to trouble you.

[sergeyklay]

I managed to understand what to do. Sorry for my stupidity.

PHP 5.4.4-14+deb7u5
PHPMailer current master branch (class.phpmailer.php contains Version 5.2.7)

PHPStrom screen:

Excuse me, I'm not sure what other information about debugging I should provide?

[Synchro]

What I was suggesting is that with the breakpoint set there, and the problem you're seeing, we would expect to hit that line twice. in your screen shot we can see that it's being called from preSend, as expected, and $str contains the unencoded subject line - what I want to know is how and when it gets called again.

[sergeyklay]

What should I do? What kind of special debugging techniques? Any ideas?

[Synchro]

Nothing too special - the lower left-hand panel shows the call stack, so you can see how it got to this point. If you click the green arrow on the left, it will continue until it hits the breakpoint for the second time (or not, if it doesn't happen), at which point we'll (hopefully) know how it got there.

[sergeyklay]

Here's what I got this time

[Synchro]

Aha! I think we've found it. The subject is encoded during preSend, but also during postSend when using the 'mail' transport in the mailPassthru function, which is why I didn't see it. I'll have more of a look tomorrow - it's 1:40am here! In the mean time, if you can switch to using sendmail (call isSendmail()) it will probably fix it temporarily.

[sergeyklay]

Ok! Good night :) Catch you later

[Synchro]

I think I spoke too soon. While encodeHeader is called in mailPassthru, it is applied to the unencoded Subject property, not a header that may have already been encoded. In your last screen shot, you can see that the subject (in $str) is not encoded. I'm sending using the mail transport and it's not having a problem.
So we've seen that encodeHeader gets called twice, legitimately - does it get called a third time? I'm still suspicious of the casing of the charset.
One interesting point: when createHeader is called when you're using the 'mail' transport, it does not create a subject header because it is created by the PHP mail function itself (see line 1651). I'm wondering if your PHP is encoding the subject during the mail() call and not in PHPMailer. We can test that easily with a standalone script:

<?php
$headers = "To: Sergey <[email protected]>\r\n".
"Date: Fri, 8 Nov 2013 10:21:26 +0100\r\n".
"From: Gleez CMS <[email protected]>\r\n".
"Message-ID: <[email protected]>\r\n".
"X-Priority: 3\r\n".
"X-Mailer: Gleez CMS 0.10.5\r\n".
"MIME-Version: 1.0\r\n".
"Content-Type: text/plain; charset=utf-8\r\n".
"Content-Transfer-Encoding: 8bit\r\n".
"\r\n";
$subject = 'Gleez CMS - Проверка регистрационных данных для Sergey';
$mailto = '[email protected]';
$body = 'hello';
mail($mailto, "=?utf-8?B?".base64_encode($subject)."?=", $body, $headers);

That works fine for me, but see if it still has the same problem for you.

[sergeyklay]

Now, I don't understand anything :(
I created a simple script, an example of which you provided above and sent the mail using fake mail server.

I has received the following:

To: [email protected]
Subject: =?UTF-8?B?PT91dGYtOD9CP1IyeGxaWG9nUTAxVElDMGcw?=
=?UTF-8?B?Si9SZ05DKzBMTFF0ZEdBMExyUXNDRFJnTkMxMExQUXVOR0IwWUxSZ05DdzBZ?=
=?UTF-8?B?YlF1TkMrMEwzUXZkR0wwWVVnMExUUXNOQzkwTDNSaTlHRklOQzAwTHZSanlC?=
=?UTF-8?B?VFpYSm5aWGs9Pz0=?=
X-PHP-Originating-Script: 1000:test.php
To: Sergey <[email protected]>
Date: Fri, 8 Nov 2013 23:59:59 +0100
From: Gleez CMS <[email protected]>
Message-ID: <[email protected]>
X-Priority: 3
X-Mailer: Gleez CMS 0.10.5
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit


hello

Evidently should have started with this test. I distract you in vain ...
Most likely something is wrong with my php settings. Sorry to trouble you, I am so sorry.

Best regards

[sergeyklay]

The main problem was that the Subject is encoded twice.

Then I began to think that may overlap in general is a standard work php mail()... Yes this is Mbstring!

mbstring.func_overload
For example, if it is set to 7, mail, strings and regular expression functions will be overloaded. 

I look at my php.ini and saw mbstring.func_overload = 7. Wow!
I replaced it with 6 and the problem disappeared! Taking into account this table http://php.net/manual/en/mbstring.overload.php the difference between the 7 and 6 - only one, overload mail() function.

Thanks!

[Synchro]

Bah! Oh well, at least we know now, and it will show up in searches here if anyone runs into it again...

[sergeyklay]

Yep )

[ooxi]

Could we maybe check for mb overloading and disable manual encoding?

[Synchro]

Yes, that's a good idea. Got a PR? :)

[Synchro]

Fixed in master.

[sergeyklay]

Good work! :+1:

[xeredin]

"=?utf-8?B?".base64_encode($subject)."?="

thank you "=?utf-8?B?".base64_encode($subject)."?=" it solved my problem

[jcaraly]

"=?utf-8?B?".base64_encode($subject)."?="

thank you "=?utf-8?B?".base64_encode($subject)."?=" it solved my problem

Thank you really much, after some times to look everywhere, that's solution work for me too !!
(on PHPMailer - PHP email class Version: 2.0.4) with php 7

[Synchro]

Please don't use that ancient version. I'm surprised it still works at all, but it also likely means you are vulnerable to several major security holes. Fix it properly and use the latest version.

[jcaraly]

Please don't use that ancient version. I'm surprised it still works at all, but it also likely means you are vulnerable to several major security holes. Fix it properly and use the latest version.

Thank you for your reply, Yes I will purpose to update to my customer !
Installed more than 7 years ago and still work properly !