Osticket: Error sending mails with German umlauts

[ChristianFranke]

Prerequisites

• [ ] Can you reproduce the problem in a fresh installation of the "develop" branch?
• [ ] Do you have any errors in the PHP error log, or javascript console?
• [X] Did you check the osTicket forums?
• [X] Did you perform a cursory search to see if your bug or enhancement is already reported?

For more information on how to write a good bug report

Description

We cannot reply to customers who have umlauts in the sender name (Ü, Ö, Ä, ß). In addition, the e-mail address is not saved in the customer profile (only an "@" is saved).

Steps to Reproduce

1. send mail to supportsystem with from-name with umlauts (for example Thomas Müller <[email protected]>)
2. add reply from support-system

Expected behavior: the reply should reach the customer

Actual behavior: there is an error (admin -> system logs):

Unable to email via SMTP:ssl://smtp.gmail.com:465 Validation failed for: ""Wolfgang Schönberg""

Moreover, the email of the customer is replaces by an "@" (see screenshots below).

Versions

osTicket-Version | v1.12 (a076918) —  Aktuell
Server-Software | Apache
MySQL-Version | 5.5.62
PHP-Version | 7.3.4

We send mails over gmail (imap / smtp).

Screenshots

System log

Customer profile

[JediKev]

@ChristianFranke

Is this for SupportSystem or osTicket? If SupportSystem, please close this and report the bug to our SupportSystem support email directly on our contact-us page here. If osTicket, keep it open and we'll look into it.

Cheers.

[ChristianFranke]

Yes, its for osTicket, Ill keep it open

[matzneda]

@ChristianFranke

I was about to update osTicket from 1.11 to 1.12. Can you confirm this Umlaut-Error only happens since you updated to v1.12, or did you have this error in 1.11 or 1.10 as well?

[ChristianFranke]

@ChristianFranke

I was about to update osTicket from 1.11 to 1.12. Can you confirm this Umlaut-Error only happens since you updated to v1.12, or did you have this error in 1.11 or 1.10 as well?

@matzneda Unfortunately we previously used version v1.9.14 (according to bootstrap.php -> THIS_VERION). GIT_VERSION was 8b927a0 by the way. And yes, these problems did not exist before.

[matzneda]

@ChristianFranke Due to security vulnerabilities I have updated to osTicket v1.12. I have tested this case again and we have no problems here. All E-Mails can be received, no matter if they are sent by creating a ticket, or when an agent replies.

Here are my system details:

osTicket Version | v1.12 (a076918) 
Web Server Software | Apache
MySQL Version | 5.7.26
PHP Version | 7.2.17

[ChristianFranke]

@ChristianFranke Due to security vulnerabilities I have updated to osTicket v1.12. I have tested this case again and we have no problems here. All E-Mails can be received, no matter if they are sent by creating a ticket, or when an agent replies.

Thanks for your reply, we can also receive all mails, only the e-mail address will be replaced by an "@" (see my screenshots above) in case of an umlaut letter in the sender's name ("From"). Then we have to search the mailheader for the original name and the original e-mail address and edit the created customer in osTicket (see my screenshots above).

[matzneda]

Thanks for clarifying. Now we have the error as well.
It only happens when creating a ticket by e-mail from an address that has an umlaut letter in the sender's name. Ticket and user is created correctly, but the user without e-mail address (only the "@") which makes it impossible to reply.

[JediKev]

@ChristianFranke

If the name has a special character such as an umlaut the email client should escape the character by double quoting the entire name. This is an email standard listed in the RFCs (RFC 5322 to be exact) that everyone is supposed to follow. If you do not escape the characters most Email parsers will not know what to do and break causing issues like this. We have made a pull request to quote the name if contains special chars when mail comes in to our software as people apparently love writing email software without checking/following the standards ;)

Cheers.

[drdaxxy]

@JediKev What PR are you referring to?

I don't think that's quite it - note that @ChristianFranke's customer's name is double-quoted in his customer profile.

Running osTicket v1.12 on Debian Stretch php7.0 7.0.33-0+deb9u3, I encountered the same issue just now.

In our case, the problematic email has a RFC 2047-encoded and double-quoted From: header, such as:

From: =?UTF-8?Q?=22M=C3=A4x_Mustermann=22?= <[email protected]>

However, we have received many tickets from customers whose names contain umlauts before. Looking at the last few, I see no problems when the email has a RFC 2047-encoded but not double-quoted From: header, such as:

From: =?UTF-8?Q?M=C3=A4x_Mustermann?= <[email protected]>

(I don't know whether email programs should double-quote the name when using RFC 2047, I have not read the relevant RFCs.)

When manually putting the address into our customer's profile, I found that osTicket refused to update the profile before I removed the quotes from the name field, which may be the actual issue here.

[JediKev]

@drdaxxy

Sorry, thought I linked it:
https://github.com/osTicket/osTicket/pull/4894

The description mentions an un-quoted email address but the same applies to names with special characters.

So the User's name had the quotes in it in their profile? (ie. The name in the User's Profile was "Mäx Mustermann" exactly?)

Cheers.

[drdaxxy]

So the User's name had the quotes in it in their profile? (ie. The name in the User's Profile was "Mäx Mustermann" exactly?)

I replaced our actual customer's name with a placeholder, but yes, it was formatted exactly like that. See the screenshots in the issue description for reference.

Note also that the email address the fetcher stored in the profile was just "@", I had to check our IMAP mailbox for the original message to find the sender's actual address.

[bmorg]

We are running into this issue as well. Somebody else was also experiencing this problem (on the forums).

At least two popular German email providers, GMX and web.de (owned by the same parent company), are sending display names containing quotes (") when the customer's name contains umlauts. This means that the quotes are actually part of the display name, not surrounding it (see above). IMO that is a bug on their side, but if I interpret RFCs 5322 and ​2047 correctly it is technically still correct.

There are two ways display names can contain quotes:

• As mentioned by @drdaxxy, MIME headers (RFC 2047) can contain encoded quotes in the display name.
• RFC 5322 already allows quotes in display names if they are properly escaped:
  display-name = phrase phrase = 1*word / obs-phrase word = atom / quoted-string quoted-string = [CFWS] DQUOTE *([FWS] qcontent) [FWS] DQUOTE [CFWS] qcontent = qtext / quoted-pair quoted-pair = ("\" (VCHAR / WSP)) / obs-qp VCHAR = %x21-7E // RFC 5234

The quotation mark " being ASCII 22 makes it a VCHAR, allowing for it to be escaped into a quoted-pair and therefore appear inside a quoted-string. So the following would be a valid From header:

From: "ab \" cd" <[email protected]>

This complexity is not being taken into account by osTicket. So when '"%s" <%s>' is being used as a pattern for email headers without escaping the display name, e.g. here and here, the resulting string cannot be parsed. To be specific, an error is returned by the call to Mail_RFC822::parseAddressList in line 739 here.

[AZimmerhofer]

Hello, everybody,

I have exactly the same problem with the latest Osticket version (v1.14.1). I also checked that the update of the PEAR file linked here is really up-to-date on the server. Everything is fine.

Email addresses of email senders with umlaut in their name are simply replaced by @. Exactly as described above/shown in screenshots.

[AZimmerhofer]

Do none of you have the same problems? @ChristianFranke @drdaxxy @JediKev

[guidokuehler]

No or better Nö

Best Regards
Guido

Von: "AZimmerhofer" [e-mail]
An: "osTicket/osTicket" [e-mail]
CC: "Subscribed" [e-mail]
Gesendet: Sonntag, 9. Februar 2020 13:51:52
Betreff: Re: [osTicket/osTicket] Error sending mails with German umlauts (#4884)

Do none of you have the same problems? @ChristianFranke @drdaxxy @JediKev

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub , or unsubscribe .

[commeco]

Hello,
we have the same problem. A customer of us (with > 500 Tickets a day) runs into that problem daily.
We can confirm that is most of the time from gmx. and web.de addresses in combination with umlauts.

Any idea?

Thank you,

Matthias

[guidokuehler]

Exist the Problem when Emails sent from gmx or web.de to OsTicket or when sending OsTicket to gmx or web.de?

Best Regards
Guido

Von: "Matthias Gutmann" [e-mail]
An: "osTicket/osTicket" [e-mail]
CC: "guidokuehler" [e-mail], "Comment" [e-mail]
Gesendet: Montag, 17. Februar 2020 14:05:10
Betreff: Re: [osTicket/osTicket] Error sending mails with German umlauts (#4884)

Hello,
we have the same problem. A customer of us (with > 500 Tickets a day) runs into that problem daily.
We can confirm that is most of the time from gmx. and web.de addresses in combination with umlauts.

Any idea?

Thank you,

Matthias

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub , or unsubscribe .

[commeco]

Hi Guido,
only if osTicket receives the mails.

Regards Matthias

[AZimmerhofer]

We have the problem in the following situations: Apparently, when the name of the e-mail sender for a new ticket is written with an umlaut. At the same time the web server must be operated by GMX or Web.de.

[guidokuehler]

Sorry for the Delay but we have nobody with this Requirements of your Situation.

But we have look in old emails from Umlaut-GMX-Senders to us and find followed.

The Emails comes before a long time from the same client via GMX to us.

/*

1. Jan 2016
   X-Mailer: Microsoft Office Outlook 12.0
   From: =?utf-8?Q?K=C3=BChler_M#####?= Content-Type: text/plain;
   charset="utf-8"

19 Dec 2018
X-Mailer: Microsoft Office Outlook 12.0
From: =?iso-8859-1?Q?K=FChler_M#####?= Content-Type: text/plain;
charset="iso-8859-1"

*/

What we can see here?

Emails are sent with different charsets.
We think, that the problem based on an specified charset?

Can you verify and compare the charsets from specified Messages in the used Support-Inbox (Compare Charsets from non-problem and problem-message!)

We hope that OSTicket Developers can look than deeper in the Code to find the Problem and find a solution for you.

Regards
Guido

Von: "AZimmerhofer" [e-mail]
An: "osTicket/osTicket" [e-mail]
CC: "guidokuehler" [e-mail], "Comment" [e-mail]
Gesendet: Montag, 17. Februar 2020 16:17:11
Betreff: Re: [osTicket/osTicket] Error sending mails with German umlauts (#4884)

We have the problem in the following situations: Apparently, when the name of the e-mail sender is written with an umlaut. At the same time the web server must be operated by GMX or Web.de.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub , or unsubscribe .

[AZimmerhofer]

We got two mails last days - Both names of the e-mail senders contain umlauts

WEB.DE -> PROBLEM
From: =?UTF-8?Q?=22Oz_G=C3=BClb#a#e=22?=

OTHER SERVER -> NO PROBLEM
From: =?iso-8859-1?Q?Sc#w=E4rzl=2C_Se##s#ian?=

[JediKev]

Does anyone have a _full_ example of an email that's causing issues? Please upload the original, raw email as a .txt file or just paste it in code brackets; when I say raw email, I mean the entire email with full raw headers and full raw body. You should obviously change sensitive data in the headers/body as you will be posting it publicly.

Once we get a full example we will be able to test appropriately.

Cheers.

[AZimmerhofer]

Hi,

here header and raw email text:

Return-Path: <[email protected]>
X-Original-To: [email protected]
Delivered-To: [email protected]
X-policyd-weight: using cached result; rate: -7
Authentication-Results: dd17714.xxx.com;
    dkim=pass (1024-bit key; unprotected) header.d=gmx.net [email protected] header.b="OLN8heOk";
    dkim-atps=neutral
Received: from xxx.xxx.net (xxx.xxx.net [xxx])
    by xxx.xxx.com (Postfix) with ESMTPS id C64D431E028D
    for <support@xxx>; Tue, 18 Feb 2020 17:54:23 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.net;
    s=badeba3b8450; t=1582044863;
    bh=ryTMsmv+F7TDnKKR6j1/RgBRRGRgzpq9i5wRLWMmxf8=;
    h=X-UI-Sender-Class:From:To:Subject:Date;
    b=OLN8heOk2fKhKYS/upDKJyCjqOFVjXz2QoYEJo5WDwkMmGCSwqqQHNFlY8+i165t2
     SliKREGSgcsqlNk2vnOkk3X1gkEXR6jJsdTrWAaWzUn6IPi9L9+82BdJmrrOvKwv2Q
     wtXb2ycDoaofDiq8mw45+V6dM4mnfwItAW1h3zzg=
X-UI-Sender-Class: 01bb95c1-4bf8-414a-932a-4f6e2808ef9c
Received: from [82.207.xxx.xxx] ([82.207.xxxx.xx]) by xxx.xxx.net
 (x.xxx.lan [172.19.xxxxx.xx]) (via HTTP); Tue, 18 Feb 2020
 17:54:22 +0100
MIME-Version: 1.0
Message-ID: <trinity-d716f320-743b-4d3e-b91d-c02653e4740c-1582044862966@3c-app-gmx-bap67>
From: =?UTF-8?Q?=22Test_=C3=A4=C3=B6=22?= <[email protected]>
To: [email protected]
Subject: Test
Content-Type: text/html; charset=UTF-8
Date: Tue, 18 Feb 2020 17:54:22 +0100
Importance: normal
Sensitivity: Normal
X-Priority: 3
X-Provags-ID: V03:K1:6gkhOWfcPctrjyzqq+gyBwrA/woOvfdU8hQfuZGd0TpNclU8dA/117KrHXSOCYsHmdUPm
 jB23UQnevKMOPTee87duYkbREVsfbE2hGqENo8t6EqckT+tWETF6XeLDW5Dyvu4MTJgh0QY7Svhy
 ASv0mAWOGo1iRVCDbdLfi/fr4bDsXCXo8Is82JlsmwnoPe+2tEN8tFS6HhNdxY+mVz87g6eklb59
 o1BBSsErufWKTLP6Jak7u9WMAEwheMTgxHW3PgvQ2TO9BMXvJMGst+7d0LZyQt2O9Aa1RQnrAZ5Z
 sE=
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:s53m3JTo3ec=:TWCO4Jxs/hm17Z+Vu6kkri
 rH0ICB13ve+SamlA++ZIRJ8J0xNPkoDYBRafL503iZM6o9aVMqqdIYBK0fc3Lu8J7kFGOXXZB
 cBdaH6kx2HyndXVWTC/RkNnNKhyqPXBB+zaiYHkDDiKeKbGmppbNsBVVz7E9PU69o010mldOF
 XByyPksmqh1qMh3Ard3vR3tlUyUYKLuixYfUPDWhQAeYfx+fRRPn+SfKsV1sH3Y2IcoVzQkAQ
 8R6FLtXCoZSdx3HZQf5FNbisZGD2hbWBhZwV8f9DQfAQtKjQzHpaWa0Xu4gxH+ImT6SrzomJq
 TVNykwszo+iRJ9lDRnPu5zpNohperLvLZqhkHmVw4jGqsV36Scd+iRDfCFeLk1dAcnwTvIKoZ
 TYJqGah0jwdc18/c8NVC3/2IXghVhtppZ0qAF8K055FDw3dtWZYIimk3WLZxazXsB8AX/ERpI
 IIGIR1dj+VULAafVvf7UpxbynrUZY7Wm2Za7gUwWh1pODZI+mLiTfnQniV3fX6d2mHXL2Mkph
 zGCiaK7MUbQNUwBOvzc8P3yh1eg+h23hLdgnYbN96UNClIl/MqDt8QMSYazH+DJP7CoGg9AgS
 DlsYn+jykmpZGCqOTktS/a6Arc63Q/n3XpH8X2Yv4g6K61273si+NPl68bF1tDKkvd2CGrALA
 HN3Vn7BouZNxE1+TowGVPN7ZYDGhC5LgEQSNwDAAObVn9Ju6yVyOR8SCVOc+WbvoLpKo1uGt+
 EORZc4iXUM4yHCFl8LPAUIJXtbnkuhiGVfo+4o6KdbLDdc4UxqPxwgsl+God48DiRdA9f8/w+
 SaPihc/
X-KasLoop: m034c6fb

<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>Test</div></div></body></html>

[JediKev]

@AZimmerhofer

Thank you, I'm finally able to replicate this issue. I will look into this and get back to you.

Cheers.

[JediKev]

@AZimmerhofer

I was only able to replicate this because I put an umlaut in the email address as well by accident which obviously won't work. So I'm back to not being able to replicate this using your example exactly. Are you sure you have the following pull request applied?

• https://github.com/osTicket/osTicket/pull/5131

Cheers.

[AZimmerhofer]

@JediKev

_Are you sure you have the following pull request applied?_
Yes.

The umlauts are in the name of the sender together with the quotation marks "Franz Müller".

[JediKev]

@AZimmerhofer

Okay, and I tested that but cannot replicate the issue using:

From: =?UTF-8?Q?=22Oz_G=C3=BClb#a#e=22?= <[email protected]>

nor:

From: =?iso-8859-1?Q?Sc#w=E4rzl=2C_Se##s#ian?= <[email protected]>

Anyone else have any other examples?

Cheers.

[AZimmerhofer]

@JediKev

Hello,
I can now reproduce the problem. Can I send an email to a ticket system of yours, @JediKev? Then you have the ticket directly with you?

[JediKev]

@AZimmerhofer

I was finally able to replicate the issue after extensive testing and I believe I have a fix. Please implement the following patch and retest:

diff --git a/include/class.mailfetch.php b/include/class.mailfetch.php
index 70adf0c..bde63b4 100644
--- a/include/class.mailfetch.php
+++ b/include/class.mailfetch.php
@@ -324,7 +324,7 @@ class MailFetcher {

         $sender=$headerinfo->from[0];
         //Just what we need...
-        $header=array('name'  => $this->mime_decode(@$sender->personal),
+        $header=array('name'  => trim($this->mime_decode(@$sender->personal), '"'),
                       'email'  => trim(strtolower($sender->mailbox).'@'.$sender->host),
                       'subject'=> $this->mime_decode(@$headerinfo->subject),
                       'mid'    => trim(@$headerinfo->message_id),

Note:
For a proper test, ensure you fetch an email with a FROM address that is not already in the system. If the FROM address is already linked to a User in osTicket it will just reuse the account instead of creating a new one (which is what we need to test).

Cheers.

[AZimmerhofer]

@JediKev
THANKS! It's working.

[JediKev]

@AZimmerhofer

THANKS! It's working.

Not a problem at all. Thank @bmorg as well as he provided very helpful information.

I also have to thank you for providing useful and pertinent examples.

Edit:
I am going to make a pull request very soon for this and link it here once complete. For anyone wondering, the root of the issue is as explained by @bmorg where some mail providers are surrounding names containing umlauts with quotes. These quotes are encoded (mostly UTF-8) and become a part of the actual name. Class mailparse trims the quotes from the name before using it anywhere in the codebase whereas mailfetch does not. This is why I couldn’t replicate the issue with piping the mail. So all we needed to do is mimic mailparse and trim the surrounding quotes in class mailfetch.

Cheers.