Postal: MAIL FROM gets overwriten by the from: in the DATA

[6temes]

Hi,

First of all, thanks for building this app. What you are doing is really great.

Our application sends emails on behalf of our users. We add the user's email in the From: field and our email address in the Sender: field. That way, the emails arrive to the destinatary with the following format:

From customer@any_domain.com via user@our_own_domain.com

Right now we are using Sendgrid and this working well, so I assume that this is allowed in the standards, but when I send the emails via our _Postal_ server, they get rejected with a 530 error.

Those are the logs:

[smtp.1:25369] [2017-05-11T11:43:31.032] DEBUG -- : [TBDOWT] => 235 Granted for ourdomain/ourdomain
[smtp.1:25369] [2017-05-11T11:43:31.150] DEBUG -- : [TBDOWT] <= MAIL FROM:<[email protected]>
[smtp.1:25369] [2017-05-11T11:43:31.151] DEBUG -- : [TBDOWT] => 250 OK
[smtp.1:25369] [2017-05-11T11:43:31.269] DEBUG -- : [TBDOWT] <= RCPT TO:<[email protected]>
[smtp.1:25369] [2017-05-11T11:43:31.270] DEBUG -- : [TBDOWT] Added external address '[email protected]'
[smtp.1:25369] [2017-05-11T11:43:31.270] DEBUG -- : [TBDOWT] => 250 OK
[smtp.1:25369] [2017-05-11T11:43:31.388] DEBUG -- : [TBDOWT] <= DATA
[smtp.1:25369] [2017-05-11T11:43:31.388] DEBUG -- : [TBDOWT] => 354 Go ahead
[smtp.1:25369] [2017-05-11T11:43:31.506] DEBUG -- : [TBDOWT] <= Date: Thu, 11 May 2017 20:43:29 +0900
[smtp.1:25369] [2017-05-11T11:43:31.663] DEBUG -- : [TBDOWT] <= To: ourdomain - Online Order <[email protected]>
[smtp.1:25369] [2017-05-11T11:43:31.664] DEBUG -- : [TBDOWT] <= From: Test <[email protected]>
[smtp.1:25369] [2017-05-11T11:43:31.664] DEBUG -- : [TBDOWT] <= Subject: Order received from English version of site
[smtp.1:25369] [2017-05-11T11:43:31.664] DEBUG -- : [TBDOWT] <= Message-ID: <[email protected]>
[smtp.1:25369] [2017-05-11T11:43:31.664] DEBUG -- : [TBDOWT] <= X-Mailer: PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)
[smtp.1:25369] [2017-05-11T11:43:31.665] DEBUG -- : [TBDOWT] <= X-SMTPAPI: { "asm_group_id": 2417 }
[smtp.1:25369] [2017-05-11T11:43:31.665] DEBUG -- : [TBDOWT] <= MIME-Version: 1.0
[smtp.1:25369] [2017-05-11T11:43:31.665] DEBUG -- : [TBDOWT] <= Content-Type: text/html; charset=utf-8
[smtp.1:25369] [2017-05-11T11:43:31.665] DEBUG -- : [TBDOWT] <=
[smtp.1:25369] [2017-05-11T11:43:31.665] DEBUG -- : [TBDOWT] <= The actual message....
[smtp.1:25369] [2017-05-11T11:43:31.666] DEBUG -- : [TBDOWT] <=
[smtp.1:25369] [2017-05-11T11:43:31.666] DEBUG -- : [TBDOWT] <= .

This are the headers when they are received by the method find_authenticated_domain_from_headers(headers):

{
  "received"=>["from ourdomain.com (::ffff:125.6.128.154 [::ffff:125.6.128.154]) by postal.6temes.cat with SMTP; Thu, 11 May 2017 11:43:31 +0000"], 
  "date"=>["Thu, 11 May 2017 20:43:29 +0900"], 
  "to"=>["ourdomain - Online Order <[email protected]>"], 
  "from"=>["Test <[email protected]>"], 
  "subject"=>[Order received from English version of site"],
  "message-id"=>["<[email protected]>"], 
  "x-mailer"=>["PHPMailer 5.2.23 (https://github.com/PHPMailer/PHPMailer)"],
  "x-smtpapi"=>["{ \"asm_group_id\": 2417 }"], 
  "mime-version"=>["1.0"], 
  "content-type"=>["text/html; charset=utf-8"]
}

In the SMTP logs, the from field is defined twice. First in the second line: MAIL FROM:<[email protected]> and then again when sending the DATA : From: Test <[email protected]>

I am not an expert in the SMTP protocol and I ignore if, in this case, PHPMailer is working as it should, but, as I said before, Sengrid is accepting those emails.

//////

So, the question is:

• Is the PHPMailer format correct and should we support it?

If yes, we will need to prevent the from in the data to overwrite the MAIL FROM set before. I guess that the change should be performed here:

https://github.com/atech/postal/blob/master/lib/postal/smtp_server/client.rb#L350

I can write the PR if we decide to go ahead.

Sorry for the long issue!!

[adamcooke]

There are a couple of things that it might be worth checking:

1. Have you enabled the "Allow sender header" option for your mail server?

2. I don't see any Sender header in the body of your message. You'll need to add this to be an address from your domain.

I'm assuming the 503 error you get is about an invalid from address. Can you confirm this? You should see it in the smtp logs if PHPMailer isn't giving it to you.

[6temes]

Hi,

Thank you for your answer.

1. Yes.

2. There's no Sender because PHPMailer is using something called Envelope From. That means that we have two different FROM fields: The SMTP command MAIL FROM and then a From: inside of the DATA.

This forum answer explains it better than me:

https://community.mcafee.com/thread/31469?tstart=0

In this wikipedia page they explain it as well:

https://en.wikipedia.org/wiki/Email_spoofing#Technical_detail

What happens is that Postal only holds one FROM in the headers Hash. When it parses the message, the first FROM (with the valid email) is assigned to headers[:from] but, then, when it parses the DATA, it finds another from (with the spoofed email) and overwrites the headers[:from].

That's why the method find_authenticated_domain_from_headers(headers) fails.

[adamcooke]

MAIL FROM's value is not inserted into the headers anywhere. The value from MAIL FROM is actually not used at all by Postal for verification.

To acheive what you want with Postal, you need to add a Sender header to the actual outgoing message manually. The From header can remain the same.

[6temes]

MAIL FROM is not inserted into the headers anywhere. The value from MAIL FROM is actually not used at all by Postal for verification.

This is my misunderstanding, then. I still don't fully understand the source code of postal.

In any case, my point is that using a different FROM in the envelope and in the body of the message is a legit way to spoof the email sender. This is commonly used by mailing lists and by PHPMailer (a widely used PHP library for sending emails). I am still researching about SMTP (complex world!), but I understand that PHPMailer designers had a reason to implement the email spoofing this way.

[adamcooke]

Other services may automatically add a Sender header based on the contents of MAIL FROM. I haven't researched this though.

You might want to take a look at the Postal PHP client which will handle this correctly for Postal.
https://github.com/atech/postal-php

[catphish]

Traditionally, SPAM filtering (SPF and DKIM in particular) was based almost entirely around the MAIL FROM (envelope address) of a message. Using a different MAIL FROM vs From: was indeed a very effective way to send an email and make it appear to come from someone else. However, due to the increasingly aggressive nature of SPAM filtering, and people's desire to block spoofed email messages, this has become increasingly unsuitable.

There is a solution to this problem that appeases the majority of SPAM filters: Set the From header as you wish, but add an additional header Sender that describes the domain that really sent the message.

In order to improve deliverability, Postal will automatically set the MAIL FROM address to match the domain found in either the From or the Sender header, and requires that your domain name appears in one of these headers.

In your case, I hope it should be easy to add a Sender header to the messages.

We could potentially improve compatibility with clients that don't add Sender headers by injecting a Sender header with the contents of the MAIL FROM command in cases where the From address doesn't match a known domain, but the MAIL FROM command does.

Postal also has a feature to disable the From / Sender header checking. This is currently not exposed in the UI and we would generally strongly discourage its use when adding a Sender header is fairly easy. See #133

[catphish]

I've created #134 with the suggestion above, and there's also #133 as a simpler but less effective alternative.

[afanjul]

Hi @6temes, we was using sendgrid with phpmailer as you ^_^, and now we are switching to postal (again as you 😄 ), and finally we are sending emails on behalf of our users (wtf! 🤣 ) hahaha. You can send the emails with phpmailer and postal using this:

$mailer = new PhpMailer();
....
$mailer->Sender = $config['sender'];
$mailer->AddCustomHeader("Sender: " . $config['sender']); //just in case

That way PhpMailer include the sender in the mail headers and all its fine! ;)

[adamcooke]

@afanjul Looks good. Postal just needs to see that Sender header and all will be fine.

[6temes]

This is the good line:

$mailer->AddCustomHeader("Sender: " . $config['sender']);

I will try. Thanks!!!