Sendgrid-python: Batch sending of emails not working

[jeffoneill]

Hello,

I'm testing my code for the latest version of the library. When I send emails one at a time (one per API call), it works fine. When I send multiple emails in an API call, only one of the emails is received.

Below is an excerpt of JSON sent for a batch of 10 emails (minimally edited to remove personal info). I'm sending to my personal gmail account using the + notation, e.g., [email protected] through [email protected].

One extra weirdness is that the Sendgrid email activity log says that all 10 emails are delivered even though they are not. I find it very strange that this works sending individually (to all 10 email addresses) but not in batches of 10 at once.

I'd greatly appreciate any help you can provide.

Jeff

{'categories': ['begin'],
 'custom_args': {'eid': '5715143556071424'},
 'from': {'email': '[email protected]'},
 'personalizations': [{'custom_args': {'vkey': 'aglzfm'},
                       'send_at': 1486243132,
                       'substitutions': {'{{description}}': u'<p>xgfddfdfhdf</p>',
                                         '{{item_title}}': u'Best Cat',
                                         '{{manager_email}}': u'[email protected]',
                                         '{{manager_name}}': u"Jeff O'Neill",
                                         '{{unsubscribe_link}}': u'https://www.example.com/remove/',
                                         '{{vote_link}}': u'https://www.example.com/vote/'},
                       'to': [{'email': u'[email protected]'}]},
                       [...]
                      {'custom_args': {'vkey': 'aglzfm'},
                       'send_at': 1486243141,
                       'substitutions': {'{{description}}': u'<p>xgfddfdfhdf</p>',
                                         '{{item_title}}': u'Best Cat',
                                         '{{manager_email}}': u'[email protected]',
                                         '{{manager_name}}': u"Jeff O'Neill",
                                         '{{unsubscribe_link}}': u'https://www.example.com/remove/',
                                         '{{vote_link}}': u'https://www.example.com/vote/'},
                       'to': [{'email': u'[email protected]'}]}],
 'reply_to': {'email': u'[email protected]'},
 'subject': u'Best Cat',
 'template_id': '6060c6e9-0376-4b52-ac15-e6befd906072'}

[thinkingserious]

Hello @jeffoneill,

It looks like you are trying to achieve this use case: https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/personalizations.html#-Sending-Two-Different-Emails-to-Two-Different-Groups-of-Recipients

At first glance, your payload looks correct.

I will need to dig in and try to reproduce. To that effect, I'm adding this to our backlog for further investigation.

With Best Regards,

Elmer

[jeffoneill]

Here is a simpler way to look at the issue.

When a single API call has multiple personalizations that send emails to the same gmail account, the gmail account will receive only one of the emails, even though the emails being sent are different from one another.

Consider this example:

data = {
    'from': {'email': '[email protected]'},
    'personalizations': [
        {'substitutions': {'{{description}}': u'<p>first</p>'},
         'to': [{'email': u'[email protected]'}]},
        {'substitutions': {'{{description}}': u'<p>second</p>'},
         'to': [{'email': u'[email protected]'}]},
        {'substitutions': {'{{description}}': u'<p>third</p>'},
         'to': [{'email': u'[email protected]'}]}
        ],
    'reply_to': {'email': u'[email protected]'},
    'subject': u'Best Cat',
    'template_id': '6060c6e9-0376-4b52-ac15-e6befd906072'}
response = SG.client.mail.send.post(request_body=data)

Three emails should be sent to the same gmail account. The three emails are different because they each have different substitutions. However, only one of these three emails will be delivered.

Only one email is delivered even if the email addresses are different (but still end up at the same gmail account). For example, changing them to [email protected], [email protected], and [email protected] still results in a single email being delivered.

I suspect that Google is doing something here. It makes sense for Google to discard identical emails, but here the emails are different from each other.

[thinkingserious]

@jeffoneill,

Thanks for the additional information, that should help me reproduce.

[jeffoneill]

I suspect that the problem is that Sendgrid is using the same SMTP-ID for all of the personalizations. Gmail is receiving multiple emails to the same account with the same SMTP-ID, assumes that they are the same email, and delivers only one.

Below is an example of delivery metadata reported by Sendgrid and you can see that the messages all have the same SMTP-ID.

EMAIL [email protected]
REASON 250 2.0.0 OK 1486478238 n5si11909140iof.45 - gsmtp 
SMTP-ID <[email protected]>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.2

EMAIL [email protected]
REASON 250 2.0.0 OK 1486478238 b20si11895501ioe.107 - gsmtp 
SMTP-ID <[email protected]>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.0

EMAIL [email protected]
REASON 250 2.0.0 OK 1486478238 g140si7310879itg.69 - gsmtp 
SMTP-ID <[email protected]>
PROCESSED STRING February 7, 2017 - 09:37:18 AM
MSGID srIZMvKuRHOuGd0rHM93Eg.filter0807p1mdw1-6811-5899DB9B-43.1

[thinkingserious]

Hello @jeffoneill,

This is an issue where you might want to dig in with our support team, until this ticket reaches the top of my queue. They can be reached at https://support.sendgrid.com.

BTW, I ran this quick test, and all three emails were delivered:

curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
        -H "Authorization: Bearer $SENDGRID_API_KEY"  \
        -H "Content-Type: application/json" \
        -d @post.json

post.json

{
    "from": {
        "name": "Example User",
        "email": "[email protected]"
    },
    "personalizations": [
        {
            "to": [
                {
                    "email": "[email protected]"
                }
            ],
            "subject": "Test Subject1",
            "substitutions": {
                "-name-": "Elmer"
            }
        },
        {
            "to": [
                {
                    "email": "[email protected]"
                }
            ],
            "subject": "Test Subject2",
            "substitutions": {
                "-name-": "Robbin"
            }
        },
        {
            "to": [
                {
                    "email": "[email protected]"
                }
            ],
            "subject": "Test Subject3",
            "substitutions": {
                "-name-": "Thomas"
            }
        }
    ],
    "content": [
        {
            "type": "text/plain",
            "value": "Hello -name-"
        },
        {
            "type": "text/html",
            "value": "Goodbye -name-"
        }
    ]
}

[jeffoneill]

Elmer,

Thanks for the feedback and trying that out.

Is your Sendgrid email based on gmail (G suite or whatever they call it
now)?

Jeff

Jeff O'Neill | Founder and Director
jeff.[email protected]

On Wed, Feb 8, 2017 at 1:56 PM, Elmer Thomas [e-mail]
wrote:

Hello @jeffoneill https://github.com/jeffoneill,

This is an issue where you might want to dig in with our support team,
until this ticket reaches the top of my queue. They can be reached at
https://support.sendgrid.com.

BTW, I ran this quick test, and all three emails were delivered:

curl -X "POST" "https://api.sendgrid.com/v3/mail/send"
-H "Authorization: Bearer $SENDGRID_API_KEY"
-H "Content-Type: application/json"
-d @post.json

post.json

{
"from": {
"name": "Example User",
"email": "[email protected]"
},
"personalizations": [
{
"to": [
{
"email": "elmer.[email protected]"
}
],
"subject": "Test Subject1",
"substitutions": {
"-name-": "Elmer"
}
},
{
"to": [
{
"email": "elmer.[email protected]"
}
],
"subject": "Test Subject2",
"substitutions": {
"-name-": "Robbin"
}
},
{
"to": [
{
"email": "elmer.[email protected]"
}
],
"subject": "Test Subject3",
"substitutions": {
"-name-": "Thomas"
}
}
],
"content": [
{
"type": "text/plain",
"value": "Hello -name-"
},
{
"type": "text/html",
"value": "Goodbye -name-"
}
]
}

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-python/issues/294#issuecomment-278426210,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKKSAIsGWpJHqEq4oWe8d27WhsyH59Nks5rag_5gaJpZM4L4lAk
.

[thinkingserious]

Yes. I also tried sending to my personal GMail account: https://www.screencast.com/t/MmI1KW8Izsb

[jeffoneill]

I'll follow up with support.

Your example also worked for me, but if you change the emails to all have
the same subject then only one is delivered.

Jeff O'Neill | Founder and Director
jeff.[email protected]

On Wed, Feb 8, 2017 at 2:03 PM, Elmer Thomas [e-mail]
wrote:

Yes. I also tried sending to my personal GMail account:
https://www.screencast.com/t/MmI1KW8Izsb

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-python/issues/294#issuecomment-278428199,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKKSLly0fsccy__1mSrJwCAh9hVUTtMks5rahGZgaJpZM4L4lAk
.

[mbernier]

@jeffoneill This could actually be gmail protecting you, rather than our pipeline doing something weird. Our received event is actually informed by the recipient email server. Gmail will thread emails based on a couple things like subject, reply-to, and others. So they may be just "protecting" you from multiple of the same emails that were all received by them at relatively the same instant.

This actually saves them from having to write multiples of the same email to disk (considering the volume of emails they receive this is extremely smart) and it will keep the recipients from complaining to gmail, or marking a legit sender's emails as spam when they made a simple mistake. (I am not saying you made a mistake, just that in a normal sending situation this would likely happen as a result of a mistake).

If you send one at a time, a second or so apart - you should receive all of them.
Also, if you send to multiple addresses you should see all of them show up.

If these two use cases do not produce the correct result, please reach out to our support team ASAP.

[jeffoneill]

Hi Matt,

That makes sense if the content of the emails is the same, but here the
content of the emails is different. I am trying to send two different
emails to the same user, but gmail is only delivering one of them.

I also tried including the "send_at" parameter and sending the emails at 10
second intervals (with a single API call from me to Senggrid) and Google
still only delivered one of them.

If I send the emails at essentially the same time using a different API
call for each, they all get delivered.

The difference between batch sending and individual sending (from me to
Sendgrid) appears to be that Sendgrid reuses the SMTP ID for batch sends.
Google turns the SMTP ID into the MESSAGE ID of the email, and these need
to be unique but they are not...

I appreciate all your input and I'm looking forward to hearing back from
your support team.

Jeff

On Wed, Feb 8, 2017 at 5:40 PM, Matt Bernier [e-mail]
wrote:

@jeffoneill https://github.com/jeffoneill This could actually be gmail
protecting you, rather than our pipeline doing something weird. Our
received event is actually informed by the recipient email server. Gmail
will thread emails based on a couple things like subject, reply-to, and
others. So they may be just "protecting" you from multiple of the same
emails that were all received by them at relatively the same instant.

This actually saves them from having to write multiples of the same email
to disk (considering the volume of emails they receive this is extremely
smart) and it will keep the recipients from complaining to gmail, or
marking a legit sender's emails as spam when they made a simple mistake. (I
am not saying you made a mistake, just that in a normal sending situation
this would likely happen as a result of a mistake).

If you send one at a time, a second or so apart - you should receive all
of them.
Also, if you send to multiple addresses you should see all of them show up.

If these two use cases do not produce the correct result, please reach out
to our support team ASAP.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-python/issues/294#issuecomment-278485804,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKKSCvNvdue3fFkiwQ8h6iXlZmBv22mks5rakRAgaJpZM4L4lAk
.

[mbernier]

I reached out to our deliverability folks to see if they have seen this behavior with Gmail before.

I will let you know as soon as I have an answer.

[thinkingserious]

Hello @jeffoneill,

All of our research points to this being an issue on GMail's side. I recommend you reach out to their support team for further assistance.

Please let us know if there is anything else we can help with.

With Best Regards,

Elmer

[jeffoneill]

Hi Elmer,

I appreciate you all taking the time to look into this, but from my
perspective, this really seems like a Sendgrid issue and not a Gmail issue.

(1) If I send two emails with two separate API calls at virtually the
same time, both get delivered.
(2) If I send the exact same two emails in a single API call, only one
gets delivered.

For both cases, the data I send to you is the SAME but the data you send to
Gmail is DIFFERENT and that causes one email to not be delivered!!!

This seems like actual mistake on your part that causes email to not be
delivered, and since email delivery is a key metric, I don't understand why
you won't fix this.

I know I'm a tiny customer so I shouldn't be taking up too much of your
time on this, but this does affect all your customers so I hope you
consider this issue more before closing it.

Thanks again, I do love your product overall.

Jeff

On Wed, Feb 15, 2017 at 12:21 PM, Elmer Thomas [e-mail]
wrote:

Hello @jeffoneill https://github.com/jeffoneill,

All of our research points to this being an issue on GMail's side. I
recommend you reach out to their support team for further assistance.

Please let us know if there is anything else we can help with.

With Best Regards,

Elmer

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/sendgrid/sendgrid-python/issues/294#issuecomment-280076539,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKKSIACK4imcNBSRKJP-2WwxzpAJMk7ks5rczQXgaJpZM4L4lAk
.

[thinkingserious]

Hi @jeffoneill,

Thanks for the additional detail. I'm afraid our team is not equipped to dig deeper with you on this one, as our focus is on our SDKs and documentation.

Could you please continue this conversation with our support team? They can be reached at https://support.sendgrid.com

Also, thanks for the kind words!

With Best Regards,

Elmer

[vivekshekar]

@jeffoneill @thinkingserious
Hey guys, Did you guys able to find out the cause for the this issue?
I'm having similar issue with sendgrid. When I send an email over batch and yes the contents are same but the emails with different testing email id's will get the email only once on gmail.

Please let me know if you have any thoughts on this.

Thank you,

[jeffoneill]

Hi Vivek,

My solution was to append a short text slug of random characters to the end of each subject using the Python snippet below. Google will deliver all the emails if the subjects are unique.

This is not an ideal solution, but it works for now. I may switch to another provider at some point if Sendgrid doesn't provide a better fix.

Jeff

chars = string.ascii_letters + string.digits
slug = ''.join(random.choice(chars) for i in range(5))
subject = "%s [%s]" % (subject, slug)

[thinkingserious]

Thanks for providing a work around @jeffoneill!

Unfortunately, this issue is out of our hands, the fix would have to be on GMail's side.