Sendgrid-nodejs: Encoding issue in subject line

Created on 4 Oct 2016 · 16Comments · Source: sendgrid/sendgrid-nodejs

Issue Summary

In default Sendgrid is using ASCII encoding in subject lines. This causes issues when using non-ASCII characters in subject line. This can be fixed by using UTF-8 encoding in subject (e.g. http://ncona.com/2011/06/using-utf-8-characters-on-an-e-mail-subject/). I think Sendgrid library should do utf-8 encoding by default so user don't need to worry about it.

Steps to Reproduce

Use Sendgrid library like this:
const helper = require('sendgrid').mail;
const subject = 'ÄäÖö';
const mail = new helper.Mail(fromEmail, subject, toEmail, content);

Check your inbox and you'll see something like this in subject line:
��

Technical details:

sendgrid-nodejs Version: 4.3.0
Node.js Version: 5.10.1

hacktoberfest help wanted bug

Source

omjokine

Most helpful comment

I'm still using 4.3.0 and it works, so I think it was fixed in backend side. But anyhow, it works now :)

omjokine on 21 Oct 2016

👍2

All 16 comments

Thanks for the issue report @omjokine! We have added it to our backlog for a fix.

thinkingserious on 4 Oct 2016

@shkabo @omjokine

Using version 4.5.0 of the library, I can not reproduce this issue:

var helper = require('sendgrid').mail;
var sg = require('sendgrid')(process.env.SENDGRID_API_KEY);

var mail = new helper.Mail();
var email = new helper.Email('[email protected]', 'Elmer Thomas');
mail.setFrom(email);

mail.setSubject('ÄäÖö');

var personalization = new helper.Personalization();
email = new helper.Email('[email protected]', 'Elmer Thomas');
personalization.addTo(email);
mail.addPersonalization(personalization);

var content = new helper.Content('text/html', '<html><body>some text here</body></html>')
mail.addContent(content);

var request = sg.emptyRequest({
  method: 'POST',
  path: '/v3/mail/send',
  body: mail.toJSON(),
});

sg.API(request, function(err, response) {
  console.log(response.statusCode);
  console.log(response.body);
  console.log(response.headers);
});

Can you please help me reproduce so that I can verify @shkabo's fix?

Thanks!

thinkingserious on 11 Oct 2016

So I've tried to replicate this issue but can't figure out what's exactly happening. I'm going to point out some of my findings.
If i try to replicate this bug via node app as (added some latin chars also):

var helper = require('sendgrid').mail;
var dotenv = require('dotenv');
dotenv.load();
var from_email = new helper.Email('[email protected]');
var to_email = new helper.Email('[email protected]');
var subject = 'šđžčć ÄäÖö';
var content = new helper.Content('text/plain', 'šđžčć ÄäÖö');
var mail = new helper.Mail(from_email, subject, to_email, content);

var sg = require('sendgrid')(process.env.SENDGRID_API_KEY);
var request = sg.emptyRequest({
  method: 'POST',
  path: '/v3/mail/send',
  body: mail.toJSON(),
});

sg.API(request, function(error, response) {
  console.log(response.statusCode);
  console.log('--------------');
  console.log(response.body);
  console.log('--------------');
  console.log(response.headers);
});

Email is received as it was sent. Bit if I try to do this via cURL:

curl --request POST \
  --url https://api.sendgrid.com/v3/mail/send \
  --header 'Authorization: Bearer <API_KEY>' \
  --header 'Accept: application/jsonn' \
  --data '{"personalizations": [{"to": [{"email": "[email protected]"}]}],"from": {"email": "[email protected]"},"subject": "šđžčć ÄäÖö","content": [{"type": "text/plain", "value": "šđžčć ÄäÖö"}]}'

you get an error which points out to missing documentation

 573{"errors":[{"message":"Invalid UTF8 in request","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}

My best guess is that this issue might be on the API it self, not on the nodejs lib.
@thinkingserious can you review this ?

shkabo on 12 Oct 2016

This works for me:

curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
        -H "Authorization: Bearer $SENDGRID_API_KEY"  \
        -H "Content-Type: application/json" \
        -d '{"personalizations": [{"to": [{"email": "[email protected]"}]}],"from": {"email": "[email protected]"},"subject": "šđžčć ÄäÖö","content": [{"type": "text/plain", "value": "šđžčć ÄäÖö"}]}'

One thing I noticed with your cURL request is that application/json is misspelled.

thinkingserious on 12 Oct 2016

shkabo@DESKTOP-9KF0479 MINGW64 ~/Desktop/js/sendgrid
$ curl -X "POST" "https://api.sendgrid.com/v3/mail/send" \
>         -H "Authorization: Bearer <API_KEY>"  \
>         -H "Content-Type: application/json" \
>         -d '{"personalizations": [{"to": [{"email": "[email protected]"}]}],"from": {"email": "[email protected]"},"subject": "šđžčć ÄäÖö","content": [{"type": "text/plain", "value": "šđžčć ÄäÖö"}]}'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   345  100   156  100   189    416    504 --:--:-- --:--:-- --:--:--   576{"errors":[{"message":"Invalid UTF8 in request","field":null,"help":"http://sendgrid.com/docs/API_Reference/Web_API_v3/Mail/errors.html#-Encoding-Errors"}]}

@thinkingserious something's not right ...

shkabo on 12 Oct 2016

Hi @shkabo,

Just tried again with no error.

Maybe something is up with the encoding in your terminal? For reference, I'm using iTerm2. I also tried with Terminal and also did not get an error.

thinkingserious on 12 Oct 2016

@thinkingserious I've tested it again. If I run it in Git Bash on Win10 (MINGw64 terminal) I get that error, but when I run it from my laptop (Ubuntu 12.04) it works ...

shkabo on 12 Oct 2016

Ah, check this out: http://stackoverflow.com/questions/10651975/unicode-utf-8-with-git-bash

Looks like there is a setting you need to switch.

thinkingserious on 12 Oct 2016

Have you done some fixes to template subject encoding? I think I had earlier issues with encoding when I gave subject direct to the template settings from the web interface. Now it seems that subject is UTF-8 encoded as it should. Even when I'm using node.js library.

So I guess this is now fixed.

omjokine on 20 Oct 2016

@omjokine,

I believe this was fixed as of v4.5.0.

thinkingserious on 20 Oct 2016

I'm still using 4.3.0 and it works, so I think it was fixed in backend side. But anyhow, it works now :)

omjokine on 21 Oct 2016

👍2

Awesome, thanks for letting us know!

thinkingserious on 21 Oct 2016

Hello

I have a same problem with personalization template
================== my template for testing =========

<!DOCTYPE html>
<html>
<head>
    <title>CAGNOTTE</title>
  <meta charset="UTF-8">
</head>
<body>
  Bonjour {{nom}} ,
<br /><br/>
Test Cagnotte
<br /><br/>
Nous avons {{phrase}} votre compte.
<br /><br/>
A ce jour, le total de votre cagnotte  est de {{nbpointssolde}} points, soit {{nbeurossolde}} euros.
<br /><br/>
</body>
</html>

================ NODE JS =================

  const sgMail = require("@sendgrid/mail");
  const utf8=require('utf8');
  sgMail.setApiKey('xxxxxxxxxxxxx');
  sgMail.setSubstitutionWrappers("{{", "}}"); // Configure the substitution tag wrappers globally

  const msg = {
    from: "[email protected]",
    templateId: "d-32ff62aaeeef4897a89a3b68e598504d",
    personalizations: [
      {
        to: [
          {
            email: "[email protected]"
          }
        ],
        dynamic_template_data: {
          nom: "TOTO",
          prenom: "Olivier",
          nbpoints: "1250",
          nbeuros: "12,50",
          nbpointssolde: "6727",
          nbeurossolde: "67,27",
          phrase: "débité"  
        }
      }
    ]
  };
sgMail.send(msg)

=========== received mail ==============

Bonjour TOTO ,
Test Cagnotte
Nous avons d�bit� votre compte.
A ce jour, le total de votre cagnotte est de 6727 points, soit 67,27 euros.

===================================

Can you help me to resolve "d�bit�" ==> "débité"

thanks