Magento2: E-Mail subject not showing "ü" german Umlaute

[kemexyz]

Preconditions

1. Magento Version 2.2.4 installed via composer
2. PHP 7.0.30-0ubuntu0.16.04.1
   (3. My magento has this module https://github.com/magepal/magento2-gmail-smtp-app)

Steps to reproduce

1. Create a e-mail template containing a "ü" (üäö german stuff) in the subject
2. Create an Order to get the E-mail
3. View the email in outlook (web.de was also reported to me)

Expected result

1. E-mail subject should display the "ü" like it was authored

Actual result

1. E-mail subject shows "?" question mark in the same position as the "ü" should be

I resolved this issue by creating a plugin for Magento\Email\Model\Template method getSubject/afterGetSubject with utf8_encode($result) to UTF-8 encode the result.
This seems to be working for now.

Note for the module mentioned in the preconditions:
https://github.com/magepal/magento2-gmail-smtp-app
This module was installed before I upgraded to Magento 2.2.4 from 2.1 and there were no issues with the encoding of the subject line

Also the System does not retrieve the subject line when opening the email template.

[ymarc77]

I noticed this same problem… Any diacritics in the subject line is converted to "?".

[kemexyz]

@ymarc77 Tell as your Magento version please

[ymarc77]

Magento 2.2.4, noticed on 2.2.3 too.

[kemexyz]

Did you recently Upgrade to 2.2.X ?

The Magento instance I'm working on was only affected when I upgraded to 2.2.4 from 2.1.x

[ymarc77]

No. I migrated from 1.8.1 to 2.2.1, then updated to 2.2.2 and 2.2.3.

[kkrieger85]

mhlej18

[magento-engcom-team]

@kkrieger85 thank you for joining. Please accept team invitation here and self-assign the issue.

[kkrieger85]

@magento-engcom-team give me 2.2.4 instance

[magento-engcom-team]

Hi @kkrieger85. Thank you for your request. I'm working on Magento 2.2.4 instance for you

[magento-engcom-team]

Hi @kkrieger85, here is your Magento instance.
Admin access: https://i-15620-2-2-4.engcom.dev.magento.com/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[kkrieger85]

I cannot not reproduce this issue with Vanilla Magento 2.2.4

[kkrieger85]

E-Mail template:

Settings:

Order email (opened in Airmail3)

Order email (opened in Gmail, webmail)

[kkrieger85]

E-Mail source:

Subject: Für völligen Spaß Your Main Website Store order confirmation

[okorshenko]

Hi @kkrieger85 thank you for your research and investigation the issue

[ghost]

I'm experiencing the same thing.

I've recently updated from 2.1.9 to 2.1.12 then 2.20 and 2.2.5:

admin:

on my email:

Should be "Pedido de notificação"

Another issue I'm facing, every time I open the template the subject is gone.

[ymarc77]

I didn't find a solution, I'm disappointed with this error…

[rosen4o]

Same issue after update to 2.2.5
Some filter is run on this variable content but can't find which and how to prevent it
I have checked into the database and all data is stored correctly but it's not visualized on template edit and also on send correctly

[jaykobi]

Same issue after update to 2.2.5

This issue realy should be reopened! Mentioned also here https://github.com/magento/magento2/issues/8094

[jaykobi]

Until this is fixed ou can replace the Umlauts (or any special char, I guess) in the subject with the corresponding html entity. The subject then will appear in /admin/email_template/ as well.

[xpoback]

The problem is present in 2.2.5

The source of all evil is not in Magento itself but in Dotmailer module v2.5.0.

I've created an issue there.

I think it's important that MAGENTO makes sure that no modules it uses break any functionality.

[xpoback]

@kemexyz Why are you closing and opening the issue again and again?

[kemexyz]

@xpoback
sorry, first one was an honest miss click, second one was because I quickly got confused why this is open again – and then I had to reopen it. To be honest: I'm ashamed of myself, sorry again!

[xpoback]

@kemexyz no problem.

I've got a temporary fix for those who _don't use Dotmailer_.

Put this somewhere in a di.xml of any module:
(I've created a new one for this fix)

    <type name="Magento\Email\Model\Template">
        <plugin name="dotmailer_template_plugin" disabled="true"/>
    </type>

This disables the plugin that is the source of the problems.

For a permanent fix, let's wait until Dotmailer fix their module and Magento includes the fixed version to their package.

[ghost]

This issue is not reproducible on a clean Magento installation.

[xpoback]

@engcom-backlog-pb

It is reproducible on Magento installed through _composer_.

Surprisingly, Magento team only tests the version that is in this GitHub repository. This version does not contain Dotmailer, Amazon, Temando etc third part extensions, which cause tons of problems, because they don't get tested, but these modules come with default Magento.

Please, test the product you _actually_ offer the customers, not what you have here in some of the branches. Third party modules do belong to the product.

[Vinai]

Since the module is included in the core distribution, it is part of a "clean" Magento installation. Pretending otherwise is just not dealing with reality. Problems in bundled third party modules need to be treated like problems in any Magento core module.
The only real alternative I can think of would be removing the problematic bundled module from all distributions.

[ghost]

@xpoback First of all, there is no mentioning of _Composer_ in the issue report. And, secondary, we are not able to fix any bug that comes from the bundled extensions. Please contact developers of specific third-party modules.
@Vinai you always can disable any module you want.

[hostep]

@engcom-backlog-pb: sorry, please don't take this personal. And yes I realise this is not the right place to discuss this, but we already tried to start up a discussion about this on Slack yesterday, which was greeted by a lot of silence, so yeah ...

Anyway, you're making the wrong assumption here. Magento recommends people to install Magento using composer. Not by cloning the git repo. So your assumption should be reversed, you should always assume that people install Magento using composer and not the other way around, unless it is explicitly mentioned in the ticket.
So you guys should also always test with a composer based installation, which will sometimes make it easier to reproduce the issue being reported.
If it then turns out that the bug is actually caused by one of these bundled extensions, then make a mention of it and maybe also already notify those vendors about the problem. (We still don't know who or how to contact them btw)

You also say: "you can just disable those modules", and yes, that is very true, but people don't immediately realise that these problems are caused by those bundled extensions and not by core Magento. So you have to understand that tickets are being opened here in the Magento repo for such issues.
Also realise that a lot of Magento customers aren't super technical and don't really understand what modules are and if they can or cannot disable them. They see Magento as one big thing.

You guys also have to have some sort of responsibility, since you include those bundled extensions by default and enable those by default. So if you want Magento to be stable, you should also take some sort of responsibility over those modules you include, even if it is just merely passing info about bugs over to those vendors and keep the issue reporter updated of what's going on.

Please, try to discuss this internally, with your colleagues, with some managers, see if this makes sense, see what can be done about this to try to improve Magento even further. Always try to move forwards, ignoring big issues like these (not this actual issue, but the one I'm bringing up right now) is not going to move Magento forwards.
And if some kind of progress is made around this, it would be great if what is decided is then being communicated to the community by some sort of blogpost or something like that.

Thanks for reading and considering this!

[ymarc77]

Thanks, Pieter @hostep

[xpoback]

@engcom-backlog-pb Installation through composer is one of the ways described in Magento 2 Docs and even mentioned as recommended.
I already contacted the authors almost two months ago and created an issue in their repository and even gave the exact place in code where the bug appears.
This is not just about this issue, sadly, it's a general problem of most bundled extensions, that no-one takes responsibility. Let's hope this is going to change in the next version.

[MarcusWolschon]

Same issue with Magento 2.2.6 installed via Composer on a System with a system-locale using UTF8 on a UTF8 database.
The broken Subject is shown in the email and in the admin web interface (list view, sent as text/html with UTF-8 as character encoding).

[zaietsv]

The same issue at version 2.2.6 http://screenshots.collabstar.com/vza/df6f2ab5.png

[zaietsv]

@kemexyz no problem.

I've got a _temporary_ fix for those who _don't use Dotmailer_.

Put this somewhere in a di.xml of any module:
(I've created a new one for this fix)

    <type name="Magento\Email\Model\Template">
        <plugin name="dotmailer_template_plugin" disabled="true"/>
    </type>

This disables the plugin that is the source of the problems.

For a permanent fix, let's wait until Dotmailer fix their module and Magento includes the fixed version to their package.

Yes, you are quite right: the Dotmailer plugin decodes a gorgeous utf-8 string to a terrible creature by utf8_decode() function http://screenshots.collabstar.com/vza/dd9f6cc8.png It is necessary to rewrite or to disable private function getProcessedTemplatePreviewAndOther() and private function getResultIfArgsEmptyForPreviewAndOther()
There are some other places in Magento with utf8_decode() to be reviewed to.

[kemexyz]

squashtoberfest

[magento-engcom-team]

@kemexyz thank you for joining. Please accept team invitation here and self-assign the issue.

[magento-engcom-team]

Hi @kemexyz. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 5. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 6. Add label Issue: Confirmed once verification is complete.

• [ ] 7. Make sure that automatic system confirms that report has been added to the backlog.

[kemexyz]

@magento-engcom-team give me 2.3-develop instance

[magento-engcom-team]

Hi @kemexyz. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @kemexyz, here is your Magento instance.
Admin access: https://i-15620-2-3-develop.engcom.dev.magento.com/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[hoangnm89]

i'm using 2.2.5, and got the same issue. in my case, the email has no subject:

update: Subject i set is: "Order Mới của bạn đây nhé...."

[magento-engcom-team]

@dmytro-ch Thank you for verifying the issue. Based on the provided information internal tickets MAGETWO-97392 were created

[milindsingh]

20384

[rahul-kachhadiya]

Ready to work on this issue at #mm19in

[magento-engcom-team]

Hi @rahul-kachhadiya. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if your want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 3. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

[rahul-kachhadiya]

@magento-engcom-team give me 2.3-develop instance

[magento-engcom-team]

Hi @rahul-kachhadiya. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @rahul-kachhadiya, here is your Magento instance.
Admin access: https://i-15620-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[magento-engcom-team]

Hi @ananth-iyer. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento-engcom-team give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!
  

• [ ] 3. Verify that the issue is reproducible on 2.2-develop branch.

  Details

  - Add the comment @magento-engcom-team give me 2.2-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.2-develop branch, please add the label Reproduced on 2.2.x

• [ ] 4. If the issue is not relevant or is not reproducible any more, feel free to close it.

[ananth-iyer]

@magento-engcom-team give me 2.3-develop instance

[magento-engcom-team]

Hi @ananth-iyer. Thank you for your request. I'm working on Magento 2.3-develop instance for you

[magento-engcom-team]

Hi @ananth-iyer, here is your Magento instance.
Admin access: https://i-15620-2-3-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[ananth-iyer]

@magento-engcom-team This issue requires email sending service enabled. Can you help me with this?

[ananth-iyer]

@magento-engcom-team give me 2.2-develop instance

[magento-engcom-team]

Hi @ananth-iyer. Thank you for your request. I'm working on Magento 2.2-develop instance for you

[magento-engcom-team]

Hi @ananth-iyer, here is your Magento instance.
Admin access: https://i-15620-2-2-develop.instances.magento-community.engineering/admin
Login: admin Password: 123123q
Instance will be terminated in up to 3 hours.

[ananth-iyer]

@magento-engcom-team @kemexyz This issue is only in 2.2-develop instance. While testing on local 2.2 instance I found that MySQL is returning correct german character ü but Magento PHP code is converting "ü" to "?". So will check which file is required for 2.2 version when I get free.

[zaietsv]

@magento-engcom-team @kemexyz This issue is only in 2.2-develop instance. While testing on local 2.2 instance I found that MySQL is returning correct german character ü but Magento PHP code is converting "ü" to "?". So will check which file is required for 2.2 version when I get free.

You need only to update a Dotmailer module to last versions. They have declared a fix six months ago.

[zaietsv]

Hereby they've released the fix for this in v3.0.2 with this commit: 635a62c https://github.com/dotmailer/dotmailer-magento2-extension/issues/517#issuecomment-436964065

[ananth-iyer]

@zaietsv Thank you for pointing it out the release commit. But I tried to update Dotmailer but I got composer error:

After that I copied the fixes in Magento 2.2 then edited email template on admin & tested email send. And found it showing german letter in email subject & body.
@kemexyz If you are unable to update Dotmailer then I would suggest upgrade to Magento 2.3.0 OR you can check other options to get fixes on your Magento 2.2.

EDITED
Because this issue is from third-party extension Dotmailer version 2.6.0 & If you are not using this extension then can disable it right away. So it will fix german letter issue.