Magento2: M2.1.2 Character encoding issue with widgets including in WYSIWYG editor

[kanduvisla]

When I try to insert a widget with the WYSIWYG edit, the characters get encoded wrong when the WYSIWYG editor is enabled.

For example, Fóó Bär becomes: Fóó Bär.

This only happens when I try to insert a widget using the WYSIWYG-editor. If the WYSIWYG-editor is disabled, the content is correct:

[sakisplus]

Check this out #5548

[dthampy]

@kanduvisla , can you check if the solution/fix from #5548 resolves your issue?

[redelschaap]

We have the same issue; #5548 does not fix the issue.

[veloraven]

@kanduvisla thank you for your report.
Your video is really cool and visual. But could you please format this issue according to the Issue reporting guidelines: with steps to reproduce, actual result and expected result: it is much easier to understand the problem itself, how to reproduce it, and to search for duplicates, when the description is formatted.

[redelschaap]

I formatted the issue to the right format for @kanduvisla (just because we need this to be fixed very soon).

Preconditions

1. Running Magento 2.1.2
2. Running on PHP 7 with nginx
3. Tested on Chrome 54.0.2840 and Firefox 29.0.1

Steps to reproduce

1. Edit or add CMS page
2. Make sure the visual editor is turned off and click `Insert Widget...
3. Select a widget and enter text with special characters
4. Click the Insert Widget button
5. The text has been inserted with special characters encoded correctly
6. Clean the content, turn on the visual editor and insert a widget
7. Select a widget and enter text with special characters
8. Click the Insert Widget button
9. The text has been inserted with special characters encoded incorrectly

Expected result

The text should always be inserted with special characters encoded correctly

Actual result

The text is only inserted with special characters encoded correctly when the visual editor is turned off. When it's turned on, special characters are encoded incorrectly while inserting a widget.

See also @kanduvisla's visual presentation.

[redelschaap]

Is there any news or an ETA for this bug? @veloraven maybe?

[dthampy]

@redelschaap , We have an internal ticket created for this issue:MAGETWO-62296, which needs to be prioritized

[redelschaap]

Is there an ETA for this bug? Our client needs to go live with a German store very soon. @dthampy maybe?

[gabs77]

Same Magento 2.1.3
Example text : En plus d'agir en tant que grossiste aux îles Saint-Pierre et Miquelon.

vendor/magento/framework/Data/Form/Element/AbstractElement.php (line 302) _escape function with htmlspecialchars seem to be the problem

[redelschaap]

There is a commit that fixes this (partly): #4232. In my case I had to adjust something in PHP as well to prevent #8130 charachters from being inserted as HTML encoded entities in widget input fields.

[koenner01]

@redelschaap, nice catch on that #4232. Fix worked like a charm for us in 2.1.3!

[magento-engcom-team]

@kanduvisla, thank you for your report.
We were not able to reproduce this issue by following the steps you provided. If you'd like to update it, please reopen the issue.
We tested the issue on 2.3.0-dev, 2.2.0, 2.1.9

[magento-engcom-team]

@kanduvisla, thank you for your report.
We were not able to reproduce this issue by following the steps you provided. If you'd like to update it, please reopen the issue.
We tested the issue on 2.3.0-dev, 2.2.0, 2.1.9