Magento2: Magento Framework Escaper - Critical log with special symbols

[ihor-sviziev]

Preconditions

1. Magento 2.3.3

Steps to reproduce

Case 1

1. Register a new customer from frontend;
2. Created one order from frontend;
3. Add & to store view name
4. Go to Admin >> Stores >> All Stores
5. Click on "Default Store View"
6. Change Name field to "Default & Store View"
7. Click "Save Store View" button
8. Remove all log files from magento2/var/log directory (in order to have only needed logs)
9. Go to Frontend >> [user name] >> My Account >> My Orders;
10. Click on "View Order" link for our order that was created in step 2;
11. See magento2/var/log directory;

Expected result

1. No exception logs should be there

Actual result

1. exception.log file appeared with one line:

[[2019-12-03 10:47:42] main.CRITICAL: DOMDocument::loadHTML(): Tag date invalid in Entity, line: 1 {"exception":"[object] (InvalidArgumentException(code: 2): DOMDocument::loadHTML(): Tag date invalid in Entity, line: 1 at /home/arthur/sites/magento/magento233/vendor/magento/framework/Escaper.php:89)"} []

Case 2

1. Register a new customer from frontend;
2. Created one order from frontend;
3. Go to Admin->Sales->Orders;
4. Click on the "View" link for our order that was created in step 2;
5. In "Order Total" block add comment "&";
6. Click on the "Submit Comment" button;
7. Remove all log files from magento2/var/log directory (in order to have only needed logs)
8. Go to Frontend >> [user name] >> My Account >> My Orders;
9. Click on "View Order" link for our order that was created in step 2;
10. See magento2/var/log directory;

Expected result

1. No exception logs should be there

Actual result

1. exception.log file appeared with one line:

[[2019-12-03 10:55:22] main.CRITICAL: DOMDocument::loadHTML(): Tag date invalid in Entity, line: 1 {"exception":"[object] (InvalidArgumentException(code: 2): DOMDocument::loadHTML(): Tag date invalid in Entity, line: 1 at /home/arthur/sites/magento/magento233/vendor/magento/framework/Escaper.php:89)"} []

Reason: On View order page in the admin we have "Purchased From" block.

This block uses \Magento\Framework\Escaper::escapeHtml with string that contains ampersand and allowed tags is "br". In this case it writes critical log. This issue appeared in Magento 2.2.0, it was introduced in https://github.com/magento/magento2/commit/59c2c9e8dd89e6452a5a861595bc88b8ebd9c5f3 + https://github.com/magento/magento2/commit/df261e75bd8ef839139da78f3c0d8bb14a0747b4 + https://github.com/magento/magento2/commit/624ee867d568fff6aa4badf46a869506b3350baf.

_For someone who will fix this issue:_
I prepared tests for this fix in my branch that are currently fails (not only this case):
https://github.com/magento/magento2/compare/2.2-develop...ihor-sviziev:escaper-critical-log-when-ampersand-is-present

[magento-engcom-team]

@ihor-sviziev, thank you for your report.
We've acknowledged the issue and added to our backlog.

[oleksandr-butenko]

hi
We have the same issue with comments in order.
The version of Magento is 2.2.5
Admin user can write an order comment with "&" for example.
Then, during load of the order page, we will have errors about using &.

Have you fixed this issue in one the next releases?
Maybe there is a patch to fix it?

Thanks.

[simonmaass]

have the same issue in 2.2.6

[GuiltyNL]

Any news on this one? As this error floods our logfile.

[jhart-efaktory]

I just did some testing and the problem seems to come from the \Magento\Framework\Escaper::escapeHtml.

The line 74 should convert the data to htmlentities

$string = mb_convert_encoding($data, 'HTML-ENTITIES', 'UTF-8');

Edit: I just found in the documentation of mb_convert_encoding that it won't escape the following characters: '\'', '"', '<', '>', or '&'. I will look at how this function evolved since 2.1.x

[dan-ding]

change:
$string = mb_convert_encoding($data, 'HTML-ENTITIES', 'UTF-8');
to:
$string = mb_convert_encoding(preg_replace('/&/', '&', html_entity_decode($data)), 'HTML-ENTITIES', 'UTF-8');

in 2.3: https://github.com/magento/magento2/blob/b1d3061537951f5442c2c185947795544aa8d57d/lib/internal/Magento/Framework/Escaper.php#L94

in 2.2:
https://github.com/magento/magento2/blob/d8267c249fdf82bfdf0a35a21b0e6dba02a52838/lib/internal/Magento/Framework/Escaper.php#L80

i think will cover most cases. if there you have codepoints that html_entity_decode doesn't handle, then i think you'll need your own lookup table to replace that.

[ihor-sviziev]

This issue was fixed in 2.3-develop branch in following commits:

• https://github.com/magento/magento2/commit/847d20f41541c65df9cc794c18be1e70112ebeb6
• https://github.com/magento/magento2/commit/fa95063d2f0ef98c7a8377f087e442d391b29273

Probably it will be included into release 2.3.1.

[GuiltyNL]

This ticket had been closed? It isn't the intention to fix this in 2.2?

[ihor-sviziev]

(From https://github.com/magento/magento2/pull/19884#issuecomment-451112512):
This issue was fixed in 2.2-develop branch in following commits:

• https://github.com/magento/magento2/commit/15c7b1ba6db51af92b77896c547d2bc0a60c64a6

Probably it will be included into release 2.2.8.

[robyerevan]

not fixed.
Reproduced in 2.3.1

[GuiltyNL]

Also reproduced in 2.3.3

[ihor-sviziev]

Hi @engcom-Alfa,
People still reporting that issue reproducing. Could you double check if this issue reproducing on 2.3-develop?

[engcom-Alfa]

Hi @ihor-sviziev,
Ok, I'll check it.

[m2-assistant[bot]]

Hi @engcom-Alfa. Thank you for working on this issue.
In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
  Details

  If the issue has a valid description, the label Issue: Format is valid will be added to the issue automatically. Please, edit issue description if needed, until label Issue: Format is valid appears.

• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue. If the report is valid, add Issue: Clear Description label to the issue by yourself.

• [ ] 3. Add Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 4. Verify that the issue is reproducible on 2.3-develop branch

  Details

  - Add the comment @magento give me 2.3-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.3-develop branch, please, add the label Reproduced on 2.3.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

• [ ] 5. Add label Issue: Confirmed once verification is complete.

• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[engcom-Alfa]

@ihor-sviziev Unfortunately, we are not able to reproduce this issue on fresh 2.3-develop and 2.3.3 instances.

Manual testing scenario:

1. Create order from frontend;

2. Add & to store view name

• Go to Admin >> Stores >> All Stores

• Click on "Default Store View"

• Change the Name field to "Default & Store View"

• Click "Save Store View" button

1. Remove all log files from magento2/var/log directory;

2. Go to Admin >> Sales >> Orders >> Click to "View" for our order;

3. See magento2/var/log directory;

Actual Result: No exceptions in the log file.

So, I have to close this issue.

Thank you!

[gerben86]

@ihor-sviziev Unfortunately, we are not able to reproduce this issue on fresh 2.3-develop and 2.3.3 instances.

Manual testing scenario:

1. Create order from frontend;
2. Add & to store view name

• Go to Admin >> Stores >> All Stores
• Click on "Default Store View"
• Change the Name field to "Default & Store View"
• Click "Save Store View" button

1. Remove all log files from magento2/var/log directory;
2. Go to Admin >> Sales >> Orders >> Click to "View" for our order;
3. See magento2/var/log directory;

Actual Result: No exceptions in the log file.

So, I have to close this issue.

Thank you!

Did you also check it with this instruction?: https://github.com/magento/magento2/issues/13269#issuecomment-412502534
Because that makes more sense then updating the name of the store.

[engcom-Alfa]

@gerben86 Yes, of course. We also tested this scenario and no exceptions occurred. Thanks!

[GuiltyNL]

@engcom-Alfa It can be reproduced by adding an order comment with just: "&". Same exception will show up in the logfile.

[magento-engcom-team]

:white_check_mark: Confirmed by @engcom-Alfa
Thank you for verifying the issue. Based on the provided information internal tickets MC-29328 were created

Issue Available: @engcom-Alfa, _You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself._

[arendarenko]

I can confirm that issue is still happening on Magento 2.3.4

[ihor-sviziev]

Hi @engcom-Alfa,
Looks like this issue was fixed in https://github.com/magento/magento2/pull/25895
Could you double check if issue still there?

[ihor-sviziev]

@magento give me 2.4-develop instance

[magento-engcom-team]

Hi @ihor-sviziev. Thank you for your request. I'm working on Magento 2.4-develop instance for you

[magento-engcom-team]

Hi @ihor-sviziev, here is your Magento instance.
Admin access: https://i-13269-2-4-develop.instances.magento-community.engineering/admin_57d3
Login: 9d102703 Password: f7abb8f53530
Instance will be terminated in up to 3 hours.

[m2-assistant[bot]]

Hi @ihor-sviziev. Thank you for working on this issue.
Looks like this issue is already verified and confirmed. But if you want to validate it one more time, please, go though the following instruction:

• [ ] 1. Add/Edit Component: XXXXX label(s) to the ticket, indicating the components it may be related to.

• [ ] 2. Verify that the issue is reproducible on 2.4-develop branch

  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and _stop verification process here_!

• [ ] 3. If the issue is not relevant or is not reproducible any more, feel free to close it.

---

• Join Magento Community Engineering Slack and ask your questions in #github channel.

[ihor-sviziev]

I can confirm - the issue was fixed in 2.4-develop branch by following changes: https://github.com/magento/magento2/pull/25895

I'm closing this issue