Magento2: Grid pulls all data and doesnt have proper client/server pagination

[joshspivey]

Have over 5,000 products or orders and the grid wont load. I have over 50,000 orders imported from magento 1 and I get XML error. I saved the xml file to the server that it was generating on.
/magento/lib/internal/Magento/Framework/View/TemplateEngine/Xhtml/Template.php

It was saving all the orders into one xml file for the entire grid. Making a 25mb file taking minutes to download.

It needs proper server side pagination here is an example here.
https://github.com/backbone-paginator/backbone.paginator

Preconditions

1. php 7.1w
2. Percona 5.6
3. Magento 2.1.3

Steps to reproduce

1. have allot of orders or products.
2. Open the grid

Expected result

1. should load only what's in the grid not all records into one file.

Actual result

1. [Screenshot, logs]

[joshspivey]

I am making a plugin that makes a new sales grid using backgrid.

I should be done with this by Jan 23rd.

Repo is here https://github.com/joshspivey/magento2-sales-grid

[joshspivey]

I got it working and it definitely is allot faster then the stock grid I will upload soon I just wanted to finish the search and do a admin page to select the columns based off order data. It pages, and sorts perfect in server side mode. I should also add the actions into it as well.

[DRAJI]

What was the solution for this issue?

[orlangur]

That's interesting, one of the main problems new grid aimed to solve was O(n) computational complexity of grid in Magento 1 caused by passing all product ids to filter (https://github.com/magento/magento2/issues/47).

Dunno how it turned out into O(n) XML configuration. @vkorotun, any ideas?

[johnvinyl]

Hi, I am new o Magento, I went with a good hosting provider but seems they are limited to what they will help me with.

I created a configurable product with 3 attributes, 1 for body_type, 1 for head_type and other for colour which is just white for now, the other 2 attributes have around 80 swatch options each.

This created around 6000 sub-products.

Now when browsing to the main configurable product in admin I get this error.

Warning: DOMDocumentFragment::appendXML(): Entity: line 1: parser error : internal error: Huge input lookup in /home/thevinyl/public_html/vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php on line 60

So cannot even do anything with the product, in the front end the product does actually seem to display,

Please can someone help me with the steps to fix this bug. I may need a simple guide, LOL

Thanks John

[swvjeff]

I'm also experiencing this when trying to access the main catalog admin page on 2.1.6. Any updates?

[mttjohnson]

The limit comes from the libxml library that PHP uses for XML processing:
http://php.net/manual/en/domdocument.loadxml.php#113676

libxml introduced a limit for safety purposes, with the ability to pass an option to override the default 10MB limit.
http://xmlsoft.org/html/libxml-parserInternals.html#XML_MAX_TEXT_LENGTH

That limit was introduced in libxml 2.9.0 release. It looks like the limit is specifically to the contents of a single element like text in the case of cdata. The entire document could be larger than 10MB as long as a single element isn't larger than 10MB.

You can check to see what version of libxml is on a system:
php -i | grep libxml

The appendXML method is not part of the DOM standard, and the DOMDocumentFragment::appendXML method does not provide the option to pass LIBXML_PARSEHUGE to the method like you can with DOMDocument::loadXML.
http://php.net/manual/en/domdocumentfragment.appendxml.php

It looks like it's the parser that complains about the size, so it may be possible to work around the issue and instead of using appendXML, use loadXML and implement appendXML differently to include the options. I have no idea if in trying to implement the option you'd run into same issues elsewhere though.

While it may be possible to work around the limitation of libxml and appendXML, I don't expect allowing 10MB of JSON data to be embedded into an HTML document is necessarily a good practice, and it may be better to avoid this scenario than accommodate it.

[sinamiandashti]

same here

[bahramdavodi]

I'm also on 2.1.7.

[kkrieger85]

One possible solution could be adding LIBXML_PARSEHUGE
in this file:
vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php:37

before:

    public function __construct(
        \Psr\Log\LoggerInterface $logger,
        $content
    ) {
        $this->logger = $logger;
        $document = new \DOMDocument(static::XML_VERSION, static::XML_ENCODING);
        $document->loadXML($content);
        $this->templateNode = $document->documentElement;
    }

After:

    public function __construct(
        \Psr\Log\LoggerInterface $logger,
        $content
    ) {
        $this->logger = $logger;
        $document = new \DOMDocument(static::XML_VERSION, static::XML_ENCODING);
        $document->loadXML($content, LIBXML_PARSEHUGE);
        $this->templateNode = $document->documentElement;
    }

Unfortunately this solution is not complete. I still get following error:

Warning: DOMDocumentFragment::appendXML(): Entity: line 1: parser error : CData section too big found in View/TemplateEngine/Xhtml/Template.php on line 61

[dn23rus]

Hi, I had same problem with order grid page after upgrade to 2.1.7. There was ui component sales_order_grid.xml in one extension with content like this:

<?xml version="1.0" encoding="UTF-8"?>
<listing xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Ui:etc/ui_configuration.xsd">
    <container name="listing_top">
        <massaction name="listing_massaction">
            <action name="...">
                <argument name="data" xsi:type="array">
                    <item name="sortOrder" xsi:type="string">100</item>
                    <item name="config" xsi:type="array">
                        <item name="type" xsi:type="string">...</item>
                        <item name="sortOrder" xsi:type="string">100</item>
                        <item name="label" xsi:type="string" translate="true">...</item>
                        <item name="url" xsi:type="url" path="..."/>
                    </item>
                </argument>
            </action>
        </massaction>
    </container>
</listing>

I renamed node <container> with <listingToolbar> and the problem gone.

[Nix-id]

Also have this problem when try open configurable product with more then 3k assigned simple products

Warning: DOMDocumentFragment::appendXML(): Entity: line 1: parser error : internal error: Huge input lookup in /var/www/html/vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php on line 60

[Nix-id]

Some magic for method appendXML()

in file /var/www/html/vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php
method public function append($content)

must be changed to:

    $target=$this->templateNode->ownerDocument;

    $source = new \DOMDocument();
    $source->loadXml($content, LIBXML_PARSEHUGE);

    $this->templateNode->appendChild(
        $target->importNode($source->documentElement, TRUE)
    );

Also you can change public function __construct and add 'LIBXML_PARSEHUGE'
$document->loadXML($content, LIBXML_PARSEHUGE);

Can someone help me with test and explaine how override this class without change code in 'vendor' folder?

[basselalaraaj]

Same here in the customer grid, im on magento 2.1.8

[kkrieger85]

@Nix-id @basselalaraaj did you try Version 2.2.x?

[basselalaraaj]

i found my problem to be in a extention.
I found a node <container> with the name "listing_top", the original type of the node was <listingToolbar>.
By using the same in name in different node, it throws the original node <listingToolbar> away, including the <paging name="listing_paging"/>.
Causing the ui grid trying to load all the data at once, instead of using page size.
I renamed node <container> with <listingToolbar> and the problem was gone.

[ghost]

@joshspivey thanks

[attiquer]

Using Magento 2.1.11 and still get this error when trying to view a configurable product with alot of configurations (around 3,000).
Please re-open this issue !

[peterjaap]

We've open sourced our solution for using configurable products with an extremely large amount of associated simple products; https://github.com/elgentos/LargeConfigProducts, which has a workaround for this issue.

[attiquer]

Brilliant, works like a charm. Thanks @peterjaap

[magento-engcom-team]

@joshspivey, thank you for your report.
The issue is already fixed in 2.2.3, 2.3.0

[peterjaap]

@magento-engcom-team could you refer to a specific commit? I'm curious to see which solution you implemented.

[ghost]

I am on 2.2.3 and am still receiving this error when trying to display a configurable product with 10300 or so associated products. I have tried peterjaap's extension but that resulted in 500 errors. Could just but sparse server resources but wanted to check and see if anyone had a different solution. Thanks in advance.

[peterjaap]

@farbewerk could you open an issue for your 500 errors at https://github.com/elgentos/LargeConfigProducts/issues ?

[simonmaass]

I am having the same issue with 6k product variations... is there any magento 2 fix in the core for this happening any time soon?

I am running magento 2.2.5 and can say it is definately NOT fixed!

[peterjaap]

@simonmaass I wouldn't bet on it... this doesn't seem to be a priority for Magento unfortunately :(

[narayan14]

We've open sourced our solution for using configurable products with an extremely large amount of associated simple products; https://github.com/elgentos/LargeConfigProducts, which has a workaround for this issue.

We have tried to install this extension but it raised an below error in command prompt. Can you please check?
Class Elgentos\LargeConfigProducts\Console\Command\PrewarmerCommand does not exist

[crwa]

Some magic for method appendXML()

in file /var/www/html/vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php
method public function append($content)

must be changed to:

  $target=$this->templateNode->ownerDocument;

  $source = new \DOMDocument();
  $source->loadXml($content, LIBXML_PARSEHUGE);

  $this->templateNode->appendChild(
      $target->importNode($source->documentElement, TRUE)
  );

Also you can change public function __construct and add 'LIBXML_PARSEHUGE'
$document->loadXML($content, LIBXML_PARSEHUGE);

Can someone help me with test and explaine how override this class without change code in 'vendor' folder?

This works for me.

[iickoo]

Hi, problem has not been fixed on version 2.3.3, after save 2k associated products to configurable product:
Warning: DOMDocumentFragment::appendXML(): Entity: line 1: parser error : internal error: Huge input lookup in .../vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php on line 60

[vignesvaran]

Hi, Am also facing the same problem. I have 15k simple products only. When try to go products grid in admin panel I got below error.

Warning: DOMDocumentFragment::appendXML(): Entity: line 1: parser error : CData section too big found in /var/www/html/vendor/magento/framework/View/TemplateEngine/Xhtml/Template.php on line 60

[simonmaass]

we are having the same issue in m2.3.4

[joshspivey]

https://github.com/joshspivey/magento2-sales-grid

On Fri, Apr 17, 2020 at 1:37 AM Simon Maass [e-mail]
wrote:

we are having the same issue in m2.3.4

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/magento/magento2/issues/8084#issuecomment-615120115,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAPL7RLMG2CRTKAR5K56FRLRNAILHANCNFSM4C33ZA3A
.

[Jimgitsit]

The package JoshSpivey/mage2-sales-grid isn't in packagist so it can't be installed via composer, plus it hasn't been updated in 4 years so I question whether or not it works in 2.3+. The readme doesn't say what versions it supports either. In short, that's not/no longer a good solution.