Prestashop: PHP Fatal error: Out of memory in CategoryDataProvider.php

[mayoji]

After upgrading a store from 1.6 to 1.7.5.1, attempts to view a product information page in the back office (e.g. /admin/index.php/sell/catalog/products/16) lead to a PHP Fatal Error:

PHP Fatal error: Out of memory (allocated 3315597312) (tried to allocate 536870912 bytes) in /var/www/html/src/Adapter/Category/CategoryDataProvider.php on line 206

The catalog has about 12,000 products in 528 categories and sub-categories.

I tried to set PHP's "memory_limit" to up to 3700M, to no avail.

PrestaShop version: 1.7.5.1
PHP version: 5.6.40

This is possibly related to:

https://github.com/PrestaShop/PrestaShop/issues/12589
http://forge.prestashop.com/browse/BOOM-5947

Note: the same store was working fine with a memory_limit set to just 256M with PS 1.6.

[khouloudbelguith]

Hi @mayoji,

Have you the same issue reported here: https://github.com/PrestaShop/PrestaShop/issues/9809?
Thanks!

[mayoji]

Hi @khouloudbelguith,

No, issue #9809 describes a situation where the browser runs out of resources because it's trying to display too much data. It's a client-side issue due to how pagination was designed.
The issue I'm having leads to the server running out of memory.

[khouloudbelguith]

@mayoji, so your problem is a server issue?
Thanks!

[mayoji]

Yes, PHP on the server runs out of memory.

[khouloudbelguith]

@mayoji, if your issue is not a PrestaShop's core bug but a server configuration, I should close the issue & you need to check the issue with your host.
Thanks!

[mayoji]

@khouloudbelguith, This is NOT a server issue. As I wrote, the same store was working fine with PS 1.6 and only 256M of PHP memory_limit. The PHP script crashes running out of memory with P.S. 1.7 and 3700M of PHP memory_limit. It's a bug or major performance issue in the code.

[khouloudbelguith]

@mayoji, thanks for your feedback.
I will create a shop with a large number of products to reproduce your issue and we'll come back to you if I need more information.
Thanks!

[mayoji]

Thanks you. Note that the number of categories & sub-categories (528 in total) may also relevant to the issue.

[khouloudbelguith]

@mayoji, That's OK.
Thanks!

[khouloudbelguith]

@mayoji, I have more than 528 categories & more than 12400 products, but it is OK.
I did not manage to reproduce the issue with PS1.7.5.2.
I attached a video record
https://drive.google.com/file/d/1qWyqlzzURHVoJd8lbEEzZHrm17ypn189/view
Here's my PHP info file
phpinfo().html.zip
Thanks to check & feedback

[mayoji]

Thanks @khouloudbelguith, I will check if I can pinpoint which part of the code uses up so much memory. Note that your demo uses v 1.7.5.2 (the issue I described was with 1.7.5.1, although there doesn't seem to be have been any changes related to this in the change log to v 1.7.5.2. More relevant might be the fact that it looks like you created a flat list of categories. A realistic store has a hierarchy with several levels of categories and sub-categories.

[mayoji]

By the way @khouloudbelguith, how much PHP memory_limit did you set when you tried? Thanks.

[khouloudbelguith]

@mayoji, in the database, the difference between a category & a sub-category is the id_parent, I don't think, it could be a problem.
About the PS1.7.5.2, this version fixes a security issue for 1.7.5.x.
memory_limit = 128M
I attached my PHP info file in my previous comment.
Thanks!

[Matt75]

@khouloudbelguith Try with a hierarchy of categories, I tested my self and I manage to reproduce. Tested some weeks ago but I think this is still relevant in 1.7.5.2

Problem is located in Category::getNestedCategories() with a lot of categories AND subcategories, too much data passed to FormComponent to generate Categories Tree in BO add/edit Product page

This process consume too much memory in real conditions.

[khouloudbelguith]

@Matt75, thanks!
I will try to create a huge number of sub-categories & feedback.

[khouloudbelguith]

Hi @Matt75,

I tried to create a huge shop, with more than 500 categories, more than 600 sub-categories, more than 10000 products => no issue reproduced.
I attached my database
1752largenumber.sql.zip
Thanks!

[Matt75]

@khouloudbelguith I use your export to deploy a PrestaShop instance.
I manage to reproduce the issue, I cannot access to add/edit Product page.

[10-May-2019 15:17:38 Europe/Paris] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /vendor/twig/twig/lib/Twig/NodeVisitor/SafeAnalysis.php on line 57
[10-May-2019 15:17:38 Europe/Paris] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /vendor/symfony/symfony/src/Symfony/Component/Debug/Exception/OutOfMemoryException.php on line 1
[10-May-2019 15:17:38 Europe/Paris] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 163840 bytes) in Unknown on line 0

[khouloudbelguith]

@Matt75, my memory_limit = 128M & it is OK.
How much memory_limit do you have on your server?
Thanks!

[Matt75]

Server information Darwin Darwin Kernel Version 18.5.0: Mon Mar 11 20:40:32 PDT 2019; root:xnu-4903.251.3~3/RELEASE_X86_64 x86_64
Server software version: Apache
PHP version: 7.1.26
Memory limit: 128M
Max execution time: 30

[khouloudbelguith]

@Matt75, I have error 500 when I try to edit a product only when debug mode is enabled.
If debug mode is disabled => OK
https://drive.google.com/file/d/1vSj0ckQOQmwGNvv4IsWlDOPK3nMymE01/view
The error 500 => it is the same issue reported here: https://github.com/PrestaShop/PrestaShop/issues/9809
Thanks!

[Matt75]

Yes I always have debug mode activate at work, I missed that

[mayoji]

@khouloudbelguith, @Matt75 I checked the code and found what's using up all the memory available to PHP. I was at first reluctant to do it because I've never used PrestaShop or PHP (I'm doing this to help a friend).

    while (isset($this->categoryList[$categoryId])) {
        $category = $this->categoryList[$categoryId];
        $categories[] = $category['name'];
        $categoryId = $category['id_parent'];
    }

If a category somehow ends up having the same id as it's parent (or more generally if there are circular references), we have an infinite loop with $categories[] = $category['name'] consuming all memory.

I have no idea how a category was created with the same id as its parent, but you may want to make sure the code for adding categories doesn't allow this.

https://github.com/PrestaShop/PrestaShop/blob/c5b9f72a4e02960155258e3f8d26018233867b47/src/Adapter/Category/CategoryDataProvider.php#L204

[khouloudbelguith]

@mayoji, thanks for your feedback.
I’ll add this to the debug roadmap so that it’s fixed. If you have already fixed it on your end or if you think you can do it, please do send us a pull request!
Thanks!

[khouloudbelguith]

Related to: #9809

[mayoji]

@khouloudbelguith we can detect circular references and prevent an infinite loop with something like:

public function getParentNamesFromList($categoryId)
{
    $categories = [];
    $visited = [$categoryId];
    while (isset($this->categoryList[$categoryId])) {
        $category = $this->categoryList[$categoryId];
        $parent_id = $category['id_parent'];
        if (in_array($parent_id, $visited)) break;  // Prevent infinite loop if parent_id is set to the id of the category or one of its sub-categories.
        $visited[] = $parent_id;
        $categories[] = $category['name'];
        $categoryId = $parent_id;
    }
    return $categories;
}

But as long as the PS code already checks that there are no circular references when creating a category (including via third-party modules), that might not be needed. I have no idea how PS works, so I don't know if modules are constrained to use a PS API to create categories or if they can write directly to the database. In my case, the (unused) category with a parent_id set to the category's id may have been created directly in the database by the contractor who originally developed the store.

[mayoji]

Of course, there might be other PS 1.6 stores out there with invalid categories, and those sites will crash after upgrading to PS 1.7 unless the code is modified.

[luigimassa]

I found that in database table of categories there is a circular reference
This shop is an upgrade from 1.5 to 1.6 to 1.7
Categories 3 and 21 were demo categories
I deleted these records manually from database and now I don't have more memory error

[darylstone]

Hello guys, is there a way to detect those circular categories? I have about 6k+ categories. Not easy to find circular references if there any exists. :D

[joejvggithub]

Hi,

I looked for circular references as I had this problem with a 1.7.6.1 migration from 1.6.11.1 and found that this snippet in ./src/Adapter/Category/CategoryDataProvider.php is using a while loop to iterate through an array. I changed it to a foreach loop and no more out of memory and categories associated with the product display properly.

public function getParentNamesFromList($categoryId)
{
    $categories = [];

    /* while (isset($this->categoryList[$categoryId])) { */
    foreach ($this->categoryList[$categoryId] as &$value) {
        $category = $value;
        $categories[] = $category['name'];
        $categoryId = $category['id_parent'];
    }

    return $categories;
}

[marsaldev]

I think that a solution could be this:

• "Load" only the associated categories
• Load only 1st level categories
• When I explode a category the CategoryDataProvider will load the subcategories (and so on), when is performed this task if the category is associated the checkbox is checked (obviously)

But, we should also display the product main category, at the moment I don't figured out how to solve that.

In this way, also a database of 40k categories isn't a problem.

@khouloudbelguith, @Matt75 wdyt?

[hugolin69]

I am having something that could be linked here
I have only 23 categories, I am only 1 sublevel (Root Categorie "1" / Home "2" / Categories / Sub Categories).

I applied the patch, will wait to see.
I have 300 products, the bug comes also when cache is disabled in PS, but still have smarty no recompile options.