Magento2: Product Duplication crash

[urielbeaupre]

We currently have magento 2.0.4 installed on Linux server running nginx and php5, we also have it installed on another machine with php7 and this issue occurs in both instalations.

We have a fairly decent sized catalog list (5000) and are on the process of restructuring the catalog. To do this we are duplicating products and this is where the problem starts, when duplicating the product magento just goes on a loop while reloading page, right after the spinning wheel disappears, this doesnt happen in every product but I can't see a common field or setting on the products that it does it on either.

I checked the debug.log while the page is stuch on loop and found the following error:

[2016-04-11 13:39:17] main.DEBUG: cache_invalidate: {"method":"POST","url":"http://test.ch.vpdev.no/adm/catalog/product/save/id/3355/set/14/key/edc2211dbf867a295c96143cf36672b71a02761d3e713d49e75332daa72e966a/active_tab/product-details/back/duplicate/active_tab/product-details/","invalidateInfo":{"tags":["catalog_product_4906"],"mode":"matchingAnyTag"},"is_exception":false} [] [2016-04-11 13:39:17] main.DEBUG: cache_invalidate: {"method":"POST","url":"http://test.ch.vpdev.no/adm/catalog/product/save/id/3355/set/14/key/edc2211dbf867a295c96143cf36672b71a02761d3e713d49e75332daa72e966a/active_tab/product-details/back/duplicate/active_tab/product-details/","invalidateInfo":{"tags":["catalog_product"],"mode":"matchingAnyTag"},"is_exception":false} [] [2016-04-11 13:39:17] main.DEBUG: cache_invalidate: {"method":"POST","url":"http://test.ch.vpdev.no/adm/catalog/product/save/id/3355/set/14/key/edc2211dbf867a295c96143cf36672b71a02761d3e713d49e75332daa72e966a/active_tab/product-details/back/duplicate/active_tab/product-details/","invalidateInfo":{"tags":["catalog_product_4906","catalog_product"],"mode":"matchingAnyTag"},"is_exception":false} []

This error is being printed out roughly 10 times per second, and once it starts the whole website becomes unavailable, we found 2 ways to bring the website back up one is to rewrite the static files (we tried both on dev mode and prod mode) and the other one is to reboot the server. Altough this doesn't really help us as the products still can't be duplicated.

Any insight on this problem will be extremly helpfull.

Best Regards
Joni Silva

[mazhalai]

@urielbeaupre that is a debug log and not an error log. it just shows when cache is being invalidated. Do you have any details in exception.log?

[urielbeaupre]

On the exception log I have this
Next exception 'Zend_Db_Statement_Exception' with message 'SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction, query was: INSERT INTO `$

[salelsol]

Same issue magento 2.0.4 when I´m tryed duplicate a product. I can´t finish action Save & Duplicate product always display "Please Wait", next I´m recived and message from server: Our automated system has noticed that the disk of server is full. When I´m checked this abnormal behavior I have realized that the images of the product I wanted to duplicate had increased to fill the entire disk space in my case in directory /media/catalog/product/r.

[simonkguest]

I am also getting the same sort of issue with 'save and duplicate' a product with magento 2.0.4 It just hangs and eventually times out with a 500 error. I dont get the "Please Wait" message anymore, but i did yesterday when i was trying to duplicate a product.

[salelsol]

• Hi, I just duplicate a product, but if Product In Websites are unchecked twice (english, spanish).
• I´m thinking this bug is related with this issue: https://github.com/magento/magento2/issues/3519 because there are many dabase entries for store view= 0 (admin: store view= 0 non existing store view for me just english:store view= 1 and spanish:store view= 2 for me) but I don´t know what can I do for solve it.

[piotrekkaminski]

@salelsol @simonkguest @urielbeaupre do you still get this issue in later versions?

[salelsol]

• Hi, I´ve solved this issue if duplicate a product in an all new product in and all new categorie, then I haven´t any issue when duplicate a product, but for old Magento 1 imported products with yours categories, I can´t duplicate any product because crash server. Lasted version tryed is Magento 2.0.6.

Best Regards
Toni

[piotrekkaminski]

Internal issue MAGETWO-53526

[simonkguest]

For me, the way i have been working around this issue;
Create a new product.
Duplicate the new product
Duplicate the duplicated product (not the original)

This has been working for me. I hope this may help

I believe that salelsol is correct I have not been able to duplicated imported products either.

[blizam]

My Products were originally created in Magento 2.0.3... never in a 1.x install

I never had a problem with product duplication myself until I upgraded to 2.1.0 recently... now when I Save an Duplicate a new product the behavior is the same as described here in this post.

If I should open a new post just holler... I was on the fence as the behavior seems the same as is noted here... the major difference being I never had my products exported from a 1.x site

-Open existing Product
-Save and Duplicate
-Spinner/Overlay comes up while it saves, then disapears
-Page/Browser loader spings
-Eventually a Server Error 500 displays

Nothing in my Apache Error Log for the 500 response either unfortunately

Save+Duplicate does appear to Save the Product... but it does not Duplicate currently for _most_ of my current products

I tested on a variety of Products... some original Products, some previously Duplicated Products... it "seems" that products I originally created I can still duplicate... but products I have that I once duplicated from another product cannot themselves be duplicated... I haven't exactly kept tracked of which ones were once duplicates... most of products are duplicates as that's often the easiest way to build out a catalog of new/existing products.

OK -- so it appears to be related to "Product in Websites" options... if I uncheck all Websites, save the product, I can then Save + Duplicate on a product on couldn't have otherwise... My "old"/originally created products seem to work fine with Save + Duplicate and a Website selected but my "newer"/duplicated products can only Save+Duplicate if I first uncheck all associated Websites

[blizam]

Eh... further issues with Images on Duplicated Products... after deleting eh original products Base Image I cannot set a new base/thumbnail image... well, I can set it... but it won't appear on the frontend for the Thumbnail or Small image... it only shows up in the actual Product View... Category View pulls in the system default image for the thumb/small display

Creating new products works fine

[blizam]

I don't seem to have any issues duplicating new/post-2.1.0 upgrade products... it's only when I duplicate products I created pre 2.1.0 that I have the various previously mentioned issues

I haven't extensively tested duplicating new products from/to a different store/store-view... but basic save+duplicate seems to be fine with newly created products

[romeof1980]

Hi there, look like a bug for me too.
I've tried with v 2.0.0 , 2.0.2 , 2.0.6 , 2.0.7
no ways to duplicate...

[cytracon]

Any news here? Same problem ...v 2.1

[zsmfyp]

Fatal error: Uncaught Error: Call to a member function getOrigData() on null in /Applications/MAMP/htdocs/z/app/code/Ess/M2ePro/Observer/StockItem/Save/After.php:54 Stack trace: #0 /Applications/MAMP/htdocs/z/app/code/Ess/M2ePro/Observer/StockItem/Save/After.php(46): Ess\M2ePro\Observer\StockItem\Save\After->processQty() #1 /Applications/MAMP/htdocs/z/app/code/Ess/M2ePro/Observer/AbstractModel.php(66): Ess\M2ePro\Observer\StockItem\Save\After->process() #2 /Applications/MAMP/htdocs/z/vendor/magento/framework/Event/Invoker/InvokerDefault.php(73): Ess\M2ePro\Observer\AbstractModel->execute(Object(Magento\Framework\Event\Observer)) #3 /Applications/MAMP/htdocs/z/vendor/magento/framework/Event/Invoker/InvokerDefault.php(61): Magento\Framework\Event\Invoker\InvokerDefault->_callObserverMethod(Object(Ess\M2ePro\Observer\StockItem\Save\After), Object(Magento\Framework\Event\Observer)) #4 /Applications/MAMP/htdocs/z/vendor/magento/framework/Event/Manager.php(66): Magento\Framework\Event\Invoker\InvokerDefault->dispatch(Array, Obj in /Applications/MAMP/htdocs/z/app/code/Ess/M2ePro/Observer/StockItem/Save/After.php on line 54
I have the same problem when i duplicate one product.

[zsmfyp]

i use in version 2.1.1

[jwallb40]

Has anyone come up with a fix for this? We're seeing the same thing on products that were migrated to magento 2. After a client attempted to duplicate a pre-migration product, some loop caused it to duplicate the image thousands of times, leaving us filling our entire disk.

[jason-zz]

@jwallb40 I have the same issue. M2 2.1.0. Images are keep creating when I try to duplicated migrated products. It only stops once PHP timeout.

[ghost]

@jwallb40 I am having same issue. When we duplicate any product, it generates almost 4000 images. We have only 13 products in store and we tried to duplicate some of them and suddenly our media folder size increase to 130GB from 80MB. This is super strange and suspecious issue.

Anybody found solution for this issue?

[ThomasNegeli]

Any news on this? Fixed in 2.1.3?

[simonkguest]

Hey Thomas

I am still running 2.1.2

I ended up starting fresh, and re entered all my 3000 odd products on a clean install and have not had the issue since.

From memory it was something to do with how my products were converted from 1.9 over to 2.1 it was something with the attribute sets.

I think i created new attributes sets from the new default set and then install something to allow change attribute sets, and changed all of the products from the old version of the attribute set to the new ones.

I have a feeling that sorted the problem,

But i ended up having other issues with images not updating with new products so i just started fresh.

I hope this helps

Regards

Simon

Ministry of Bass
1 / 655 South Road
Black Forest
SA 5035
08 83715121
www.ministryofbass.com.au

From: ThomasNegeli
Sent: Wednesday, January 25, 2017 8:27 PM
To: magento/magento2
Cc: simonkguest ; Mention
Subject: Re: [magento/magento2] Product Duplication crash (#4096)

Any news on this? Fixed in 2.1.3?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[shamoon]

Having this too, media folder suddenly 28GB.

This is ridiculous, its a major bug!

[mikehenze]

Magento 2.1.4 still has this very big problem...

[Wojtasik]

Magento 2.1.1 - same issue.
New product, then save & duplicate - works fine.
Migrated product, the save & duplicate - fail, php takes 100% CPU and POSTs data.

[orlangur]

@Wojtasik it is pretty hard to guess what went wrong without an isolated or at least reproducible case.

Maybe somebody managed to spot it constantly with something like
install M1 with sample data -> run official migration tool to move products to M2 -> duplicate certain product and observe issue?

[scientic]

Same issue here.
Magento 2.1.4
How to stop the duplication process?
Is there a php command to kill this ?
I tried to delete the product where I think it started with via Admin, but that throws an internal server error. probably because it's still busy copying.
I have deleted the product via phpadmin directly from the database.
Still it keeps duplicating over and over.
Our shop is on a shared environment. So resetting the server is not an option

[Wojtasik]

@scienitc we have SSH access, so I've killed the process. Do you have SSH access to the server?

[scientic]

@wojtasik yes I do have SSH access

[Wojtasik]

So basically what I've done:

• run "top"
• check php process id (it should take 100% cpu)
• run kill -9 processId
  Done. It should be ok now.

[scientic]

thanks, I will try that

[shamoon]

Being able to kill the process is not a solution. My client forgot that I asked him not to try the 'save and duplicate' option today and we ended up with a PHP process using 99% CPU crippling the server as well as 48,000 duplicate files of the same 3 product images!

I can confirm that this only happens with migrated products, creating a fresh product and duplicating it seems to work fine.

[scientic]

I managed to stop this by going into the database and delete the product that was being copied over and over again.

On 11 Mar 2017, at 02:18, Michael Shamoon [e-mail] wrote:

Being able to kill the process is not a solution. My client forgot that I asked him not to try the 'save and duplicate' option today and we ended up with a PHP process using 99% CPU crippling the server as well as 48,000 duplicate files of the same 3 product images!

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/magento/magento2/issues/4096#issuecomment-285827677, or mute the thread https://github.com/notifications/unsubscribe-auth/ALnAquYPZQUr9zEG09SHBHWQ6wHl_GwDks5rkfZUgaJpZM4IEbO5.

[shamoon]

I didn't even realize there are duplicate entries in the database as well, where are they?

[ryangall7]

We had a similar issue with 2.1.4. I was able to isolate it to the way url rewrites were getting generated in the copying process. The error is a bit misleading it has to do with url_path not url_key. When copying a product a new url_key was being generated from the old one with a "-1" appended to it however the duplicate product still has a value in the url_path attribute. When the url rewrite is getting generated the url_path takes precedence in over the url_key for some reason (I'm not totally sure why both of these are needed.)

To fix we just need to force the url_path attribute to null during the copping process. This is my new copy() function from \Magento\Catalog\Model\Product\Copier:

    /**
     * Create product duplicate
     *
     * @param \Magento\Catalog\Model\Product $product
     * @return \Magento\Catalog\Model\Product
     */
    public function copy(\Magento\Catalog\Model\Product $product)
    {
        $product->getWebsiteIds();
        $product->getCategoryIds();

        /** @var \Magento\Catalog\Model\Product $duplicate */
        $duplicate = $this->productFactory->create();
        $duplicate->setData($product->getData());
        $duplicate->setOptions([]);
        $duplicate->setIsDuplicate(true);
        $duplicate->setOriginalId($product->getEntityId());
        $duplicate->setStatus(\Magento\Catalog\Model\Product\Attribute\Source\Status::STATUS_DISABLED);
        $duplicate->setCreatedAt(null);
        $duplicate->setUpdatedAt(null);
        $duplicate->setId(null);
        //url path must be set to null to generate url_rewrites correctly
        $duplicate->setUrlPath(null);
        $duplicate->setStoreId(\Magento\Store\Model\Store::DEFAULT_STORE_ID);

        $this->copyConstructor->build($product, $duplicate);
        $isDuplicateSaved = false;


        $i = 0;
        do {
            $urlKey = $duplicate->getUrlKey();
            $urlKey = preg_match('/(.*)-(\d+)$/', $urlKey, $matches)
                ? $matches[1] . '-' . ($matches[2] + 1)
                : $urlKey . '-1';
            $duplicate->setUrlKey($urlKey);

            try {
                $duplicate->save();
                $isDuplicateSaved = true;
            } catch (\Magento\Framework\Exception\AlreadyExistsException $e) {
                $error = $e->getRawMessage();
                //if errored 20 times break
                if($i > 20){
                    $isDuplicateSaved = true;
                }
            }
            //count to aviod run away loop
            $i ++;
        } while (!$isDuplicateSaved);

        $this->getOptionRepository()->duplicate($product, $duplicate);
        $metadata = $this->getMetadataPool()->getMetadata(ProductInterface::class);
        $product->getResource()->duplicate(
            $product->getData($metadata->getLinkField()),
            $duplicate->getData($metadata->getLinkField())
        );
        return $duplicate;
    }

you can also see that I've added a counter for the do-while loop so it will break after 20 attempts, this do-while loop is the whats causing the images to be copied over and over, so I figured it would be safer to just cut this off at 20 even if I fixed the url_key issue.

It probably should also be investigated why the images are being processed before the save is complete.

[FreederFree]

Hey @ryangall7,

thanks for your solution! Does this still work for you? If yes: how can I use it without changing core-code? Is there a way to override it? If yes, could you tell me, how you've done it?

Thanks in advance!

[macbombastic]

Thank you ... setting the url to null worked, at least to copy an imported product (from Magento 1.9)

[ryangall7]

Hey @FreederFree,

Yes, we added it to a custom module and overwrote the class with dependency injection, if your not familiar here's is a pretty good tutorial: http://inchoo.net/magento-2/overriding-classes-magento-2/

Here is my di.xml

<?xml version="1.0"?>
<!--
/**
 * @author Amasty Team
 * @copyright Copyright (c) 2017 Amasty (https://www.amasty.com)
 * @package Amasty_Stockstatus
 */
-->
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">

    <preference for="Magento\Catalog\Model\Product\Copier" type="NAMESPACE\MODULE_NAME\Model\Product\Copier" />

</config>

and here is the overriding class NAMESPACE/MODULE_NAME/Model/Product/Copier.php

<?php
/**
 * Catalog product copier. Creates product duplicate
 *
 * Copyright © 2016 Magento. All rights reserved.
 * See COPYING.txt for license details.
 */
namespace NAMESPACE\MODULE_NAME\Model\Product;

use \Magento\Catalog\Api\Data\ProductInterface;
use \Magento\Catalog\Model\Product\CopyConstructorInterface;

class Copier extends \Magento\Catalog\Model\Product\Copier
{
    /**
     * @var Option\Repository
     */
    protected $optionRepository;

    /**
     * @var CopyConstructorInterface
     */
    protected $copyConstructor;

    /**
     * @var \Magento\Catalog\Model\ProductFactory
     */
    protected $productFactory;

    /**
     * @var \Magento\Framework\EntityManager\MetadataPool
     */
    protected $metadataPool;

    /**
     * @param CopyConstructorInterface $copyConstructor
     * @param \Magento\Catalog\Model\ProductFactory $productFactory
     */
    public function __construct(
        CopyConstructorInterface $copyConstructor,
        \Magento\Catalog\Model\ProductFactory $productFactory
    ) {
        $this->productFactory = $productFactory;
        $this->copyConstructor = $copyConstructor;
    }

    /**
     * Create product duplicate
     *
     * @param \Magento\Catalog\Model\Product $product
     * @return \Magento\Catalog\Model\Product
     */
    public function copy(\Magento\Catalog\Model\Product $product)
    {
        $product->getWebsiteIds();
        $product->getCategoryIds();

        /** @var \Magento\Catalog\Model\Product $duplicate */
        $duplicate = $this->productFactory->create();
        $duplicate->setData($product->getData());
        $duplicate->setOptions([]);
        $duplicate->setIsDuplicate(true);
        $duplicate->setOriginalId($product->getEntityId());
        $duplicate->setStatus(\Magento\Catalog\Model\Product\Attribute\Source\Status::STATUS_DISABLED);
        $duplicate->setCreatedAt(null);
        $duplicate->setUpdatedAt(null);
        $duplicate->setId(null);
        $duplicate->setUrlPath(null);
        $duplicate->setStoreId(\Magento\Store\Model\Store::DEFAULT_STORE_ID);

        $this->copyConstructor->build($product, $duplicate);
        $isDuplicateSaved = false;


        $i = 0;
        do {
            $urlKey = $duplicate->getUrlKey();
            $urlKey = preg_match('/(.*)-(\d+)$/', $urlKey, $matches)
                ? $matches[1] . '-' . ($matches[2] + 1)
                : $urlKey . '-1';
            $duplicate->setUrlKey($urlKey);

            try {
                $duplicate->save();
                $isDuplicateSaved = true;
            } catch (\Magento\Framework\Exception\AlreadyExistsException $e) {
                $error = $e->getRawMessage();
                //if errored 20 times break
                if($i > 20){
                    $isDuplicateSaved = true;
                }
            }
            //count to aviod run away loop
            $i ++;
        } while (!$isDuplicateSaved);

        $this->getOptionRepository()->duplicate($product, $duplicate);
        $metadata = $this->getMetadataPool()->getMetadata(ProductInterface::class);
        $product->getResource()->duplicate(
            $product->getData($metadata->getLinkField()),
            $duplicate->getData($metadata->getLinkField())
        );
        return $duplicate;
    }

    /**
     * @return Option\Repository
     * @deprecated
     */
    private function getOptionRepository()
    {
        if (null === $this->optionRepository) {
            $this->optionRepository = \Magento\Framework\App\ObjectManager::getInstance()
                ->get('Magento\Catalog\Model\Product\Option\Repository');
        }
        return $this->optionRepository;
    }

    /**
     * @return \Magento\Framework\EntityManager\MetadataPool
     * @deprecated
     */
    private function getMetadataPool()
    {
        if (null === $this->metadataPool) {
            $this->metadataPool = \Magento\Framework\App\ObjectManager::getInstance()
                ->get('Magento\Framework\EntityManager\MetadataPool');
        }
        return $this->metadataPool;
    }
}

[FreederFree]

@ryangall7: Thank you for your detailed answer, the tutorial and your code! It helped a lot!

Now as I have my module which overrides the Copier class, I get the following error while duplicating a Product:

Fatal error: Uncaught Error: Call to undefined methodd Magento\Catalog\Model\Product\Option\Repository::duplicate()

What could went wrong here?

I'm running Magento 2.0.13 by the way

[onepack]

This is a huge issue!
My client has duplicated 16 pre-migration products and now they ran out of 100GB of extra storage they just bought.
So now the shop with 6000 products is using 200GB to just... run....
I will advice them not to use the duplicate option for now until I tested the option described here.

[onepack]

I have just made 110GB free by removing all the duplicates... My god this is a mess.
For now I instructed the client to remove the images from the product before duplicating it.
They can put them back after they are finished.
Will try one of the above solutions later today.

[r-martins]

Almost impossible to fix on 2.0.7 without upgrading it and facing another thousand problems. :(
In 2.0.x we don't have the EntityManager component, required in the above solution.

What I did was: create a module with a plugin that overrides \Magento\Catalog\Model\Product\Copier (with surround approach) and simply adds a $duplicate->setUrlPath($urlKey . '.html'); after $duplicate->setUrlKey($urlKey);. 👍

[kamilszewczyk]

Same issue still occurs on 2.1.7

[riprod]

@r-martins Please can you explain in a little more detail and what I need to do. I can confirm that this bug is still in Magento 2.1.7
Has anyone figured out how to delete all the duplicate images it creates with a script or do I have to do it manually ? There are 10's of 1000s

[ThomasNegeli]

I've built a module to cleanup product, category and swatches media which is not used.
Let me know if it helps: https://github.com/ThomasNegeli/M2CliTools
Currently testet with 2.1.5. Hope it works with 2.1.7 too.

[gerben86]

I have exactly the same issue in Magento 2.1.9

[orlangur]

This issue seems to be fixed on 2.2.0 branch, see https://github.com/magento/magento2/issues/9466#issuecomment-339456558 for details.

Credit for investigation goes to @hostep 👍

[hostep]

@orlangur: would be great if a fix could get backported to version 2.1.x though, it's a really serious problem...

[VirtusB]

The problem still exists in 2.2.4. We went from 4 GB media to 80 GB media

[hostep]

Thanks for the update @VirtusB!

It looks like they re-added catching that Magento\UrlRewrite\Model\Exception\UrlAlreadyExistsException exception in 2.2.4 in: https://github.com/magento/magento2/commit/ac4dd33ea39e3d25226291282560ca3083635e7b which I explained about over here

@orlangur: I'm re-opening ticket #9466 if you don't mind, since the issue in 2.2.x was never really solved and was accidentally hidden away and has now re-appeared in 2.2.4 (not tested myself, but the evidence seems to point in that direction).
Issue #9466 seems to be more relevant then this issue for the disk space eating bug, so not re-opening this one.

[ubedarab]

@hostep I am also facing same problem with 2.2.4. with duplication space filled by duplicate images.
have any one get proper solution ?

[mattyl]

I can confirm this has just happed to us in 2.2.4 - migrated product, duplicate - 10,000s of images created till you kill the process

[mattyl]

exception log - site is multi store
[2018-06-14 13:04:21] main.CRITICAL: The file "/home/m2originlive/public_html/shop/pub/media/tmp/catalog/product/s/i/silver-hybrid-logo.jpg" doesn't exist or not a file {"exception":"[object] (Magento\Framework\Exception\FileSystemException(code: 0): The file "/home/m2originlive/public_html/shop/pub/media/tmp/catalog/product/s/i/silver-hybrid-logo.jpg" doesn't exist or not a file at /home/m2originlive/public_html/shop/vendor/magento/framework/Filesystem/Directory/Write.php:70)"} []

[hostep]

@mattyl: might be best to continue in #9466, that's the issue which has status "Open" an has more detailed information about the problem. Especially this comment: https://github.com/magento/magento2/issues/9466#issuecomment-305824682
Also: feel free to upvote issue #9466 so it gets more attention from the Magento devs, since it's a pretty big problem.