Data-migration-tool: Migrating from Magento CE 1.9.0.1 to Magento EE 2.1.0 1062 Duplicate entry '1-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME'

[kiranmaip]

[PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME'

[Exception]
Notice: Undefined offset: 1 in vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 261

[victor-v-rad]

Hi @kiranmaip
The error from the first screenshot has a hint. You probably migrate data for the second attempt without reverting DB of Magento 2 to its initial state.

[kiranmaip]

Hello Victor:

We did revert the db to its original state via a fresh installation of M2. This is in a sense reverting the db to its original state. We get the same error.

[victor-v-rad]

Try to use the latest version of the tool 2.1.3 It has fixes which may help.

[mikebranderhorst]

I've read somewhere that it could be due to a duplicate Default attributeset in the Magento 1 DB.

[dfelton]

@kiranmaip If in your Magento 2 Database you have renamed any of your attribute sets to "Migration_Default" then you'll encounter this error from your first screenshot when the EAV Step attempts to migrate the Magento 1 attributes sets into the Magento 2 database. Please be aware that the data migration tool will COPY the attribute sets exactly over from the M1 to M2 database. It does not merge into the existing Magento 2 attribute sets. It is also good to know that the migration tool will retain the original attribute set ids for all migrated attribute sets. Any attribute sets in the M2 site which exist prior to migration, and have a clashing attribute_set_id with the M1 attribute sets, will have their attribute_set_id revised to start from the next available id after the last M1 attribute set ID.

@mikebranderhorst The Magento 1 DB schema has a UNIQUE KEY on the entity_type_id and attribute_set_name columns so a proper Magento 1 database will never have this scenario. You'll find the definition for this UNIQUE KEY in app/code/core/Mage/Eav/sql/eav_setup/mysql4-install-0.7.0.php line 114.

[lano-vargas]

I am facing the same problem I've tried latest version of migration toll( 2.1.3 )but the error still persist has anyone got a solution or an update on this please share. Thank's

[iphigenie]

I have now read a lot of previous threads with similar ideas and I am no further.

This happens after my first migration was successful in the past, but now we are on higher versions so new version of tool and updated configuration

I am getting this message in alternance with Notice: Undefined offset: 1 in vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 266

1) i run the full integrity checks so no orphans or duplicates or foreign key issues
2) there are no duplicate attribute sets
3) i am running with -reset which should clear things but it seems does not?
4) m2's eav_attribute_set table ends up completely empty after this error (I assume that creates the alternance of error messages, i.e. when empty I get "undefined offset")

I can't reset to my "right at installation" database backup as version number has gone up in between - so "clean database" seems unachievable.

I have managed to pass through by
1) deleting all products
2) rebuilding the eav_attribute_set and eav_attribute_group tables to have only the original entries, but with the IDs from after my first migration (so starting at 14 in my case)

Some of that surely should be done by "--reset" surely?

[lano-vargas]

I have fixed the my problem by adding a missing entry in eav_attribute_set.
What I know is that Magento 1 eav_attribute_group, eav_attribute_set has no Foreign Keys 👎
while Magento 2 has 👍 .
The problem was that somebody deleted and Attribute Set from admin and because there was no Foreign Keys there was no reenforcement in deleting references entries in other table so there you go Magento 2 has Foreign Keys 👍
Thank's J.

[iphigenie]

For others who encounter this - I had a version of this which had nothing to do with my data and everything to do with the data migration tool leaving the new database in a broken state if it does not complete. It says "reverting" but not everything is reverted.

Just encountered this again in the last version

Of course the simplest way is to have a backup of clean m2 database before you start, and restore it between each run on the migration tool while you determine what tables, fields etc to map or ignore in configuration. But if you didn't do that and are stuck with a messy database, here's what happens

ON AN "ERROR" DURING THE EAV STAGE, WHEN THE SYSTEM SAYS "REVERTING" IT DOES NOT REVERT THE ATTRIBUTE_SET AND ATTRIBUTE_GROUP DATA FULLY, AND IT LEAVES THE DATA IN AN ERROR CAUSING STATE (The attribute sets and attribute groups are also not reverted properly if you use --reset)

1. m2's eav_attribute_set table ends up containing the "migration_" entries after the undefined offset error or any other error during a further substep of the eav process (hence causing step 2 "duplicate" error next time)
2. m2's eav_attribute_set table ends up completely empty after this error (causing step 1 "undefined offset" next time)

on and on forever! Nothing you can do to your source data will fix it, because the error is in the new database

You need to re-create the clean eav_attribute_set table and/or clean all the migration_ entries in eav_attribute_group

you will need to do this after every errored run

I do think the tool should re-set a clean eav_attribute_set and eav_attribute_group on a -reset run in order to avoid that error. Or at the very least not claim it is "reverting" but warn the user that they have to do some reverting manually

[iphigenie]

Been asked more detail about the table cleanup I had to do, for people stuck in this. This has nothing to do with fixing the bug but to reset your tables to a state where the tool can run again without getting stuck

It seems the tool deletes all attribute_set entries, then creates all the migrated entries, then creates all the standard entries. If it fails, it leaves the table in a messed up state.
In order to run the migration tool fresh, you need to reset the tables to a clean set.

There are 2 possible errors in a loop, "duplicate" or "offset"

If you are currently getting the error "duplicate" you need to delete entries.

in eav_attribute_set delete all the migration_ entries

mine looked like this:

(attribute_set_id, entity_type_id, attribute_set_name, sort_order)
(1, 1, 'Migration_Default', 100),
(2, 2, 'Migration_Default', 100),
(3, 3, 'Migration_Default', 100),
(4, 4, 'Migration_Default', 100),
(5, 5, 'Migration_Default', 100),
(6, 6, 'Migration_Default', 100),
(7, 7, 'Migration_Default', 100),
(8, 8, 'Migration_Default', 100),
(9, 4, 'Migration_Vint', 100),
(10, 4, 'Migration_Case', 100),
(11, 4, 'Migration_GroupVint', 100),
(13, 4, 'Migration_nonwin', 100),
(14, 1, 'Default', 2),
(15, 2, 'Default', 2),
(16, 3, 'Default', 1),
(17, 4, 'Default', 1),
(18, 5, 'Default', 1),
(19, 6, 'Default', 1),
(20, 7, 'Default', 1),
(21, 8, 'Default', 1);

I recommend after you delete all the migration_ entries that you make a dump of the table so you can recreate that clean table in seconds (ids 14 to 21)

You need to also look in eav_attribute_group and delete any migration entries in there too

If you get the undefined offset error you need to refill the eav_attribute_set table

If you don't know at what ID to start, you might be able to determine it in the eav_attribute_group tableOtherwise look at how many you have in your old site and start at 1 above
for me starting at 14 it was

INSERT INTO eav_attribute_set (attribute_set_id, entity_type_id, attribute_set_name, sort_order) VALUES(14, 1, 'Default', 2),
(15, 2, 'Default', 2),
(16, 3, 'Default', 1),
(17, 4, 'Default', 1),
(18, 5, 'Default', 1),
(19, 6, 'Default', 1),
(20, 7, 'Default', 1),
(21, 8, 'Default', 1);

adapt with your number and your prefix

I think it didn't mess with the eav_attribute_group table, that was left correct, but it's worth checking. For the record, here's my clean (again, adapt for starting id and prefix)
INSERT INTO eav_attribute_group (attribute_group_id, attribute_set_id, attribute_group_name, sort_order, default_id, attribute_group_code, tab_group_code) VALUES(73, 14, 'General', 1, 1, 'general', NULL),
(74, 15, 'General', 1, 1, 'general', NULL),
(75, 16, 'General', 10, 1, 'general', NULL),
(76, 16, 'General Information', 2, , 'general-information', NULL),
(77, 16, 'Display Settings', 20, , 'display-settings', NULL),
(78, 16, 'Custom Design', 30, , 'custom-design', NULL),
(79, 17, 'Product Details', 10, 1, 'product-details', 'basic'),
(80, 17, 'Advanced Pricing', 40, , 'advanced-pricing', 'advanced'),
(81, 17, 'Search Engine Optimization', 30, , 'search-engine-optimization', 'basic'),(82, 17, 'Images', 20, , 'image-management', 'basic'),
(83, 17, 'Design', 50, , 'design', 'advanced'),
(84, 17, 'Autosettings', 60, , 'autosettings', 'advanced'),
(85, 17, 'Content', 15, , 'content', 'basic'),
(86, 17, 'Schedule Design Update', 55, , 'schedule-design-update', 'advanced'),(87, 17, 'Bundle Items', 16, , 'bundle-items', NULL),
(88, 18, 'General', 1, 1, 'general', NULL),
(89, 19, 'General', 1, 1, 'general', NULL),
(90, 20, 'General', 1, 1, 'general', NULL),
(91, 21, 'General', 1, 1, 'general', NULL),
(92, 17, 'Gift Options', 61, , 'gift-options', NULL),

I hope this helps -

NOTE: if you have additional magento2 modules, there could well be more attribute_groups and more attribute_sets that need to exist and match. I found peeking in eav_entity_type (column default)attribute_set_id) helpful to find if modules have added attribute sets

In short - every attribute_set_id that exists in eav_attribute_groups also needs to exist in eav_attribute_sets, and point to the correct eav_entity_type (otherwise: invalid offset... error). There should be no migrated entries in eav_attribute_sets (otherwise: duplicate... error)

On Thu, Jul 6, 2017, at 08:58 PM, Steve Robbins wrote:

@iphigenie[1] Can you explain what you mean by "You need to re-create the clean eav_attribute_set table"? I'm currently stuck on this issue.> —
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub[2], or mute the thread[3]. >

Links:

1. https://github.com/iphigenie
2. https://github.com/magento/data-migration-tool/issues/204#issuecomment-313502739
3. https://github.com/notifications/unsubscribe-auth/ABAcUhPcMHqoU-lXY8VNutEdArrEGl_Tks5sLTxqgaJpZM4LQb3i

[iphigenie]

still happening in 2.2.2

[wedob13]

I still have this issue on 2.2.2

[Tarek-Adra]

having this issue on 2.2.3 also

[niners52]

2.2.4 as well

[Tarek-Adra]

@niners52
I've created a simple code to migrate product by category from Magento 1 to 2 using web services, it's super basic but it's works

checkout here https://github.com/Tarek-Adra/Product-Migrating-from-Magento-1.9-to-Magento-2-using-SOAP-and-Rest-Services

[iphigenie]

this is still happening in the latest version and the solution is still to "rebuild" the eav_attribute_set table to have all the IDs mentioned in the eav_attribute_group each time you encounter an error that stops the migration

[gouridash]

I am getting the same issue with the 2.2.6 migration tool. I am stuck on the below issue.

[PDOException]
SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '1-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME
'

Can someone guide me how to overcome this?

[rakeshdonga]

i have same issue facing with magento 2.3.0

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '4-Migration_Default' for key 'EAV_ATTRIBUTE_SET_ENTITY_TYPE_ID_ATTRIBUTE_SET_NAME'

[grahamslam]

I had a tough time with this also, but my solution was to find duplicate ID values in the magento 1.9 version database in the eav tables, (going off memory) I believe it was the eav_entity_attribute table that allowed duplicate ID's without a foreign key restraint , that in version 2.2.6 it IS a foreign key restraint in the database. My client had previous installs and uninstalls of custom extensions that did not clean up the database upon removal.

So my solution was to remove duplicated values in the eav tables in the original database before migrating to the new version. Compare values from tables eav_attribute_set, eav_entity_type, eav_attribute_group, eav_attribute and eav_entity_attribute and remove records with keys that no longer exist.

[iphigenie]

confirming that the error handling in 2.3.1 seems to avoid leaving the attribute sets in an unhappy state. Excellent

[davidcollen]

confirming that the error handling in 2.3.1 seems to avoid leaving the attribute sets in an unhappy state. Excellent

I'm still having this problem with 2.3.1. It doesn't appear to be resolved

[Jeeva-Rathinam]

@iphigenie @wedob13 @Tarek-Adra @niners52 @davidcollen Anyone finds the solution? Thanks to all

[rakibabu]

Problem not resolved at 2.3.5 migration.

[terfex]

the error is in the new database

You need to re-create the clean eav_attribute_set table and/or clean all the migration_ entries in eav_attribute_group

you will need to do this after every errored run

@iphigenie Thanks, your solution worked on 2.3.3. I agree this tool should be able to remove this with the reset option.

[HeiderSati]

Same problem still occurring on Magento 2.3.5, very annoying, I mean why not displaying the offending record being inserted so that we could at least find out what to delete rather than display a very vague error like:

Duplicate entry '9-images' for key 'EAV_ATTRIBUTE_GROUP_ATTRIBUTE_SET_ID_ATTRIBUTE_GROUP_CODE'

I know, looking at this, it's the "eav_attribute_group" table, column: "attribute_set_id", ... but the "9-images" does not exists anywhere, neither does the "9" nor the "images" in any of these, ...

Removing the 'Migration_' entries from the eav_attribute_group table and also cleaning the "eav_attribute_set" table results in another error:

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')) ORDER BY entity_attribute_id ASC
LIMIT 2524' at line 1, query was: SELECT eav_entity_attribute.* FROM eav_entity_attribute WHERE (entity_attribute_id >= 0) AND (attribute_id IN ()) ORDER BY entity_attribute_id ASC
LIMIT 2524

I've been working on this since morning, absolutely frustrating and time consuming, what happened to "good old test" before releasing?!!?...

[HeiderSati]

Ok,... Debugged the damn thing,... now understood what's going on:

1) Download the latest "Data Migration Tool" (I'm on Magento CLI 2.3.5 at the time I wrote this response)

2) Goto vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php

3) Go to line 386, you should see the following code:

    foreach ($sourceRecords as $recordData) {
        $recordData['attribute_group_id'] = null;
        $sourceRecord = $this->factory->create(['document' => $sourceDocument, 'data' => $recordData]);
        $destinationRecord = $this->factory->create(['document' => $destinationDocument]);
        $recordTransformer->transform($sourceRecord, $destinationRecord);
        $recordsToSave->addRecord($destinationRecord);
    }

4) Logic of the above is: Read the source record, reset the attribute_group_id ( = null ) and then inject into the new DB, this generate a new ID for the NULL'd record to be used later... seems pretty straight-forward?!... not exactly... the issue is that due to previous import/migration (which is what all of us developers do), the chances are; these records are already there on the DEST DB, doesn't matter if they are using a different ID it would still fail due to the constraint that says "contents (not ID) must NOT be duplicated"..

In other words, the addRecord() is adding the record into a resultset, but the attribute_group_id=NULL is telling the system to create a new one since there is no ID.

5) Before the fix:
IMPORTANT NOTE: I am always totally against editing release/firmware files of Magento Original Releases, however, unfortunately on this occasion, it's a must as the out-of-the-box does not work with an existing set of data unless you are migrating into a brand-new/fresh M2 install, which is impractical since you would build things and then re-migrate and then go live.

IMPORTANT: From a high level perspective; a gentle note: I take no responsibility for any damage you may cause to your data, use this procedure on "your own risk":

6) Now the Fix:
The new logic will be:
A) Read the Source record (same)
B) Check on the destination DB; "Do I have it already"?
C) If the record does not exist, then create it (using the same logic as above)
D) If however the record is already there, then update it instead instead. The current bug is that the system will fail if the SQL server thinks that the record is there using the INDEXes.

Now, I am not worried about the ID being used all over the place with other historic EVA records, mainly because I already did a previous migration on a previous update and the EVA data (master items) should be the same, if however you haven't done a previous data migration then you probably should upgrade into a brand new vanilla Magento 2 release that has not been used yet. My bet is that you are probably here because you are having the same problem and that you have done this before but can't do this again like what we are all experiencing.

7) Looking into the database, the following are the current indexes:

EAV_ATTRIBUTE_GROUP_ATTRIBUTE_SET_ID_ATTRIBUTE_GORUP_CODE: (attribute_set_id + attribute_group_code) UNIQUE
EAV_ATTRIBUTE_GROUP_ATTRIBUTE_SET_ID_ATTRIBUTE_GORUP_NAME: (attribute_set_id + attribute_group_name) UNIQUE

In English the attribute is considered a duplicate if its (set_id+group_code) or (set_id+group_name) are there already.
Therefore, to pull the record from the DEST database for comparison, we can safely load the setID+GroupName since this is what the $recordData is coming back with (i.e. NAME but no CODE) which is fine.

8) Now the fix in detail, replace the above foreach(...) block with the following:

    foreach ($sourceRecords as $recordData)
    {

// Heider: Fix Start:
// Replacee:
//
// $recordData['attribute_group_id'] = null;
// $sourceRecord = $this->factory->create(['document' => $sourceDocument, 'data' => $recordData]);
// $destinationRecord = $this->factory->create(['document' => $destinationDocument]);
// $recordTransformer->transform($sourceRecord, $destinationRecord);
// $recordsToSave->addRecord($destinationRecord);

//
// With:
//
$PageSize = $this->destination->getRecordsCount($documentName);
$IDField = null; // 'attribute_group_id'; // null; // $this->source->getIdentityField($documentName);

        $destAdaptor = $this->destination->getAdapter();
        $destRecord = $destAdaptor->loadPage
        (
              $documentName
            , 0
            , $this->destination->getRecordsCount($documentName)
            , null
            , null
            , new \Zend_Db_Expr(sprintf('attribute_set_id = %s and attribute_group_name = \'%s\'', $recordData["attribute_set_id"], $recordData["attribute_group_name"] ))
        );


        if (! $destRecord)
        {   // Follow the exact old process... (i.e. add it into a Q and then SaveRecords() below after this ForEach...
            $recordData['attribute_group_id'] = null;
            $sourceRecord = $this->factory->create(['document' => $sourceDocument, 'data' => $recordData]);
            $destinationRecord = $this->factory->create(['document' => $destinationDocument]);
            $recordTransformer->transform($sourceRecord, $destinationRecord);
            $recordsToSave->addRecord($destinationRecord);
        } else {
            if (is_array($destRecord))
            {
                $recordData['attribute_group_id'] = $destRecord[0]['attribute_group_id'];   // We will always ONLY have 1 item in the array if anything comes back from SQL due to the UNIQUE Indexing
            } elseif (is_object($destRecord)) {     // Should NEVER arrive to this step, but just in case, extra protection...
                $recordData['attribute_group_id'] = $destRecord['attribute_group_id'];
            }

            // We should always have something in $recordData['attribute_group_id'], but below another safety, just in case...
            if ( ($recordData['attribute_group_id']) && ($recordData['attribute_group_id'] > 0) )
            {
                $destAdaptor->updateChangedRecords($documentName, $recordData);         // Update it on the fly.
            }
        }

// Heider: Fix End:

    }

// Heider: Added another Safety here, maily because we could have a case whereas all records already there giving $recordsToSeve = null:
if ($recordsToSave)
{
$this->saveRecords($destinationDocument, $recordsToSave);
}

9) The above code will work 100%, also another note that the SOURCE and DESTINATION implementation of the DB classes aren't the same, i.e. the reason the code is slightly longer than it should be mainly for extra protections in place, for some reason the $this->source->getRecords does not work in the same way as $this->destination->getRecords,... don't know why someone in their own wisdom decided to cripple the WHERE clause from the DEST object causing the $destRecord to return everything!?... Therefore, the fix was to go back into the Abstract class, understand how it's working and then use it's getAdaptor() to do the reading instead... Anyway, the code works and now my script is passing this ugly stage...

Now, there is another issue that I will looking into (nothing to do with this one, but related in the same way) whereas the error has ... AND (attribute_id IN ()) ORDER BY... it's another annoying problem that I need to resolve to complete the migration...

I hope this helps.

Regards
Heider

[HeiderSati]

After the above fix, for people having the other problem of:

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')) ORDER BY entity_attribute_id ASC
LIMIT 2524' at line 1, query was: SELECT eav_entity_attribute.* FROM eav_entity_attribute WHERE (entity_attribute_id >= 0) AND (attribute_id IN ()) ORDER BY entity_attribute_id ASC
LIMIT 2524

In Mysql.php line 91:

SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ')) ORDER BY entity_attribute_id ASC
LIMIT 2524' at line 1

I fixed this one also on the following tickets:
https://github.com/magento/data-migration-tool/pull/829
https://github.com/magento/data-migration-tool/issues/827

[abdelkebir]

You need maybe (depends on your case) to drop the following tables:

drop table migration_backup_eav_attribute_group;

drop table migration_backup_eav_attribute_set;

drop table migration_backup_eav_entity_type;

Worked for me, but it depends on your case, I think every case is special.

[anatoliidolia]

1) DELETE FROM eav_attribute_set WHERE attribute_set_name != 'Default';
2) You need to migrate to the fresh database, reinstall M2 DB, don't use -r

Have a nice migration

[kanevbg]

@anatoliidolia What is the point of deleting your data before migration?

[anatoliidolia]

@kanevbg When you restart the process, the error described in point 1 occurs
This can be fixed by running my command: DELETE FROM m2_DB.eav_attribute_set WHERE attribute_set_name != 'Default';
Or use a fresh base