Magento2: app:config:import fails - flag column 'flag_data' is too short ?

[peterjaap]

I got this error; Import failed: Unable to unserialize value, string is corrupted. when running php bin/magento app:config:import to import my app/etc/config.php.

It turns out, the importer does a comparison in \Magento\Config\Model\Config\Importer::import. To do this comparison, it retrieves a 'flag'; the system_config_snapshot flag. This is saved in the database. The problem is; the flag_data column in the flag table in Magento 2 is a TEXT, which is too short; it can only hold 65535 characters! It therefore cuts off the serialized data, making it corrupt.

Maybe this is a solution; ALTER TABLE flag MODIFY flag_data LONGTEXT COMMENT 'Flag Data'; ?

Preconditions

1. Magento 2.2

Steps to reproduce

1. Dump a large enough configuration file and import it (message me for an example file, it contains client data but willing to share with Magento employees) -- you should also be able to create a large config.php by inserting large data sets into the translation table, since it also exports i18n settings.

Expected result

1. It should import app/etc/config.php

Actual result

1. Message; Import failed: Unable to unserialize value, string is corrupted.

[peterjaap]

I circumvented this issue by altering flag_data to a LONGTEXT (from a TEXT) and invalidating the hashes so it will force import my config file; UPDATE flag SET flag_data = '{"system":"","scopes":"","themes":""}' WHERE flag_code = 'config_hash'. Flush your cache after running this query.

Another workaround for me was to remove 'i18n' and 'system' from app/etc/config.php, run UPDATE flag SET flag_data = '{}' WHERE flag_code = 'system_config_snapshot'; and php bin/magento app:config:import.

[peterjaap]

Additional thought; app:config:dump dumps EVERYTHING. Maybe the CLI command needs some extra features to be able to only export it partially (scopes, system, themes, i18n) ?

I'm thinking of leaving out system and i18n by default.

Especially the i18n part can grow very big when somebody has used the inline translation functionality a lot, thereby hitting the MySQL TEXT character limit quite soon.

[peterjaap]

Also, bin/magento setup:upgrade [--keep-generated] runs app:config:import. But when app:config:import fails (see here), setup:upgrade still gives exit code 0. Because it doesn't return a non-zero exit code, my deployment continues even though there is something amiss!

[heldchen]

Additional thought; app:config:dump dumps EVERYTHING.

ideally, there would be a mode that only dumps differences from the current config to the default values.

[peterjaap]

To create a large enough config.php, use https://lipsum.lipsum.com/ to generate a minimum of 66000 bytes of text and place it for example in the default_custom_instructions field in config.php.

Then run php bin/magento app:config:import and run php bin/magento app:config:import again;

Processing configurations data from configuration file...
Import failed: Unable to unserialize value, string is corrupted.

[peterjaap]

Here's a vanilla M2.2 file with a bunch of lipsum; https://gist.github.com/peterjaap/9db722c0c698a1493c61b77dd16af415

[giacmir]

I had this issue on a production website where my serialized config was 71,347 characters long against the 65,535 chars allowed by a TEXT field.

[wejobes]

I also had this issue recently. We are getting ready to deploy a site into production and we have been testing using the pipeline deployment methods as mentioned in the Magento 2.2 docs. To get around the issue, I made a quick, small module that updates the flag_data field to be a MEDIUM_TEXT field. Once in place, and I cleared out the config data in the flag table, everything deployed as expected.

[andy-rush-beerhawk]

I was also having this problem with app:config:import
1) i counted the number of characters in the truncated json before i made the changes: 65534
2) updated the table:
ALTER TABLE flag MODIFY flag_data LONGTEXT;
UPDATE flag SET flag_data = '{"system":"","scopes":"","themes":""}' WHERE flag_code = 'config_hash';
UPDATE flag SET flag_data = '{}' WHERE flag_code = 'system_config_snapshot';
3) Then I ran app:config:import >>> the config import worked and the flag table was updated
5) then I was curious, how large is our config, so i went to count the json again, have a guess how many characters there were? 65534.

I have a feeling this issue may not be fixed with an sql update.

[magento-engcom-team]

@peterjaap, thank you for your report.
We've acknowledged the issue and added to our backlog.

[patvdleer]

I'm having the exact same thing but with setup:upgrade.

I did the following to come to the same conclusion;
Looking into Magento\Framework\Serialize\Serializer class Serialize in the function unserialize I first tried to var_dump the string, since it's pretty long I looked at writing it to a file and validating it online but that failed and notice it got cut off. var_dump(strlen($string)); reveals the length to be 65535.

--EDIT0--
I have deleted my config.php and regenerated it (via module:enable --all) .Strange enough, my config.php is about ~4.8kb, and my env.php is about ~11.7kb.

--EDIT1--
Turn out I read over a comment of @peterjaap , https://github.com/magento/magento2/issues/11657#issuecomment-338918089 , after cleaning the flag table it seems to work again.

[lracicot]

Suggestion for the fix: why store all the configurations in the flag? Why not just a hash of the string?

[gwharton]

Seeing this is 2.2.7
Manually changing flag table flag_data field to MEDIUMTEXT or LONGTEXT seems to fix it for me.

[mattegalli]

I fix the problem just updating the flag_path of _system_config_snapshot_ to NULL in flag table.
Then re-run app:config:import

[MichaelThessel]

Seems like this is still a problem 2 years later (2.3.2). This is especially sad as all it takes to fix this is to change a column.

[hostep]

It's got picked up again this week luckily: https://github.com/magento/magento2/pull/13580 🙂

[magento-engcom-team]

Hi @peterjaap. Thank you for your report.
The issue has been fixed in magento/magento2#13580 by @aschrammel in 2.3-develop branch
Related commit(s):

• 7cde2fc890c9f43d82b7aaa0f963994d3e0565e3
• 83371e140b0af24fb36e1bd338027a8a01745f9a
• c5f50a8a6fb71785a04bc16af00761bef864d8e6

The fix will be available with the upcoming 2.3.4 release.

[pmsteil]

On 2.3.3 this workaround was successful:

ALTER TABLE flag MODIFY flag_data LONGTEXT;
UPDATE flag SET flag_data = '{"system":"","scopes":"","themes":""}' WHERE flag_code = 'config_hash';
UPDATE flag SET flag_data = '{}' WHERE flag_code = 'system_config_snapshot';
bin/magento app:config:import