Magento2: DB upgrade freezes due to filesystem traversal

[sshymko]

When Magento is running on multiple web-nodes, media files need be shared between them. One way to do it is to mount the directory /pub/media as a network filesystem, such as NFS or EFS. In such scenario file system operations become way more costly. It was discovered that the DB upgrade unnecessarily traverses files in the media directory causing it to hang up for a long time.

Preconditions

1. Magento 2.1.3 installed via Composer
2. Media directory /pub/media is mounted as NFS/EFS filesystem
3. Media directory is large in size, for example, ~7G
4. Magento application mode: production

Steps to reproduce

1. Run database schema and data upgrade:

php bin/magento setup:upgrade

Expected result

1. DB upgrade completes in reasonable amount of time regardless of the media directory size
2. Media directory is not involved into the DB upgrade process
3. No filesystem integrity validations in the production mode

Actual result

1. DB upgrade recursively traverses the media directory to check permissions of each file via \Magento\Framework\Setup\FilePermissions::getMissingWritablePathsForInstallation()
2. DB upgrade hangs up in the middle of execution

The issue is caused by the code in the Setup sub-application that cannot be targeted by any customization mechanisms, including DI and plugins. That makes it impossible to develop a workaround w/o patching the Magento codebase.

[vrann]

@sshymko thank you for report. Magento is working on internal ticket MAGETWO-65186 reported by @samm-git, also based on the live installation.

[andimov]

Ticket MAGETWO-65186 was closed and delivered to the mainline https://github.com/magento/magento2/commit/76a33bd762489d731b4030067df448b71fa33218.
We decided to decrease max depth for recursive permission check to 2 level as the solution.

[hostep]

Thanks @andimov, your change reduced the deploy time from one of our webshops (which uses a loadbalancer with NFS sharing) from 24 minutes to 10 minutes. Where the bin/magento setup:upgrade step previously took around 300 seconds per server, and now only takes 14 seconds per server. Pretty significant change!

[sshymko]

@hostep

setup:upgrade step previously took around 300 seconds per server

Database is shared between web-servers, so the DB upgrade must be done once, not per server.

[hostep]

@sshymko: we rely on setup:upgrade to also modify the app/etc/config.php file, because we currently don't version that file in Magento 2.1.x. That's why we run it on multiple nodes.

But your comment is correct though, if setup:upgrade would only manipulate the database we should only run it on a single server.

[sshymko]

@hostep
Storing the deployment configuration app/etc/env.php and app/etc/config.php under the version control has worked for us very well.

[hostep]

Yes, I know it's recommended to store config.php in version control, we will probably start doing this in Magento 2.2.x. But you should never put env.php in version control, it contains sensitive data (like the database password), and you should never put sensitive data in version control :)