Magento2: Integration tests create stub modules in app/code

[Vinai]

When even a single integration test is executed, the following files and directories are created:

app/code/Magento
├── TestModuleDirectoryZipCodes
│   ├── etc
│   │   ├── module.xml
│   │   └── zip_codes.xml
│   └── registration.php
├── TestModuleFakePaymentMethod
│   ├── Gateway
│   │   └── Command
│   │       └── DoNothingCommand.php
│   ├── etc
│   │   ├── config.xml
│   │   ├── di.xml
│   │   └── module.xml
│   └── registration.php
└── TestModuleSample
    ├── composer.json
    ├── etc
    │   └── module.xml
    └── registration.php

(Note: the module Magento_TestModuleFakePaymentMethod is not generated in Magento 2.2.0).
This is a problem because running integration tests should not affect the installed Magento instance.

Preconditions

Reproduced on fresh installations of Magento 2.2.0 and 2.2.2. I haven't tried on Magento 2.2.1.

Steps to reproduce

1. Install Magento, e.g. composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition . and so on.
2. Run all integration tests or create a custom module with nothing but a single integration test.
3. Run the test, e.g. /var/www/example/vendor/phpunit/phpunit/phpunit --configuration /var/www/example/dev/tests/integration/phpunit.xml /var/www/example/app/code/Example/Foo/Test/Integration

Expected result

1. Only the specified tests should run in isolation.
2. Potential production code should not be affected.

Actual result

The modules are created in app/code/Magento potentially affect production code and are created independently if they are used in any test.

It should be possible to create the modules under dev/tests/integration/tmp in the sandbox directory and register them in the test process with the component registrar instead, leaving the app/code directory unmodified.

[magento-engcom-team]

Hi @Vinai. Thank you for the report. We are moving your feature request to the special project. You can track this issue here: https://github.com/magento-engcom/community-features/issues/38

[avstudnitz]

@magento-engcom-team This is not a feature request, but clearly a bug. Please reopen and fix.

[miguelbalparda]

Reopening this, sorry about that.

[avstudnitz]

The problem with this behavior: I can add app/code/Magento to my .gitignore, but even if I do, and I call bin/magento setup:upgrade some time after running the integration tests, the modules are being added to app/etc/config.php which is not in my .gitignore (and shouldn't be).

[ghost]

@Vinai, thank you for your report.
We've acknowledged the issue and added to our backlog.

[jalogut]

There is another mayor problem with that. If you use a build pipeline, these modules might go to your production environment as you create a zip of the code after successfully executing integration tests. If that happens, checkout gets broken in your production environment.

See: https://github.com/magento/magento2/issues/12679

Of course, people can remove these modules before creating the artifact to deploy but it would help if they will not be created at all. People will probably face this issue the first time and they will get a broken checkout for several hours as the bug is really difficult to figure out.

[schmengler]

The problem is even bigger as soon as queue tests come into play:

    'Magento_TestModuleAsyncAmqp' => 1,
    'Magento_TestModuleDirectoryZipCodes' => 1,
    'Magento_TestModuleFakePaymentMethod' => 1,
    'Magento_TestModuleMessageQueueConfigOverride' => 1,
    'Magento_TestModuleMessageQueueConfiguration' => 1,
    'Magento_TestModuleSample' => 1,
    'Magento_TestModuleSynchronousAmqp' => 1,

Those test modules fire up a "few" queue consumers:

$ bin/magento queue:consumer:list
quoteItemCleaner
inventoryQtyCounter
queue.for.multiple.topics.test.c.deprecated
mixed.sync.and.async.queue.consumer.deprecated
queue.for.multiple.topics.test.d.deprecated
deprecatedConfigAsyncStringConsumer
deprecatedConfigAsyncMixedConsumer
deprecatedConfigAsyncBoolConsumer
deprecatedConfigSyncBoolConsumer
overlappingConsumerDeclaration
synchronousRpcTestConsumer.deprecated
queue.for.multiple.topics.test.c
queue.for.multiple.topics.test.d
queue.for.multiple.topics.test.y
queue.for.multiple.topics.test.z
mixed.sync.and.async.queue.consumer
mtmh.queue.1.consumer
mtmh.queue.2.consumer
wildcard.queue.one.consumer
wildcard.queue.two.consumer
wildcard.queue.three.consumer
wildcard.queue.four.consumer
consumer1
consumer2
consumer3
consumer4
consumer5
synchronousRpcTestConsumer
RemoteServiceTestConsumer
queue.for.multiple.topics.test.a
queue.for.multiple.topics.test.b

That's one PHP process per entry, hogging resources...

[avstudnitz]

I am working on this #squashtoberfest

[magento-engcom-team]

Hi @Vinai. Thank you for your report.
The issue has been fixed in magento/magento2#18459 by @avstudnitz in 2.3-develop branch
Related commit(s):

• 4e44cb93fd18ae2fcd8fbad134c6aa956781425e
• 03667812817331309c81bfebcb643f8ee0d389c4
• 140d27b95a5e3517fff8c58cabb93628e97f42f7
• 882c6916873c0c8f4ad160413944cc392a228fc4
• 2ff2c4e5048794635c16c2294ed1684b4023a6fd
• d039298d36c86141b0281e1d947d46a4cc748e30
• efbf1545028330517ea82e710ccef188db8264f9

The fix will be available with the upcoming 2.3.3 release.