Magento2: Travis CI breaks on out of the box composer build

[barbazul]

When building a project from the composer create-project command as suggested in the documentation, the resulting build will not be green in Travis CI. I know there is some handling necessary (like some execution permissions and building your auth.json) but when all is set, still all kinds of tests fail

Preconditions

Magento 2.2 installed from composer repository
PHP 7.1

Steps to reproduce

1. Create project: composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition magento2
2. Create a Github repository (SemExpert/magento2-composer-built in my example) and setup Travis CI
3. Initialize git: cd magento2 && git init && git remote add origin https://github.com/SemExpert/magento2-composer-built.git
4. Give execution permission to Travis files: chmod +x dev/travis/*.sh
5. Edit .travis.yml, add auth.json setup according to https://gordonlesti.com/magento2-extension-development-with-travis-ci/
6. Edit .travis.yml, remove PHP 7.0 from matrix and add execution permission to dev/travis/before_script.sh in before_script section
7. cd magento2 && composer install
8. cd magento2 && git add . && git commit -m "commit" && git push

Expected result

1. Build should be green

Actual result

1. https://travis-ci.org/SemExpert/magento2-composer-built

[orlangur]

Please provide a place in official documentation suggesting Travis usage like described.

[barbazul]

@orlangur There is no mention on how to use Travis in the documentation. I am trying to self-discover this process in the hopes of actually writing such missing documentation. My current attempt is trying to use the travis file as it is provided by Magento, ideally, without modifications.

The few modifications made were to circumvent the initialization errors as they showed up, up to the point where tests were actually run.

1. before_install.sh and before_script.sh would not run without manually granting execution permissions
2. I built Magento using PHP 7.1 so some of the installed dependencies would not run in PHP 7.0 therefore I removed PHP 7.0 from the matrix
3. Added auth.json with my credentials to be able to pull packages from Magento repository

Even if this attempt is completely invalidated and Magento does no expect developers to use the provided .travis.yml and instead create their own, still, all tests fail when running them as described in this documentation http://devdocs.magento.com/guides/v2.2/test/testing.html in the same ways described in the linked Travis report.

At this moment, the whole test suite is of little use to agencies trying to build a CI process (using Travis or any other tool) as it fails out of the box.

Summary of the errors, for your convenience:

Unit tests 1 error 6 failures

• Magento\Setup\Test\Unit\Module\Di\Code\Reader\FileClassScannerTest::testClassKeywordInMiddleOfFile
• Magento\ProductVideo\Test\Unit\Controller\Adminhtml\Product\Gallery\RetrieveImageTest::testExecute
• Magento\Framework\Api\Test\Unit\Code\Generator\GenerateMapperTest::testGenerate
• Magento\Framework\Code\Test\Unit\Generator\InterfaceGeneratorTest::testGenerate with data set "Valid data" (array(), '', '')
• Magento\Framework\ObjectManager\Test\Unit\Code\Generator\FactoryTest::testGenerate
• Magento\Framework\ObjectManager\Test\Unit\Code\Generator\GenerateRepositoryTest::testGenerate with data set #0 ('Magento\Framework\ObjectManag...sitory', 'Magento\Framework\ObjectManag...Sample', 'SampleRepository')
• Magento\Framework\ObjectManager\Test\Unit\Code\Generator\GenerateRepositoryTest::testGenerate with data set #1 ('Magento\Framework\ObjectManag...sitory', 'Magento\Framework\ObjectManag...Sample', 'TSampleRepository')

Static tests 2 files with 39 errors

• vendor/shopialfb/facebook-module/Controller/Adminhtml/Index/Index.php
• vendor/shopialfb/facebook-module/view/adminhtml/templates/connect_facebook_view.phtml

JS Spec All green =)

JS Static Extreme number of errors. No summary is provided

* Integration tests* 5 failures

• Magento\Framework\Code\GeneratorTest::testGenerateClassFactoryWithNamespace
• Magento\Framework\Code\GeneratorTest::testGenerateClassProxyWithNamespace
• Magento\Framework\Code\GeneratorTest::testGenerateClassInterceptorWithNamespace
• Magento\Framework\Code\GeneratorTest::testGenerateClassExtensionAttributesInterfaceFactoryWithNamespace
• Magento\Ui\Component\ConfigurationTest::testConfiguration

Funcional tests Timeout

[orlangur]

You cannot expect green builds using Travis configuration the way it is not used (= enforced) in core.

First unit test failure was fixed recently in https://github.com/magento/magento2/pull/12144.

vendor/shopialfb

Not even a part of this repository.

all tests fail when running them as described in this documentation http://devdocs.magento.com/guides/v2.2/test/testing.html in the same ways described in the linked Travis report

Only in case of Composer-based installation which is not a part of CICD loop currently. Feel free to report concrete issues similar to https://github.com/magento/magento2/issues/11230 and/or contribute corresponding fixes as pull requests.