Contao: Error: clear cache, or start installtool from Contao-Manager (1.0.4)

[wernergraser]

manager 1.0.4 / Contao 4.4.26 / PHP7.2.10

After Contao Update 4.4.26 is no more possible to start the installtool or clear Prod.Chache
install:
`request.INFO: Matched route "contao_install". {"route":"contao_install","route_parameters":{"_scope":"backend","_token_check":true,"_controller":"Contao\InstallationBundle\Controller\InstallationController::installAction","_route":"contao_install"},"request_uri":"[...]/contao/install","method":"HEAD"} []

request.INFO: Matched route "contao_install". {"route":"contao_install","route_parameters":{"_scope":"backend","_token_check":true,"_controller":"Contao\InstallationBundle\Controller\InstallationController::installAction","_route":"contao_install"},"request_uri":"[...]/contao/install","method":"GET"} []

app.CRITICAL: An exception occurred. {"exception":"[object] (Symfony\Component\Debug\Exception\OutOfMemoryException(code: 0):
Error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 1126400 bytes) at /../contao/vendor/twig/twig/lib/Twig/Loader/Filesystem.php:145)"} []`

cache:
`$ /opt/plesk/php/7.2/bin/php '-q' '/var/www/vhosts/.../vendor/contao/manager-bundle/bin/contao-console' 'cache:clear' '--env=prod' '--no-warmup' 2>&1
// Clearing the cache for the prod environment with debug
// false
[OK] Cache for the "prod" environment (debug=false) was successfully cleared.
Process terminated with exit code 0
Result: OK
$ /opt/plesk/php/7.2/bin/php '-q' '/var/www/vhosts/.../vendor/contao/manager-bundle/bin/contao-console' 'cache:warmup' '--env=prod' 2>&1
// Warming up the cache for the prod environment with debug
// false
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1236992 bytes) in /var/www/vhosts/.../vendor/twig/twig/lib/Twig/Lexer.php on line 178
In Lexer.php line 178:

Error: Allowed memory size of 134217728 bytes exhausted (tried to allocate
1236992 bytes)

cache:warmup [--no-optional-warmers] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--]

Process terminated with exit code 255
Result: Unknown error`

No matter how much memory I allocate…

[andreheeke]

I can reproduce the issue. Tested in console and Contao Manager.

PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 3044264 bytes) in /var/www/clients/client32/web164/web/vendor/twig/twig/lib/Twig/Loader/Filesystem.php on line 145

[fritzmg]

Your PHP CLI environment has only 512 MiB of memory available, which is not enough.

[andreheeke]

@fritzmg Sure, but which provider offers more than 512MiB of memory at the PHP CLI? Before Contao 4.4.26 it works fine.

[fritzmg]

You should discuss these problems on the Contao Community forum.

[colonelpanick]

Change of allocated memory does not change the outcome. Error is thrown even with over 1GB. Bug is somehow introduced with current update...

[leofeyer]

Since the issue occurs in the Twig lexer and filesystem loader, I assume it has something to do with Twig? @andreheeke How exactly did you reproduce the issue?

[grafzahl]

Same here, when i try to get into

contao/install

with correct Auth Informations it just drops

[2018-10-15 16:12:34] request.INFO: Matched route "contao_install". {"route":"contao_install","route_parameters":{"_scope":"backend","_token_check":true,"_controller":"Contao\\InstallationBundle\\Controller\\InstallationController::installAction","_route":"contao_install"},"request_uri":"http://xxxxx.de/contao/install","method":"HEAD"} [] [2018-10-15 16:12:34] request.INFO: Matched route "contao_install". {"route":"contao_install","route_parameters":{"_scope":"backend","_token_check":true,"_controller":"Contao\\InstallationBundle\\Controller\\InstallationController::installAction","_route":"contao_install"},"request_uri":"http://xxxxx.de/contao/install","method":"GET"} [] [2018-10-15 16:12:36] app.CRITICAL: An exception occurred. {"exception":"[object] (Symfony\\Component\\Debug\\Exception\\OutOfMemoryException(code: 0): Error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 1388544 bytes) at /www/htdocs/xxxxxx/xxxxxx.de/vendor/twig/twig/lib/Twig/Lexer.php:173)"} []
into the log file... Same happens very often when i try to install/uninstall new packages

[andreheeke]

@leofeyer I only tried to update the dependencies in the Contao Manager and via the SSH console. While warming up the cache I got the above mentioned error and the instance was aborted.

[mapfei]

I can reproduce the issue. Tested in console and Contao Manager. Cleared the cache by deleting var/cache but installtool doesn't work either.

[fritzmg]

Does Symfony provide any sort of memory usage profiling for console commands (like Composer does)?

[megmed]

I get the Install Tool running as follows (workaround):

1.) move the complete templates folder (outside the Contao folder)
2.) delete everything under var/cache/
3.) call warmup (shell)... Note! Replace php7-71STABLE-CLI with your version

php7-71STABLE-CLI vendor/bin/contao-console cache:warmup

4.) move templates folder back again

After that I can open the Install Tool

[fritzmg]

@megmed so even if you run the cache:warmup directly on the shell, it only works if you remove the templates/ folder beforehand? Otherwise you get a memory overflow?

[fritzmg]

I have received the templates/ folder of an affected user. In that case the "Eclipse" theme from Premium Contao Themes is used. The templates folder contains 694 files in total.

I can confirm that the PHP peak memory usage during cache:warmup jumps from 29.95 MiB in an empty Contao 4.4.26 installation to around __597.06 MiB__ when those files are present in the templates/ directory, which seems quite a lot. I'll try to analyze this further.

@contao/developers any ideas though why files in the templates/ directory will fill up the memory during cache:warmup (in the prod environment)? Why would files in the templates/ directory affect the Symfony Application Cache warmup at all?

According to the user, this happens with _any_ file present in that directory, not just template files. Also it only started to happen after these package updates:

  - Updating tecnickcom/tcpdf (6.2.22 => 6.2.26): Downloading (100%)
  - Updating symfony/symfony (v3.4.15 => v3.4.17): Downloading (100%)
  - Updating php-http/client-common (1.8.0 => 1.8.1): Downloading (100%)
  - Updating friendsofsymfony/http-cache (2.5.0 => 2.5.2): Downloading (100%)
  - Updating friendsofsymfony/http-cache-bundle (2.4.1 => 2.5.1): Downloading (100%)
  - Updating terminal42/header-replay-bundle (1.5.0 => 1.5.3): Downloading (100%)
  - Updating sensio/framework-extra-bundle (v5.2.0 => v5.2.1): Downloading (100%)
  - Updating composer/ca-bundle (1.1.2 => 1.1.3): Downloading (100%)

So may be some change within symfony/symfony might cause this?

[megmed]

@fritzmg yes even if I run it directly on the shell it only works if I remove the templates folder.

I’m also using the Eclipse theme by PCT.

[grafzahl]

Can Confirm, my instance also uses eclipse, which has quite a big templates folder. I renamed the templates folder to templates2, deleted the cache, and everything worked like it should, after i finished my work, i renamed the folder again - Seems to work for now.

I am curious why the templates folder seems to have an effect on cache:warmup?

[dmolineus]

I can confirm this issue. Twig scans the templates folder since Symfony 3.4 (See docs). Is it on purpose that this option is not changed?

I can workaround the issue with following config.yml:

twig:
  default_path: '%kernel.project_dir%/app/Resources/views'

Not sure if it's a general issue or an issue of PCT themes. According to the forum and also in my project, a theme from PCT is used. Maybe @timgatzky is already aware of this issue.

[Toflar]

Ah I see, Symfony puts twig templates into templates as well. That's the issue we're having here. I guess it only expects to find twig templates there, nothing else.

[fritzmg]

Not sure if it's a general issue or an issue of PCT themes.

My guess is it's a general issue and the problem is simply exacerbated by the sheer amount of files that the PCT theme puts into the templates folder.

[fritzmg]

Btw. this does not happen with symfony/symfony 3.4.15. It happens with 3.4.16 or 3.4.17.

[aschempp]

Looks like this was a bugfix in Symfony, prior to 3.4.16 the cache warmer did not check the templates folder: https://github.com/symfony/symfony/pull/27764

[fritzmg]

Indeed. So is adding

twig:
  default_path: '%kernel.project_dir%/app/Resources/views'

to the manager-bundle (and the documentation of the core-bundle) the correct solution then?

[dmolineus]

Indeed. So is adding default_path: '%kernel.project_dir%/app/Resources/views'

https://github.com/symfony/twig-bundle/blob/a2d8c6e9f5735d6d3824e85ddeb6b3b0892f8bb3/TemplateIterator.php#L66

It could be set to any folder which does not exist. The question is: Shouldn't twig only scan files which has *.twig as file extension or doesn't twig care about the file extension anyway?

[Toflar]

Contao checks the templates folder during cache:clear (or more specifically during cache:warmup which is automatically called when you call cache:clear).
It recursively searches the whole template folder to locate all the template names and their paths and puts it into its cache.
For some reason, there seems to be a memory leak there. I could profile this but I need the templates folder for that. Or @fritzmg does it :-D

[Toflar]

Yeah it definitely is a general issue. Symfony tries to parse all the templates during cache:warmup so they are compiled and ready.
I guess it doesn't check file extensions so it loads the files into memory, trying to parse the template. Which means it e.g. loads template images to try and parse it.
This is not going to be an easy one to fix...

[fritzmg]

Btw. wouldn't

twig:
  default_path: null

be more correct, technically? app/Resources/views is hard coded in the twig-bundle anyway and presumably this way you disable the default path.

[fritzmg]

Btw. wouldn't

twig:
  default_path: null

be more correct, technically? app/Resources/views is hard coded in the twig-bundle anyway and presumably this way you disable the default path.

That does not work unfortunately:

Fatal error: Uncaught TypeError: Argument 1 passed to Symfony\Component\Config\Resource\FileExistenceResource::__construct() must be of the type string, null given, called in …/vendor/symfony/twig-bundle/DependencyInjection/TwigExtension.php on line 119 and defined in …/vendor/symfony/config/Resource/FileExistenceResource.php:31
Stack trace:
#0 …/vendor/symfony/twig-bundle/DependencyInjection/TwigExtension.php(119): Symfony\Component\Config\Resource\FileExistenceResource->__construct(NULL)
#1 …/vendor/symfony/dependency-injection/Compiler/MergeExtensionConfigurationPass.php(76): Symfony\Bundle\TwigBundle\DependencyInjection\TwigExtension->load(Array, Object(Symfony\Component\DependencyInjection\Compiler\MergeExtensionConfigurationContainerBuilder))

[dmolineus]

Symfony will deprecate the app/Resources/views template path and rely on twig.default_path in the future. We should keep that in mind than finding a solution.

https://symfony.com/blog/new-in-symfony-4-2-important-deprecations#deprecated-a-template-directory

[leofeyer]

So we create a PR to adjust Symfony so it scans for *.twig files only?

[m-vo]

That sounds reasonable to me. Symfony uses the file extension to distinguish which engine to use (php / twig) anyway. I'm wondering if these defaults can be overwritten however.

[tschero]

I can confirm this issue using a well filled templates folder. A big Problem if it´s not possible for a user to rise the memory_limit.

[fritzmg]

@tschero in the mean time you can use the aforementioned workaround.

[leofeyer]

As discussed in Mumble on December 20th, the Symfony template cache is not limited to .twig files, therefore we want to implement Fritz' workaround in the managed edition.

[fritzmg]

Credit goes to @dmolineus https://github.com/contao/contao/issues/108#issuecomment-431815138 😁

[leofeyer]

Implemented in 9a7ae410540b6a946e2c6a25d5fbbe1069049f14.