Hhvm: Stack overflow on certain DB requests

[craigcarnell]

Loading certain CMS pages on Magento for example, this is difficult to narrow down at the moment

\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Adapter/Abstract.php on line 980
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Adapter/Abstract.php on line 980
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Adapter/Abstract.php on line 980
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Adapter/Abstract.php on line 980
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843
\nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140729142916/lib/Zend/Db/Select.php on line 843

[LiraNuna]

An isolated test case would be very helpful in getting us to fix this.

[r3wt]

Allow me to interject, but i believe this is just another manisfestation of a previous issue with databases. the common denominator seems to be a call to clone. i believe that data is failing to be cloned from one pointer to another. I have nothing to back that up other than a hunch and personal experience of errors occuring when trying to pass a mysqli instance to a function with global.

[craigcarnell]

Magento is unfortunately a beast, perhaps @danslo has some idea

[danslo]

Sorry, I have no idea. Can't reproduce it on a clean install so it's hard to say what this might be.

[karpa13a]

@danslo may be add some kind of debug info into?

[craigcarnell]

I've added some additional logging to my hphp and the product page appears normally without issue, but clearly for some reason on occasion there is a stack overflow. I've matched 2 errors as an example using the timestamp:

hphp error.log:

[Thu Jul 31 12:35:00 2014] [hphp] [12467:7fcb01bff700:1370:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843

Magento system.log:

system-uk.log:2014-07-31T12:35:00+00:00 ERR (uk_default): /product-page?utm_content=&utm_term= 
system-uk.log-REQUEST:   GET|{"utm_content":"","utm_term":""}
system-uk.log-TIME: 7.778518s
system-uk.log-ADDRESS: xxx
system-uk.log-USER AGENT: Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53
system-uk.log-FILE: 20140731114142/app/code/core/Mage/Core/functions.php:247
system-uk.log-Error: Stack overflow  in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843

Another one:

[Thu Jul 31 15:06:33 2014] [hphp] [12467:7fcaf1fff700:3744:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 1021

Associated request:

system-uk.log:2014-07-31T15:06:33+00:00 ERR (uk_default): /checkout/cart/index 
system-uk.log-REQUEST: -
system-uk.log-TIME: 4.436483s
system-uk.log-ADDRESS: xxx
system-uk.log-USER AGENT: Mozilla/5.0 (Windows NT 6.3; Win64; x64; Trident/7.0; rv:11.0) like Gecko
system-uk.log-FILE: 20140731114142/app/code/core/Mage/Core/functions.php:247
system-uk.log-Error: Stack overflow  in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 1021

Some more errors:

[Thu Jul 31 12:30:50 2014] [hphp] [12467:7fcb01bff700:1319:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 12:32:50 2014] [hphp] [12467:7fcaf67ff700:1337:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 12:35:00 2014] [hphp] [12467:7fcb01bff700:1370:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 12:41:29 2014] [hphp] [12467:7fcaf67ff700:1425:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 12:43:29 2014] [hphp] [12467:7fcb06bff700:1465:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 13:18:30 2014] [hphp] [12467:7fcac9bff700:1821:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Varien/Object.php on line 629
[Thu Jul 31 13:25:32 2014] [hphp] [12467:7fcaea7ff700:2075:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 980
[Thu Jul 31 13:34:50 2014] [hphp] [12467:7fcb0efff700:2316:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 1021
[Thu Jul 31 13:35:18 2014] [hphp] [12467:7fcab3bff700:2109:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Varien/Db/Select.php on line 281
[Thu Jul 31 13:35:30 2014] [hphp] [12467:7fcaea7ff700:2238:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 13:35:32 2014] [hphp] [12467:7fcae97ff700:2243:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 980
[Thu Jul 31 13:36:03 2014] [hphp] [12467:7fcad47ff700:2138:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 13:37:30 2014] [hphp] [12467:7fcac9bff700:2115:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 13:37:30 2014] [hphp] [12467:7fcab3bff700:2115:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 13:39:58 2014] [hphp] [12467:7fcb01bff700:2344:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 13:40:29 2014] [hphp] [12467:7fcb143ff700:2423:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 13:43:06 2014] [hphp] [12467:7fcb0efff700:2469:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 980
[Thu Jul 31 13:45:21 2014] [hphp] [12467:7fcaea7ff700:2412:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 13:51:49 2014] [hphp] [12467:7fcab3bff700:2373:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 13:53:53 2014] [hphp] [12467:7fcabefff700:2444:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Select.php on line 843
[Thu Jul 31 14:05:41 2014] [hphp] [12467:7fcb01bff700:2746:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 14:07:41 2014] [hphp] [12467:7fcab3bff700:2598:000001] [] \nFatal error: entire web request took longer than 120 seconds and timed out in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Statement/Pdo.php on line 228
[Thu Jul 31 14:23:39 2014] [hphp] [12467:7fcaf67ff700:3041:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 1021
[Thu Jul 31 14:39:20 2014] [hphp] [12467:7fcac9bff700:3133:000001] [] \nFatal error: Stack overflow in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 980

[craigcarnell]

These errors I cannot reproduce locally or on a dev box but only show on production

[danslo]

What version of Magento is this by the way? And HHVM version?

[craigcarnell]

Magento Enterprise 1.14.0.1

HipHop VM 3.3.0-dev (rel)
Compiler: heads/master-0-gaf1ed367ff13ec0e72e5f0b9305f4474517b10da
Repo schema: 0c44dc5b77df247d699591b424bdd2ceb93220c2

[craigcarnell]

I have checked the logs on my PHP node (I have a LB, one running HPHP, the other PHP) and I see similar things. Therefore I don't think this is perhaps a HPHP issue.

my PHP host I also see:

2014/07/31 15:03:06 [error] 29370#0: *46113 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx, server: xxx, request: "GET /catalogsearch/result/?q=xxx HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "xxx", referrer: "xxx"

error.log:2014/07/31 11:47:07 [error] 29369#0: *2536 FastCGI sent in stderr: "PHP message: PHP Fatal error:  Maximum execution time of 120 seconds exceeded in /var/www/vhosts/magento-capistrano/releases/20140731114142/lib/Zend/Db/Adapter/Abstract.php on line 859" while reading response header from upstream, client: 91.125.14.244, server: xxx, request: "GET /catalogsearch/result/index/?limit=all&q=xxx HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "xxx", referrer: "xxx"

[danslo]

You could try logging SQL by enabling $_debug and $_logAllQueries in Varien_Db_Adapter_Pdo_Mysql and then try to match the query times with the overflows. There's also a $_logCallStack option but be careful because your log file may grow extremely big.

Other than that, I'd love to help but I don't think I can do much else until there's a reproducable scenario.

[fredemmott]

Closing, still needs more info (https://github.com/facebook/hhvm/wiki/Human-Timeouts) - will reopen if provided.

[TheBlaze]

Hello,

We are also running Magento on HHVM and ran into the same error:

[Mon Mar 21 12:32:35 2016] [hphp] [18947:7f98d8bff700:122:000001] [] \nFatal error: Stack overflow in /mnt/web/magento_1141/htdocs/lib/Zend/Db/Select.php on line 843

We can repeat the error. If you have a certain frontend cookie the error happens every time, on every request. The problem is, that it only happens on the production server. What can I do, to help solve this? What information do you need and how can I get it?

Thank you,
Blaž

[paulbiss]

Knowing the PHP stack trace would be a start, if there's a builtin you're calling that sets this off that would also be good to know.

[aftabnaveed]

This issue still exists, and consistent with cookies.

[babar-ali]

Clearing magento session resolve this issue, but its temporary solution.
Not yet found the reason how these cookies generated and how to stopped them permanently.

[ryne-andal]

Suffering through the same issue here. Not 100% certain on the specific cause, but clearing sessions resolves the issue temporarily, then it happens again.

[aftabnaveed]

@ryne-andal try clearing your session container if you are using Redis or Memcache then flush its entire cache.

We used the HHVM for 3 years and it worked well for us, I guess it is now time to move on.
Since it seems to have no strong user base Facebook will prioritize the issues which affect them. It's better to start thinking about moving to PHP7 and you won't be disappointed.

[ryne-andal]

Yeah I've begun transitioning to PHP7 as well, @aftabnaveed. Upon clearing all caching and sessions, it still occurs almost immediately. I'm suspecting one of the third-party modules aren't playing nicely with HHVM, since it doesn't occur on my local running PHP7. I'll finish the transition and deploy accordingly, was just hoping to resolve the issue with a minimal amount of work since it's a legacy system.

Appreciate the assistance and good luck with migrations to PHP7.

[aftabnaveed]

@ryne-andal we came with a workaround though. If you are using nginx then catch the 500 error response from the upstream and proxy your request internally to another script which clears the corrupted session for you and then redirect the user silently.

I know that is a dirty hack but it has been a lifesaver for us during last holiday season and we are still relying on it. The only drawback to that solution is that the customer gets kicked out and has to re-login.

[lexidor]

I am going over old issues on this repository, to see which ones are very unlikely to be worked on.

This issue has been created some time ago and there is not enough information in this issue to reproduce this issue on a modern build of hhvm. I am closing this issue, since the chances that the owner of this issue wants to invest effort into this seems small. If you are reading this and have a way to reproduce this issue on a modern hhvm version, please reopen the issue and add this repro.