Jetpack: Jetpack JSON API access crashes PHP-FPM running PHP 7.1

[michaelnordmeyer]

Steps to reproduce the issue

• Install Jetpack as the only plugin, enable JSON API, connect self-hosted blog to WordPress.com
• Access self-hosted blog using either WordPress.com or the WordPress.com macOS app to display blog posts

What I expected

Blog posts are displayed.

What happened instead

App didn't display any content, only outlines of soon to be displayed blog posts.

Observation

PHP task crashes on every API access.

This behaviour also happens for other related API access.

Environment

• WordPress 4.7.3
• Jetpack 4.7
• Ubuntu Server 16.10 64bit
• nginx/1.11.9
• PHP 7.1.2-4+deb.sury.org~yakkety+1
• MySQL 5.7.17-0ubuntu0.16.10.1

Error logs

These are the nginx error logs intertwined with php7.1-fpm.log lines for better readability. Hostname was replaced by example.com.

2017/03/08 22:03:45 [error] 24445#24445: *122060 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.89.76, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010624&nonce=nR7WFrlVUR&body-hash=hurPrlvKTkYR0JyNzGVyo79r4LU%3D&signature=UxRZ3JYIGr698YYsa9kpIHEvOFw%3D"

[08-Mar-2017 22:03:45] WARNING: [pool www] child 19742 exited on signal 11 (SIGSEGV) after 23.396837 seconds from start
[08-Mar-2017 22:03:45] NOTICE: [pool www] child 19746 started

2017/03/08 22:03:46 [error] 24445#24445: *122064 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.110, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cMBfZx3i1j&body-hash=AY82TQbx4apsl%2FQ7piLrvLuh4Iw%3D&signature=IxgM%2BxOhIH6dav33yzH3K7aWnCM%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19719 exited on signal 11 (SIGSEGV) after 88.240395 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19747 started

2017/03/08 22:03:46 [error] 24445#24445: *122065 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.118, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=cPD7THgNoe&body-hash=I%2Bryd9dbgRyu0RxrIIWYMKCMA%2FI%3D&signature=ACfhZdsezGbDQNO98JfGnO9lbqQ%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19746 exited on signal 11 (SIGSEGV) after 1.566387 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19749 started

2017/03/08 22:03:46 [error] 24445#24445: *122069 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.0.87.138, server: example.com, request: "POST /xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.1-fpm.sock:", host: "example.com", referrer: "https://example.com/xmlrpc.php?for=jetpack&token=0XhK4%284EfeBjlEf%2A%23XCat%239%5EL%407FKxHC%3A1%3A1&timestamp=1489010626&nonce=DFmVKPQAwe&body-hash=wHY17ONIuFUDZEK3FDddt11BQa0%3D&signature=ZICaFXHuefFbvbVUqB4vYrVi%2FZ4%3D"

[08-Mar-2017 22:03:46] WARNING: [pool www] child 19747 exited on signal 11 (SIGSEGV) after 0.199491 seconds from start
[08-Mar-2017 22:03:46] NOTICE: [pool www] child 19751 started

[jeherve]

cc @georgestephanis who's worked on other PHP 7.1 oddities lately.

[Ipstenu]

This is affecting me too BTW.

[georgestephanis]

@Ipstenu @michaelnordmeyer :

• Can you get any sort of a backtrace before the segfault?
• Does it work if you disable opcache via init_set() ?
• Does it work if you pull back to PHP 7.0?
• Does it work if you disable the 'related posts' module?

[michaelnordmeyer]

@georgestephanis:

• No, my PHP is not a debug build
• Yes
• It worked before switching from PHP 7.0 to 7.1 some weeks ago. If 7.0 would have been installed I would have added test info to the issue in the first place.
• Only Jetpack with JSON API is enabled, no other Jetpack modules or 3rd-party plugins

[Ipstenu]

• No :(
• No
• It works on PHP 7 just fine.
• That happens to be disabled anyway

A quick and amusing check, on PHP 7.1 http://public-api.wordpress.com/rest/v1/sites/ipstenu.org shows up as a 404, but it works fine on PHP 7.

[georgestephanis]

Heard back on an upstream bug report that I believe is related -- I'm going to try and track down what we can do to work around this, but for info for y'all in progress -- https://bugs.php.net/bug.php?id=74213

[camaran]

I'm also have this problem, switch to php 7.0.x and work

[michaelnordmeyer]

After upgrading the environment today to

• Jetpack 4.8.1
• Ubuntu Server 17.04 64bit
• PHP 7.1.4-1+deb.sury.org~zesty+1

there haven't been any crashes anymore. 👍

[camaran]

Hi, i have this error again.
i think the problem is opcache

[jeherve]

@camaran Could you try to update PHP to 7.1.4? It includes the following fix, that may help you: https://bugs.php.net/bug.php?id=74213

Let me know how it goes!

[camaran]

Hi, i installed php 7.1.4 but i still with the problem.
I can open a ticket on jetpack support if it's neccessary :-)

[rraptorr]

I am also experiencing the same crash with PHP 7.1.4 (from deb.sury.org), Wordpress 4.7.3 and Jetpack 4.8.2 running on Debian Jessie.

[jeherve]

@camaran @rraptorr Do you get errors similar to the ones mentioned in the original report above?

[rraptorr]

I am getting segfaults on xmlrpc.php file. The problem is gone when switching back to php 7.0 or removing jetpack. This seems pretty similar to the original report.

I was testing every PHP 7.1 version, since PHP 7.1.0. I was always getting this segfault and I am getting it still with PHP 7.1.4.

[ChrisWiegman]

After some testing I'm hitting the same issue:

Ubuntu 16.04
PHP v7.1.4-1+deb.sury.org~xenial+1
WordPress 4.7.4
Jetpack 4.8.2

The following in wp-config lets me perform a sync with Jetpack for the time being but I'm hoping this doesn't have to be permanent:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

[xyzulu]

php 7.1.4 does not help with this issue
Before it get's closed again, perhaps let a few of us test ;)

[michaelnordmeyer]

Looks like I have the problem again after a while of success with either the WordPress.com and iOS app. 😩

The only configuration change in the last couple of days was removing php-xdebug. The rest were ordinary OS updates and reboots because of an updated kernel.

Selectively turning off the opcache does work for me as well:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

[tbutcaru]

I have raised this issue last week.
If I switch back to php56 Wordpress Android App works, but with php7.1.4 doesn't work.

I have added opcache.enable = 0 to my domain's php.ini, but I cannot see any change.

[tbutcaru]

It is indeed a problem with opcache.
I have disabled opcache by creating a .user.ini file inside Wordpress's root dir containing the following line:
opcache.enable=0
and no more errors, Wordpress Android App works 💃

[AubreyHewes]

Disabling opcache in wp-config.php for xmlrpc-only as @ChrisWiegman suggests fixes the problem;-) Cheers @ChrisWiegman !

The amount of headers sent/header validity still seems to me to be the possible root cause?

https://github.com/Automattic/jetpack/issues/3546
https://github.com/dokku/dokku/issues/2607

My setup is a wp/jetpack via dokku (herokuish) ~ nginx-fpm

If you don't want the extra config in the wordpress config you can also blacklist xmlrpc in the opcache config...

http://php.net/manual/en/opcache.configuration.php#ini.opcache.blacklist-filename

https://github.com/Automattic/jetpack/issues/6106

[camaran]

i think now is fixed with php 7.1.5

[rraptorr]

Well, for me it segfaults just the same with 7.1.5

[jeherve]

Also reported here:
https://wordpress.org/support/topic/php7-1-out-of-memory-json-endpoints-php-on-line-123/

[HansVanEijsden]

Same problems with PHP 7.1.5 indeed. @ChrisWiegman thanks for your temporary workaround.

[kevinlisota]

I can confirm the segfault in PHP-FPM 7.1.5 as well. Easily replicated by hitting the "perform full sync" button on WordPress.com. Crashed every time.

[joendotcom]

3216712-t

[joendotcom]

Also same issue in 3231491-t, user downgraded to PHP 7.0 which resolved the issue.

[Ipstenu]

So far, it looks like this is working for me as well:

if ( defined( 'XMLRPC_REQUEST' ) && true === XMLRPC_REQUEST ) { ini_set( 'opcache.enable', 0 ); }

[xyzulu]

@Ipstenu that isn't really a fix.. it's just a workaround IMHO

[Ipstenu]

I didn't call it a fix. I just said that it appears to be working as I'm one of the people facing this problem. If I don't report back with what has, and has not, worked, how can Automattic or anyone else know if their actual fix is headed in the right direction? :)

[xyzulu]

@Ipstenu Sorry Mika, hope I didn't sound harsh.. this workaround has been mentioned before and will always work due to it disabling the feature in php that is not playing well with Jetpack in this instance. Going back months Automattic are well aware of this. (check the many threads where it is mentioned here on Github if you like)

Again.. no attack intended.. sorry for the wrong use of words

[Ipstenu]

_Appears_ to always work.

More data, confirming the fix works in more environments, and replying to ones own bug reports, is considered good behavior in the open source community. I don't want to be DenverCoder9.

And it's spelled with an A. Not an E.

[xyzulu]

It's an opcache error.. disabling opcache in php 7.1 will always work around this error

[Ipstenu]

If XMLRPC v OpCache is the only part that causes the core dump, sure.

[zinigor]

Thank you all for the input! Just for the record, here's a bug report that I have submitted to PHP about this exact problem. It's not XMLRPC that causes the segfault, but apparently it's some code that gets called as part of the JSON API request that gets sent via XMLRPC.

If you happen to know why that code can cause a segfault with opcache, or maybe how I can narrow down the exact cause, I'll be very grateful:
https://bugs.php.net/bug.php?id=74543

[kevinlisota]

Has anyone tried this with PHP 7.1.6 yet? There is one opcache bug fix in that update.

Actually, I just tried 7.1.6 and the segfault bug still exists.

[lschuyler]

Also reported here 3261025-t

[zinigor]

@kevinlisota just tested on 7.1.6 too, processes still segfault for me when responding to WordPress.com JSON API. What were the issues that were fixed with opcache? Can you post some links please?

[kevinlisota]

Here is the changelog for PHP 7.1.6: http://php.net/ChangeLog-7.php#7.1.6

There is one bugfix to opcache here: https://bugs.php.net/bug.php?id=74596

[HansVanEijsden]

Changelog PHP 7.1.7RC1: https://github.com/php/php-src/blob/PHP-7.1.7/NEWS

I don't know if it fixes it?

. Fixed bug #74663 (Segfault with opcache.memory_protect and
validate_timestamp). (Laruence)
. Revert opcache.enable_cli to default disabled. (Nikita)

[lschuyler]

Also reported in 3266547-t

[michakpl]

It's still not working in PHP 7.1.7

[kevinlisota]

@zinigor Any way to get PHP development team to look at the bug you filed? I see no activity on it

https://bugs.php.net/bug.php?id=74543

[kristarella]

Possible report 3332067-t

[zinigor]

@kevinlisota A good way would be to create a simple way to reproduce the segfault, which I haven't been able to do. You can see how far I have been able to progress in the bug report, but I haven't had more time to dig in. If you are able to produce a contained reproduction case, that would help tremendously!

[lamdayap]

https://wordpress.org/support/topic/jetpack-causing-502-bad-gateway/

User is experiencing similar issues with PHP 7.0.15-0ubuntu0.16.04.4

[aheckler]

614936-zen

Sent them this workaround: https://github.com/Automattic/jetpack/issues/6629#issuecomment-298219239

[mzakariya]

630882-zen

Sent same work around as above. User would prefer to not troubleshoot.

[mzakariya]

635556-zen

Sent same work around as above.

[danjjohnson]

620933-zen sent suggested workarounds.

[mzakariya]

645154-zen

Sent same work around as above.

[railto]

Currently running on PHP 7.1.8 and having same issue. Disabling opcache works as a workaround

[dausruddin]

PHP 7.1.9 and having the same problem.
Also disabling opcache fixed it too.

[liuyigh]

PHP 7.1.8 on serverpilot and having the same problem. Trouble shooting for hours until reading this thread. Switch to 7.0 on serverpilot, then it works right away. Thanks, guys!

[liuyigh]

The Jetpack people noticed me about the freshly baked 5.3 update today:

https://jetpack.com/2017/09/05/jetpack-5-3-php-7-1-compatibility/

Now works well with php 7.1. Hooray!