Composer: zlib_decode(): data error

[radutopala]

My composer.json:

{
    "name": "test/test",
    "license": "proprietary",
    "type": "project",
    "autoload": {
        "psr-4": {
            "": "src/"
        },
        "classmap": [
            "app/AppKernel.php",
            "app/AppCache.php"
        ]
    },
    "autoload-dev": {
        "psr-4": {
            "Tests\\": "tests/"
        }
    },
    "require": {
        "php": ">=7.0.11",
        "symfony/symfony": "3.1.*",
        "doctrine/orm": "^2.5",
        "doctrine/doctrine-bundle": "^1.6",
        "doctrine/doctrine-cache-bundle": "^1.2",
        "doctrine/doctrine-migrations-bundle": "^1.1",
        "symfony/swiftmailer-bundle": "^2.3",
        "symfony/monolog-bundle": "^2.8",
        "symfony/assetic-bundle": "^2.8",
        "symfony/polyfill-apcu": "^1.0",
        "sensio/distribution-bundle": "^5.0",
        "sensio/framework-extra-bundle": "^3.0.2",
        "incenteev/composer-parameter-handler": "^2.0",
        "snc/redis-bundle": "2.x-dev",
        "predis/predis": "^1.0",
        "lexik/jwt-authentication-bundle": "^1.5",
        "tss/automailer-bundle": "dev-master",
        "knplabs/knp-paginator-bundle": "^2.5",
        "fabpot/goutte": "^3.1",
        "whiteoctober/breadcrumbs-bundle": "^1.2",
        "spraed/pdf-generator-bundle": "^1.3",
        "vich/uploader-bundle": "1.2.0",
        "friendsofsymfony/jsrouting-bundle": "^1.6",
        "gregwar/captcha-bundle": "^2.0",
        "liip/imagine-bundle": "^1.6",
        "knplabs/knp-menu-bundle": "^2.0",
        "fabpot/php-cs-fixer": "^1.12",
        "symfony/serializer": "^3.1",
        "jms/serializer": "^1.3",
        "jms/serializer-bundle": "^1.1"

    },
    "require-dev": {
        "sensio/generator-bundle": "^3.0",
        "symfony/phpunit-bridge": "^3.0"
    },
    "scripts": {
        "post-install-cmd": [
            "Incenteev\\ParameterHandler\\ScriptHandler::buildParameters",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::prepareDeploymentTarget"
        ],
        "post-update-cmd": [
            "Incenteev\\ParameterHandler\\ScriptHandler::buildParameters",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::buildBootstrap",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::clearCache",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installAssets",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::installRequirementsFile",
            "Sensio\\Bundle\\DistributionBundle\\Composer\\ScriptHandler::prepareDeploymentTarget"
        ]
    },
    "extra": {
        "symfony-app-dir": "app",
        "symfony-bin-dir": "bin",
        "symfony-var-dir": "var",
        "symfony-web-dir": "web",
        "symfony-tests-dir": "tests",
        "symfony-assets-install": "relative",
        "incenteev-parameters": {
            "file": "app/config/parameters.yml"
        }
    }
}

Output of composer diagnose:

Checking composer.json: WARNING
Defining autoload.psr-4 with an empty namespace prefix is a bad idea for performance
require.snc/redis-bundle : exact version constraints (2.x-dev) should be avoided if the package follows semantic versioning
require.tss/automailer-bundle : unbound version constraints (dev-master) should be avoided
require.vich/uploader-bundle : exact version constraints (1.2.0) should be avoided if the package follows semantic versioning
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys:
Tags Public Key Fingerprint: 57815BA2 7E54DC31 7ECC7CC5 573090D0  87719BA6 8F3BB723 4E5D42D0 84A14642
Dev Public Key Fingerprint: 4AC45767 E5EC2265 2F0C1167 CBBB8A2B  0C708369 153E328C AD90147D AFE50952
OK
Checking composer version: OK

When I run this command:

composer update --lock -vvv

I get the following output:

Reading ./composer.json
Loading config file ./composer.json
Checked CA file /etc/ssl/certs/ca-certificates.crt: valid
Executing command (/var/www): git branch --no-color --no-abbrev -v
Failed to initialize global composer: Composer could not find the config file: /home/vagrant/.config/composer/composer.json
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
Reading /var/www/vendor/composer/installed.json
Loading plugin PackageVersions\Installer
Running 1.2.1 (2016-09-12 11:27:19) with PHP 7.0.11-1+deb.sury.org~trusty+1 on Linux / 3.13.0-86-generic
Reading ./composer.lock
Loading composer repositories with package information
Downloading https://packagist.org/packages.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/packages.json into cache
Updating dependencies (including require-dev)
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2013.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2014.json from cache
Downloading http://packagist.org/p/provider-2015%245cb89636f37ca04dbb7ceb6ba2f12ccbc6cd3a53b2ae3528c090a56f90975e18.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2015.json into cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2016-01.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2016-04.json from cache
Downloading http://packagist.org/p/provider-2016-07%247062176fff1b75c50a0a1383e29cc73864f4c0ce30d243c1234e0833be1193c0.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2016-07.json into cache
Downloading http://packagist.org/p/provider-2016-10%244d79c57bd512ea7c9843446e53ae95329eef94e3be3511f5f701a5f400739bd0.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-2016-10.json into cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-archived.json from cache
Downloading http://packagist.org/p/provider-latest%24dbfb3faaaffdfaec3494564d129e9a9310fc4a753efad4fcc8b4f91cf787398a.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/p-provider-latest.json into cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$lexer.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$annotations.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-util.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-paragonie$random-compat.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-php70.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-php56.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-mbstring.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$intl.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-intl-icu.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$icu.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-php54.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$inflector.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$collections.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$cache.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$common.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-jdorn$sql-formatter.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$doctrine-bridge.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$doctrine-cache-bundle.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$dbal.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$console.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$framework-bundle.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-doctrine$doctrine-bundle.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$security-acl.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$debug.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$config.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$dependency-injection.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$event-dispatcher.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$filesystem.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$http-kernel.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$routing.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$stopwatch.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$templating.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$translation.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$http-foundation.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$finder.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$security-core.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$security-csrf.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$asset.json from cache
Reading /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$class-loader.json from cache
Downloading http://packagist.org/p/symfony/cache%24a1f76f69d06d444880b68ccf242fb1bd45fc3e1ab332313999d2669d3a5e2672.json
Failed to decode response: zlib_decode(): data error
Retrying with degraded mode, check https://getcomposer.org/doc/articles/troubleshooting.md#degraded-mode for more info
Downloading http://packagist.org/p/symfony/cache%24a1f76f69d06d444880b68ccf242fb1bd45fc3e1ab332313999d2669d3a5e2672.json
Downloading http://packagist.org/p/symfony/cache%24a1f76f69d06d444880b68ccf242fb1bd45fc3e1ab332313999d2669d3a5e2672.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$cache.json into cache
Downloading http://packagist.org/p/psr/log%244f0edf1b639fc9b0914c0c8d13bef27cdaa71100c3719780a0459b9dd799753d.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/provider-psr$log.json into cache
Downloading http://packagist.org/p/symfony/polyfill-php55%242c54472a8856ee39440a29ee7182c3205d88d60817040f9dea8604d9eb7f2a41.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-php55.json into cache
Downloading http://packagist.org/p/symfony/polyfill-apcu%24a910411a4953aeae3659a62e7334cc11b50e9066d4dc6a6b63ff4a665879ebb1.json
Writing /home/vagrant/.cache/composer/repo/https---packagist.org/provider-symfony$polyfill-apcu.json into cache
Downloading http://packagist.org/p/psr/cache%24ed2696a6c78734dcaf8b8eb5ba4b7ebc9d2822015a1211521e390c514093e303.json
Downloading http://packagist.org/p/psr/cache%24ed2696a6c78734dcaf8b8eb5ba4b7ebc9d2822015a1211521e390c514093e303.json
Downloading http://packagist.org/p/psr/cache%24ed2696a6c78734dcaf8b8eb5ba4b7ebc9d2822015a1211521e390c514093e303.json


  [ErrorException]
  zlib_decode(): data error


Exception trace:
 () at phar:///bin/composer/src/Composer/Util/RemoteFilesystem.php:388
 Composer\Util\ErrorHandler::handle() at n/a:n/a
 zlib_decode() at phar:///bin/composer/src/Composer/Util/RemoteFilesystem.php:388
 Composer\Util\RemoteFilesystem->get() at phar:///bin/composer/src/Composer/Util/RemoteFilesystem.php:101
 Composer\Util\RemoteFilesystem->getContents() at phar:///bin/composer/src/Composer/Repository/ComposerRepository.php:646
 Composer\Repository\ComposerRepository->fetchFile() at phar:///bin/composer/src/Composer/Repository/ComposerRepository.php:332
 Composer\Repository\ComposerRepository->whatProvides() at phar:///bin/composer/src/Composer/DependencyResolver/Pool.php:204
 Composer\DependencyResolver\Pool->computeWhatProvides() at phar:///bin/composer/src/Composer/DependencyResolver/Pool.php:193
 Composer\DependencyResolver\Pool->whatProvides() at phar:///bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:161
 Composer\DependencyResolver\RuleSetGenerator->whitelistFromPackage() at phar:///bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:322
 Composer\DependencyResolver\RuleSetGenerator->getRulesFor() at phar:///bin/composer/src/Composer/DependencyResolver/Solver.php:214
 Composer\DependencyResolver\Solver->solve() at phar:///bin/composer/src/Composer/Installer.php:461
 Composer\Installer->doInstall() at phar:///bin/composer/src/Composer/Installer.php:216
 Composer\Installer->run() at phar:///bin/composer/src/Composer/Command/UpdateCommand.php:174
 Composer\Command\UpdateCommand->execute() at phar:///bin/composer/vendor/symfony/console/Command/Command.php:259
 Symfony\Component\Console\Command\Command->run() at phar:///bin/composer/vendor/symfony/console/Application.php:847
 Symfony\Component\Console\Application->doRunCommand() at phar:///bin/composer/vendor/symfony/console/Application.php:192
 Symfony\Component\Console\Application->doRun() at phar:///bin/composer/src/Composer/Console/Application.php:231
 Composer\Console\Application->doRun() at phar:///bin/composer/vendor/symfony/console/Application.php:123
 Symfony\Component\Console\Application->run() at phar:///bin/composer/src/Composer/Console/Application.php:104
 Composer\Console\Application->run() at phar:///bin/composer/bin/composer:43
 require() at /bin/composer:24

And I expected this to happen:
Expected to update the composer.lock

[radutopala]

The problem seems to appear on this line https://github.com/composer/composer/blob/1.2.1/src/Composer/Util/RemoteFilesystem.php#L388

[radutopala]

Error is reproducible with all 1.2.*, not reproducible with 1.1.3.

[alcohol]

Please search existing issues for possible solutions. This is not a Composer issue but an environment issue.

[radutopala]

@alcohol not sure it's just an environment issue, because composer 1.1.3 works fine in the same environment, as I said above.

Have searched over the tickets, those are related:

• https://github.com/composer/composer/issues/5421 - still open
• https://github.com/composer/composer/issues/5350 - closed
• https://github.com/composer/composer/issues/4121 - closed

[alcohol]

Unless you can provide a way for me to reproduce it consistently, it is most definitely an environment issue.

[radutopala]

@alcohol this fixed my issue: https://github.com/composer/composer/issues/4121#issuecomment-123701983. So basically forcing https did it..

The issue was happening in Vagrant boxes, with any kind of php or composer version. Using https://gist.github.com/radutopala/8761ac647ee165d373b0c6d727cd56ea, intermittently I got from 1 to 20 retries until successful decoding the gzip.

[alcohol]

Let me guess, using a shared directory also with your host system?

[radutopala]

@alcohol yes

[alcohol]

What happens if you run it inside vagrant, without a shared directory? E.g. purely inside the vagrant box, storage inside the vagrant box too.

[radutopala]

@alcohol apparently the same error:

vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Error in Decoding process! Re-fetching ...
Retry 1 | Error in Decoding process! Re-fetching ...
Retry 2 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Error in Decoding process! Re-fetching ...
Retry 1 | Error in Decoding process! Re-fetching ...
Retry 2 | Error in Decoding process! Re-fetching ...
Retry 3 | Error in Decoding process! Re-fetching ...
Retry 4 | Error in Decoding process! Re-fetching ...
Retry 5 | Error in Decoding process! Re-fetching ...
Retry 6 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Error in Decoding process! Re-fetching ...
Retry 1 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Error in Decoding process! Re-fetching ...
Retry 1 | Error in Decoding process! Re-fetching ...
Retry 2 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Error in Decoding process! Re-fetching ...
Retry 1 | Error in Decoding process! Re-fetching ...
Retry 2 | Error in Decoding process! Re-fetching ...
Retry 3 | Error in Decoding process! Re-fetching ...
Retry 4 | Error in Decoding process! Re-fetching ...
Retry 5 | Error in Decoding process! Re-fetching ...
Retry 6 | Error in Decoding process! Re-fetching ...
Retry 7 | Error in Decoding process! Re-fetching ...
Retry 8 | Error in Decoding process! Re-fetching ...
Retry 9 | Decoded successfully!

So it doesn't matter where the test is taken, either in shared folder or in an internal box folder.

For what is worth mentioning, I have VirtualBox 5.0.28 and vagrant 1.8.1.

[alcohol]

Not really worth mentioning since we do not maintain virtualbox nor vagrant (though I doubt vagrant is at fault here, likely virtualbox).

[radutopala]

Well .. even though composer maintainers are not tight to VirtualBox nor vagrant, other users may come across the same issue I had.

I just discovered that VirtualBox 5.0.28 is the cause!

And vagrant nicely downgraded VirtualBox to 5.0.10:

==>  Provider 'virtualbox' not found. We'll automatically install it now...
     The installation process will start below. Human interaction may be
     required at some points. If you're uncomfortable with automatically
     installing this provider, you can safely Ctrl-C this process and install
     it manually.
==>  Downloading VirtualBox 5.0.10...
     This may not be the latest version of VirtualBox, but it is a version
     that is known to work well. Over time, we'll update the version that
     is installed.

Now the test is decoding correctly the gzip-ed content every time:

vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!
vagrant@box:~$ php test.php
Retry 0 | Decoded successfully!

So .. the solution is to let vagrant install the right VirtualBox version.

[jaumesala]

Just in case somebody is redirected here because of the zlib_decode() error while trying to update dependencies using composer update

[ErrorException]
  zlib_decode(): data error

I confirm the bug is related to VirtualBox, either 5.0.28 or 5.1.8 (last stable build at the moment) is affected.
As @radutopala points out, downgrading to 5.0.26 solves the problem.

I had the same problem in 3 different machines, macOS 10.10 and 10.12.
In all of them composer diagnose did not report any problem.

I did many tests, create clean VM instances of Laravel Homestead and installing the Laravel framework, forcing IPv4 in the VM, disabling ssl verification of packagist, etc...

Nothing worked until I downgraded to 5.0.26.

[ankrtl]

downgrading to 5.0.26 also works on OSX 10.11.6

[cbj4074]

@radutopala Brilliant! Indeed it is VirtualBox.

@jaumesala Great work pinpointing the latest specific VirtualBox version that does not exhibit this problem!

I just tried downgrading to VirtualBox 5.0.26 and sure enough, the problem no longer occurs. You two spared me hours of further frustration, and I deeply appreciate it.

Given that this does not seem to be a problem with Composer, I opened a thread on the VirtualBox forums. If nobody can offer a viable solution there, I'll open an actual bug report with VirtualBox.

https://forums.virtualbox.org/viewtopic.php?f=3&t=80396&p=376826

It would be helpful if other folks could visit the above thread and chime-in with guest OS version and any other information that might be useful to the VirtualBox developers.

[udaiveerS]

I had this error on Homestead image with Vagrant 1.8.1 and Virtual box 5.1.8 on Mac Os. Downgrading to Virtualbox 5.0.26 and composer update completes without a problem.

[mihhac]

Hi, I had the same problem. Downgrading VirtualBox from 5.1.8 to 5.1.6 helped.

[bassrock]

Same here there is definitely an issue with the latest virtual box and shared nfs folders.

[thats4shaw]

I encountered this issue today with the exact same setup as @udaiveerS. Downgrading VirtualBox to 5.1.6 solved it for me also.

[cbj4074]

@mihhac @thats4shaw Good to know that 5.1.6 is the latest version of the 5.1 branch that doesn't suffer from this problem.

Does anyone have the networking expertise to chime-in on the VirtualBox thread at https://forums.virtualbox.org/viewtopic.php?f=3&t=80396&p=376826 ? That would be hugely helpful to the rest of us. :)

[Ingramz]

@cbj4074 it's highly unlikely you will find such people here 😆 , but if you give VirtualBox guys a way to reproduce this issue, they'll figure it out themselves.

Start by specifying your host and guest operating systems, then either use a prebuilt virtual machine available online or specify you installed it from an image. After that add the commands they need to run to get php and the test script running on the box and what the expected and actual outcomes are.

[cbj4074]

@Ingramz Thanks! All great advice. Maybe I can dedicate a couple of hours to this today.

I would really like to eliminate PHP from the equation entirely. Maybe it is a distraction from the underlying issue. If we could simply demonstrate that files cannot reliably be "gunzipped", regardless of their source, that would be ideal.

But maybe PHP is part of the problem. I do find it curious that the OS itself wouldn't barf in the nearest trash can if gunzipping failed across-the-board. Presumably gzip is used all over the place, from the package manager to log-rotation.

[Ingramz]

@cbj4074 as far as I know it doesn't matter what you give them, the first thing they will look at are recent changes by performing a binary search on the commits. They probably know better themselves where to look after that.

[magnetik]

Maybe it's worth taking a look at the strace output of the test script: https://gist.github.com/magnetik/82a9d1d9c34401b6268ba07a949cc491#file-strace-txt

Looking at the end, it only looks like the connection is interrupted ?.

[alcohol]

@magnetik can you reproduce that consistently though?

[magnetik]

When running the script in the gist, the first 3-20 tries fails. Not consistently.

I'm using sury PPA

> % php -v
PHP 7.0.12-1+deb.sury.org~xenial+1 (cli) ( NTS )
Copyright (c) 1997-2016 The PHP Group
Zend Engine v3.0.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.12-1+deb.sury.org~xenial+1, Copyright (c) 1999-2016, by Zend Technologies

[alcohol]

What do you mean, except for the last one? It has an exit() in the else branch, so it should abort after the first failure. Not sure what the purpose of $retries is, since the implementation has no proper loop nor check.

[magnetik]

I've edited my answer since it was not clear enough. Sometimes it decodes successfully after 3 tries, sometimes after 20 tries.

I've logged the output of the file_get_content, when it fails it's 3KB, and the successful it 4KB.

I've added the log in the gist :

• https://gist.github.com/magnetik/82a9d1d9c34401b6268ba07a949cc491#file-try_error-txt
• https://gist.github.com/magnetik/82a9d1d9c34401b6268ba07a949cc491#file-try_success-txt

[alcohol]

Can you try running your script against https://dump.robbast.nl/tmp.json ?

[magnetik]

Same, 3 tries 3KB, and last one is 4KB.

[alcohol]

Welp. Completely different server, but same issue. That is not a good sign.

[alcohol]

What if you run it with:

    $opts = array(
        'http' => array(
            'header' => array(
                "Accept-Encoding: application/json, text/plain",
                "Connection: close",
                "User-Agent: Composer/1.2.1 (Linux; 3.13.0-86-generic; PHP 7.0.12)"
            ),
            'ignore_errors' => "1",
            'protocol_version' => "1.1"
        )
    );

[magnetik]

Wait ! I messed up last test, sorry about that.

With the URL https://dump.robbast.nl/tmp.json it works well with the gzip accept encoding.

[alcohol]

Oh, it consistently works?

[magnetik]

I ran it 50 times, works everytime.

[alcohol]

Yeah I can see that :p

log entries

Nov 07 10:48:37 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.061 - *.*.*.*
Nov 07 10:48:37 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:37 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:37 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.059 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.056 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.060 - *.*.*.*
Nov 07 10:48:38 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.059 - *.*.*.*
Nov 07 10:48:39 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.058 - *.*.*.*
Nov 07 10:48:39 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 40586 0.054 - *.*.*.*
Nov 07 10:48:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:48:57 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:48:58 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:48:59 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:49:35 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:49:36 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:49:37 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:01 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:51:04 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:05 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:05 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.061 12.70 *.*.*.*
Nov 07 10:51:06 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:07 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:07 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:08 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:51:09 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:09 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:10 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:11 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:51:35 vps.robbast.nl nginx[10324]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:52 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:52 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:51:52 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:53 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:53 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:53 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:53 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:54 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:54 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:54 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:54 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:51:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:55 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:51:56 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:56 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:56 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:51:56 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:56 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:57 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:57 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:51:57 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:57 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:58 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:51:58 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.061 12.70 *.*.*.*
Nov 07 10:51:58 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:58 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:51:59 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:51:59 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:51:59 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.061 12.70 *.*.*.*
Nov 07 10:51:59 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:52:00 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:52:00 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:52:00 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:52:00 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:52:01 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.061 12.70 *.*.*.*
Nov 07 10:52:01 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*
Nov 07 10:52:01 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.056 12.70 *.*.*.*
Nov 07 10:52:01 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:52:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:52:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:52:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.055 12.70 *.*.*.*
Nov 07 10:52:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.060 12.70 *.*.*.*
Nov 07 10:52:02 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:52:03 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.058 12.70 *.*.*.*
Nov 07 10:52:03 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.057 12.70 *.*.*.*
Nov 07 10:52:03 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.054 12.70 *.*.*.*
Nov 07 10:52:03 vps.robbast.nl nginx[10323]: dump.robbast.nl 200 "GET /tmp.json HTTP/1.1" 3207 0.059 12.70 *.*.*.*

Not sure what the diff in size caused between the first few requests and the latter.

[magnetik]

I've ran the command with the non gzip output too (when I identified that I was not testing with your URL).

[alcohol]

Ah that must have been the first few requests.

[Seldaek]

I also get two types of requests in the packagist.org logs from your IP:

[07/Nov/2016:09:50:39 +0000] "GET /p/paragonie/random_compat%24f49f654e75ef0944f24b3ffb74b531f0f5a757a807bcbda5b511b85d41ff7dcb.json HTTP/1.1" 200 40586 "-" "Composer/1.2.1 (Linux; 3.13.0-86-generic; PHP 7.0.12)"

[07/Nov/2016:09:50:52 +0000] "GET /p/paragonie/random_compat%24f49f654e75ef0944f24b3ffb74b531f0f5a757a807bcbda5b511b85d41ff7dcb.json HTTP/1.1" 200 3207 "-" "Composer/1.2.1 (Linux; 3.13.0-86-generic; PHP 7.0.12)"

Same sizes as on @alcohol's server.

[alcohol]

Really looks like a networking issue. My server is in Amsterdam (NL), while packagist is somewhere in France if I am not mistaken?

[magnetik]

When doing the test with application/json;text/plain (and tweaking the decode with json_decode and test for null), I've the same error on the packagist server.

I received the following size, in that order:

• 38KB
• 33KB
• 40KB (success)

It's the last 3 requests you should have on your server with my IP.

[alcohol]

Did you run those with strace? Did you get the connection reset errors on the smaller sizes and not on the successful request?

[Seldaek]

@magnetik can you try with https:// on packagist as well? And try with http:// on @alcohol's server? Because that's the one difference I see here, is that you used https on his and http on packagist. If that solves it, then use composer config -g repo.packagist composer https://packagist.org to force https always on packagist. That tends to solve bad proxies and snoopy networks that mess with traffic.

[magnetik]

@alcohol running strace on packagist, with http and plain text. I do not see the connection reset : https://gist.github.com/magnetik/82a9d1d9c34401b6268ba07a949cc491#file-strace_http_packagist_plain-txt

But the following write sizes : 8192 + 8192 + 8192 + 8192 + 4621 = 37389
And when the success : 8192 + 8192 + 8192 + 8192 + 7818= 40586

@Seldaek when using https with gzip, running 50 tests, it works well.

[alcohol]

If https solves it, that is both good and scary :p

[alcohol]

@Seldaek I forgot that I enforce https on my server. So he cannot test with http on mine.

[magnetik]

Well it ran successfully 50 times, then I tried to run composer update on a problem and got an error.

When testing with the URL https://packagist.org/p/lncd/oauth2%244010b595abf2d3c3a38b84ffd41f6a4a1587c4a54886559347b686a5e0116355.json, i've also the error : Retry 0 | PHP Warning: file_get_contents(): SSL: Connection reset by peer

[alcohol]

That damn peer..

Does it consistently reset the connection?

I updated the tmp.json file I was hosting to be identical to the oauth one. You can try to vary between the packagist hosted file and my server, to see if there is a difference.

[magnetik]

5 requests fails over 20.

[alcohol]

And on my host?

[magnetik]

Same, maybe a little less, I had 3 fails in 35 requests

[alcohol]

Weird, unable to reproduce it myself. I see all your requests and they all seem to succeed (assuming the connection reset occurs after connecting). I count 56 requests.

[magnetik]

When using curl directly, I've always the same size :
curl https://packagist.org/p/lncd/oauth2%244010b595abf2d3c3a38b84ffd41f6a4a1587c4a54886559347b686a5e0116355.json 2>/dev/null | wc -m

Given that virtualbox 5.1.6 did not had the issue, it should be something on that side that is closing the connection too early.

[Seldaek]

curl by default doesn't use gzip though for example, and maybe other parameters are different. At least try with --compressed to enable gzip.

[magnetik]

I've an answer from virtualbox, and it seems to be on their side : https://forums.virtualbox.org/viewtopic.php?f=3&t=80396#p377627

Thanks for the input that allowed to pinpoint the issue on their side.

Let's continue the discussion there.

[josecelano]

I had the same problem. Fixed downgrading VirtualBox to Version 5.0.26.

Host:
Windows 8.1 Pro
VirtualBox: VirtualBox-5.1.8          <- IT DOES NOT WORK
VirtualBox: Version 5.0.26 r108824    <- IT WORKS
Guest:
Ubuntu 16.04.1 LTS
PHP: PHP 5.6.27-1+deb.sury.org~xenial+1 (cli)

Some contributor should add this solution more visible in the issue #4121 (comments are closed). Google points your first to the other issue.

[radutopala]

At least we can reference #4121 here.

[magnetik]

Issue is fixed for me for the to be released Virtualbox 5.1.9.

You can try it yourself with Virtualbox test build > 111846 https://www.virtualbox.org/wiki/Testbuilds

[kcaporaso]

@magnetik, confirmed it myself as well. VirtualBox 5.1.9 development build resolved my issue. Thanks everyone!

[hectornguyen]

Jesus christ!
How Virtualbox version affects to composer update ??? I have to reinstall my Vagrant box over and over until I found this issue....

VirtualBox 5.0.28 should be eliminated from Earth. Thank Oracle.

[stof]

@hectornguyen Virtualbox had a bug breaking HTTP traffic using Gzip.
And composer update can precisely make HTTP requests to servers using Gzip on the JSON responses.

[markbennett1973]

This is now working fine for me after upgrading to VirtualBox 5.0.30, so it looks like Oracle have fixed it now.

[psgganesh]

Virtualbox 5.0.30 r112061 worked well in my case...

[Oliboy50]

Virtualbox 5.1.14 r112924 (currently the latest release) works too

[brazorf]

Im still having this issue with 5.1.22 r115126 on windows host