Packagist: TransportException 404 Not Found for packagist.org urls

[Krinkle]

Filing a separate issue following from #470, given this is not related to SlowTimer.

This still happens on a regular basis in Wikimedia CI pipeline. Is there a way in which we can prevent this issue in a more general way, that hopefully also avoids too much load or storage pressure on your side?

See https://phabricator.wikimedia.org/P7484$1052 for a recent transcript of a build in which this error occurred.

For that build, it seems like the failure was printed only 4 seconds after the composer require command.

01:00:57  DEBUG:quibble.cmd:composer require composer/spdx-licenses=1.4.0 psy/psysh=0.9.6 monolog/monolog=~1.22.1 phpunit/phpunit=4.8.36 \|\| ^6.5 wikimedia/avro=1.8.0 mediawiki/mediawiki-codesniffer=21.0.0 wmde/hamcrest-html-matchers=^0.1.0 seld/jsonlint=1.7.1 wikimedia/testing-access-wrapper=~1.0 jakub-onderka/php-parallel-lint=0.9.2 jetbrains/phpstorm-stubs=dev-master#38ff1a581b297f7901e961b8c923862ea80c3b96 hamcrest/hamcrest-php=^2.0 nmred/kafka-php=0.1.5 justinrainbow/json-schema=~5.2 nikic/php-parser=3.1.3 cache/integration-tests=0.16.0
--
01:01:00
01:01:00    [Composer\Downloader\TransportException]
01:01:00    The "http://packagist.org/p/provider-latest%2489d14de6aab31e05c38c5a490ca286c3895c7713604e767413aeb3ee5f9d0784.json" file could not be downloaded (HTTP/1.1 404 Not Found)
01:01:00
01:01:00  Exception trace:
01:01:00   () at /srv/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:430
01:01:00   Composer\Util\RemoteFilesystem->get() at /srv/composer/vendor/composer/composer/src/Composer/Util/RemoteFilesystem.php:101
01:01:00   Composer\Util\RemoteFilesystem->getContents() at 

Also:

• https://github.com/composer/packagist/issues/470
• https://github.com/composer/composer/issues/2198
• https://github.com/composer/composer/issues/2228
• https://github.com/composer/composer/issues/7021
• https://github.com/composer/composer/issues/7404
• https://github.com/composer/composer/issues/7599
• Downstream: https://phabricator.wikimedia.org/T182266

[Krinkle]

As of last week this is happening multiple times a day. After having seemingly been almost unseen for months. Looks like a recent regressions perhaps?

I'm also seeing it on Travis CI now, which is a first.

https://travis-ci.org/wikimedia/mediawiki/jobs/422335462

$ composer --version
Composer version 1.7.2 2018-08-16 16:57:12

(154.23s)$ composer install --prefer-source --quiet --no-interaction

  [Composer\Downloader\TransportException]                                                                                                                                         
  The "http://repo.packagist.org/p/provider-latest%24233d2d1a537144f307feb64eaddbf9a265c5ec874d0ac94f08743a653aec2100.json"
  file could not be downloaded (HTTP/1.1 404 Not Found)

Note that the run-time there is also measured. Travis clocks it at 154s (2 minutes and 34 seconds), which is far less than the previously stated age of "older than 10 minutes" for files that may be removed on the Packagist servers, per https://github.com/composer/packagist/issues/470#issuecomment-394966004.

[denzkiuy]

Hi, I'm also having the same issue. I was trying to perform "composer update" on my laravel project but I encountered the exact same error . I have tried several possible workarounds suggested by different people but nothing worked for me. I hope anyone could help.

After composer update:

[Seldaek]

This should not happen anymore now, it was a race condition between mirrors but I put a fallback in place now so they should go fetch the missing file if a race happens. Please do tell if you see it again.

[cloudways-haider]

Hi, @Seldaek we are still facing the same issue upon running composer require barryvdh/laravel-snappy

[justkristin]

Same issue here.

The "http://packagist.org/p/symfony/process%24ac4ca6b027eeff047d0e37f31208a8d978b98da6c16989614914a639e632ca84.json" file could not be downloaded: failed to open stream: Connection timed out

[belzaaron]

Issue popped up again, overnight. Here's the trace(s) this time:

New Project:

  [Composer\Downloader\TransportException]
  The "https://packagist.org/p/provider-2019-04%244380046a2dfa549bfcf196bfc2c1a5cca9d58c63fb9b8e836367bfe2c01f062b.json" file could not be downloaded (HTTP/1.1 404 Not Found)

Exiting Project:

  [Composer\Downloader\TransportException]                                                                                                                                      
  The "https://packagist.org/p/provider-2018-07%241e5761c3575f5f87c86fa2ca8099d924ad72d7995718b1bfedf2c3fc27241568.json" file could not be downloaded (HTTP/1.1 404 Not Found) 

Happens on any new Laravel project and any call to Composer/Packagist on existing ones. Also may be a different url per-project.

Not a very helpful string of numbers/letters for the url. Also WaybackMachine has no record for this. So I'm stuck here.

@Seldaek letting you know because of https://github.com/composer/packagist/issues/940#issuecomment-418476300

[phil-davis]

@Seldaek I am also experiencing this a lot today. e.g. https://drone.owncloud.com/owncloud/core/18699 and pick any of the red jobs.
https://drone.owncloud.com/owncloud/core/18699/130

  [Composer\Downloader\TransportException]                                                                                                                                   
The "https://packagist.org/p/provider-2016%24e6f26a3fa2ea1261459b39ffa1f461ca4fb2ae031d46b8ecb204fc2bdb38bd1c.json" file could not be downloaded (HTTP/1.1 404 Not Found)

and the errors have various different provider-yyyy*.json file names.

I am guessing that there is some problem with packagist server(s) somewhere?

Is there a status page or twitter thread or... where real-time status of the packagist servers is posted?
That would be handy - to know if there is already a known problem being worked.

And how are we supposed to report this sort of error? which is probably a real-time packagist mirror/server sysadmin problem not a packagist software bug.

[Seldaek]

Hey, sorry looks like one of the mirror nodes failed to sync properly after I did some updates yesterday.. Should be fixed now.

[phil-davis]

@Seldaek
1) is there somewhere that reports known packagist status? similar to https://www.githubstatus.com/ and https://www.traviscistatus.com/ etc.
2) is there somewhere that we can report when there seems to be an issue?

Because otherwise we are left trying to report via the bug-reporting mechanism on GitHub, and that is not the ideal way to report system-network problems.

[SV-ZeroOne]

Hello, I recently encountered the same following error when trying to composer require seostats/seostats package.
Was first getting this error
[Composer\Downloader\TransportException] The "https://packagist.org/p/provider-2019-04%244380046a2dfa549bfcf196bfc2c1a5cca9d58c63fb9b8e836367bfe2c01f062b.json" file could not be downloaded (HTTP/1.1 404 Not Found)

Then tried a composer clear-cache
Now getting this error
[Composer\Downloader\TransportException] Content-Length mismatch, received 551423 bytes out of the expected 2019004

Can anybody explain why this is happening?

Here is my composer diagnose results
Checking platform settings: OK
Checking git settings: OK
Checking http connectivity to packagist: OK
Checking https connectivity to packagist: OK
Checking github.com rate limit: OK
Checking disk free space: OK
Checking pubkeys:
Checking composer version: OK
Composer version: 1.8.6
PHP version: 7.2.20
PHP binary path: /opt/cpanel/ea-php72/root/usr/bin/php

[Krinkle]

This is happening again, seems to affect Wikimedia CI consistently in each job for the past 20 minutes.

– https://phabricator.wikimedia.org/T233264

[Seldaek]

@Krinkle did it get resolved? I messed something up and we had delays in processing metadata updates last night.

[hashar]

@Seldaek that seems to have disappeared now (our task https://phabricator.wikimedia.org/T233264 ).

I guess we should really look at setting up a packagist mirror for CI purposes.

[Krinkle]

Thanks @Seldaek! All green now.

[Krinkle]

@Seldaek It would appear the issue is back as of a few hours ago. Affects about 1/10 CI builds.

20:10:56   [Composer\Downloader\TransportException]                                                                                                                                   
20:10:56   The "https://packagist.org/p/provider-2017%2462534e44c7215c59b58e5ae6fba06ba61a46d713507c3a58bd130e1940517d96.json" file could not be downloaded (HTTP/1.1 404 Not Found)  

[phil-davis]

I am also seeing this sort of thing:

 [Composer\Downloader\TransportException]                                                                                                                                      
  The "https://packagist.org/p/provider-2019-07%24d56f91aee95bccb2a4018a08708bc0990ba8888f45389df7de412062affd4651.json" file could not be downloaded (HTTP/1.1 404 Not Found)

e.g. https://drone.owncloud.com/owncloud/files_texteditor/452/1/2

I had lots of these errors in overnight jobs.

[mariushoch]

So 51.38.227.34 at least seems to be flaky:

curl -v 'https://packagist.org/p/provider-2017%242361f4e66f731385b3f231a53c107c99231cbeb7496b2194a0eba6aa033fe4df.json' --resolve 'packagist:443:51.38.227.34'

[johny-smith]

Hi, im having the same problem with

composer create-project --repository-url=https://repo.magento.com/ magento/project-community-edition .

Downloading https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b87715596246536ed9fc707c.json

Downloading https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b87715596246536ed9fc707c.json


  [Composer\Downloader\TransportException]                                                              
  The "https://repo.packagist.org/p/symfony/process%245c5f786d7473c01dfa73e8b7d64e9bf4421871a0b8771559  
  6246536ed9fc707c.json" file could not be downloaded (HTTP/1.1 404 Not Found)                          


Exception trace:
 () at phar:///usr/bin/composer/src/Composer/Util/RemoteFilesystem.php:440
 Composer\Util\RemoteFilesystem->get() at phar:///usr/bin/composer/src/Composer/Util/RemoteFilesystem.php:104
 Composer\Util\RemoteFilesystem->getContents() at phar:///usr/bin/composer/src/Composer/Repository/ComposerRepository.php:687
 Composer\Repository\ComposerRepository->fetchFile() at phar:///usr/bin/composer/src/Composer/Repository/ComposerRepository.php:358
 Composer\Repository\ComposerRepository->whatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Pool.php:204
 Composer\DependencyResolver\Pool->computeWhatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Pool.php:193
 Composer\DependencyResolver\Pool->whatProvides() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:164
 Composer\DependencyResolver\RuleSetGenerator->whitelistFromPackage() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:304
 Composer\DependencyResolver\RuleSetGenerator->whitelistFromJobs() at phar:///usr/bin/composer/src/Composer/DependencyResolver/RuleSetGenerator.php:355
 Composer\DependencyResolver\RuleSetGenerator->getRulesFor() at phar:///usr/bin/composer/src/Composer/DependencyResolver/Solver.php:217
 Composer\DependencyResolver\Solver->solve() at phar:///usr/bin/composer/src/Composer/Installer.php:475
 Composer\Installer->doInstall() at phar:///usr/bin/composer/src/Composer/Installer.php:229
 Composer\Installer->run() at phar:///usr/bin/composer/src/Composer/Command/CreateProjectCommand.php:195
 Composer\Command\CreateProjectCommand->installProject() at phar:///usr/bin/composer/src/Composer/Command/CreateProjectCommand.php:145
 Composer\Command\CreateProjectCommand->execute() at phar:///usr/bin/composer/vendor/symfony/console/Command/Command.php:245
 Symfony\Component\Console\Command\Command->run() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:835
 Symfony\Component\Console\Application->doRunCommand() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:185
 Symfony\Component\Console\Application->doRun() at phar:///usr/bin/composer/src/Composer/Console/Application.php:267
 Composer\Console\Application->doRun() at phar:///usr/bin/composer/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at phar:///usr/bin/composer/src/Composer/Console/Application.php:106
 Composer\Console\Application->run() at phar:///usr/bin/composer/bin/composer:61
 require() at /usr/bin/composer:24

create-project [-s|--stability STABILITY] [--prefer-source] [--prefer-dist] [--repository REPOSITORY] [--repository-url REPOSITORY-URL] [--dev] [--no-dev] [--no-custom-installers] [--no-scripts] [--no-progress] [--no-secure-http] [--keep-vcs] [--remove-vcs] [--no-install] [--ignore-platform-reqs] [--] [<package>] [<directory>] [<version>]

[jasonquick]

I've just run in to this as well.

Never had a problem before until tonight, when I worked late on a Saturday to push through a big project and now I'm finally done and can't build it. I've been getting 404's on random packagist URLs.

If I run composer again (after clearing the cache) it'll successfully download the one that 404'd and then fail on another one, or fail on a completely different one altogether. I've never had this issue before, and I can browse to the URLs that 404'd just fine.

EDIT: In desperation I just kept running composer over and over and it eventually was able to download all the files and complete the build.

[rpounder]

I've had this issue for several days now, might be one of the mirrors because it works randomly if I just keep doing it over 30 mins or so;

[ComposerDownloaderTransportException]
The "https://packagist.org/p/provider-2019-07%24925f76d9450d25d63f482102431fca16a8a791a1977d123c0611acb360ff8a59.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[johny-smith]

Is there a way to get around this?

[dzabel]

This mirror works for me: https://packagist.phpcomposer.com

use:
composer config -g repo.packagist composer https://packagist.phpcomposer.com

and/or add to your repository section in composer.json

                { "type": "composer", "url": "https://packagist.phpcomposer.com"},

[johny-smith]

that worked ! thanks a lot!

where did you find that extra mirror?

[bcastagna]

You shouldn't use a mirror you don't know.

[jcheron]

You shouldn't use a mirror you don't know.

@bcastagna It is at least known from packagist: see https://packagist.org/mirrors
And it looks better than a TransportException (404 error).

[bcastagna]

@jcheron for sure, I knew it was in this list but @johny-smith asked this _after_ using it, so I wanted to advice him to not use a mirror he doesn't know.

[mlzd3y4n]

I have found something interesting. Only some of requests are result in 404. The same JSON URL something works, but sometimes it doesn't.

I tested one URL and In the period of about 30 seconds, I made about 20 requests, and half of requests are 404.
https://packagist.org/p/provider-2019-10%243b30327406e9822af1b110a34317eac5f6f3e14031157304639e32e0bef726f9.json

So I made some dirty fix, as a temporary solution, in order to repeat (multiple times) download of the single file when it fails at the very first try.

Hardest part is to debug and find the code you need to change (if you see the source for the first time, like me), but it's very easy to do couple of steps as I described in this gist:
https://gist.github.com/mlzd3y4n/206447ec4e2f180ff08be42f628c9983

[Seldaek]

This has been resolved, we had an issue with the metadata mirroring over the weekend sorry about that. I would recommend removing the mirror configs you set as workarounds.

[slakbal]

This is back today: running with latest version:

  [Composer\Downloader\TransportException]                                                                                                                                                 
  The "https://repo.packagist.org/p/symfony/polyfill-iconv%24b166f62b97f898a535e7affe6ffa17c65e4ba9217812e70d62efa41d9ceb2794.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[checcoux]

[ComposerDownloaderTransportException]
The "http://repo.packagist.org/p/provider-2013%24e141175196c33d87744406e650a7ace8e320d4d8a1033eb1e8ecc93f
70ab190b.json" file could not be downloaded: failed to open stream: HTTP request failed!

using last version

[devdadco]

Same here
[ComposerDownloaderTransportException]
The "https://packagist.org/p/paragonie/random_compat%244a8fbabca55a661e93fb
f2683edec96d140551495786c09184236c31934ed17a.json" file could not be downlo
aded (HTTP/1.1 404 Not Found)

@Seldaek whattup?

[Sidlegionair]

@Seldaek Seemed to be back again today, I also had this last week as @ptrm04 reported. Any idea what is causing this? It's interrupting our services pretty hard (Its functioning now, but we were updating some websites this morning and all 200 of them failed due to this error).

The "https://packagist.org/p/provider-2017%240e110647cae1251cc87614228c5ee5ac1c4d69290931bf25b9633b9cb61b389c.json" file could not be downloaded (HTTP/1.1 404 Not Found)

[phil-davis]

I saw this problem also last night and today. I forgot where to find this closed issue! And I raised https://github.com/composer/composer/issues/9112

Is it possible to have somewhere that we can report packagist "run-time-production" issues that are not code bugs?

[Seldaek]

Can you please do a ping packagist.org or something and let me know which IP it resolves to for you (from the machine experiencing the issue)?

[Sidlegionair]

Hi @Seldaek,

Here you go:

[xxxx@xxxx xxxx]$ ping packagist.org
PING packagist.org (54.37.2.184) 56(84) bytes of data.
64 bytes from ip-54-37-2.eu (54.37.2.184): icmp_seq=1 ttl=49 time=11.1 ms
64 bytes from ip-54-37-2.eu (54.37.2.184): icmp_seq=2 ttl=49 time=11.3 ms

[phil-davis]

It's a bit tricky. The CI systems are a number of drone agents in a data center. They start as they are required, no chance to "login" to them and do ordinary commands.

Locally at my desk I am seeing both 139.99.120.115 and 51.79.162.48 - I did composer clearcache, cleaned my repo and started composer install and it is running OK.

I don't know how to reproduce it, because sometimes today I got success on a whole set of CI pipelines, and other times there were up to a dozen pipelines that failed like this.

[Seldaek]

Ok found the problem I think, hopefully you can confirm it is back to normal now.

[Sidlegionair]

Hi @Seldaek,

I am still experiencing these issues

`Installation failed, reverting ./composer.json to its original content.

[ComposerDownloaderTransportException]
The "https://packagist.org/p/provider-2016%24213e60dbd2e7f903a68123906dc41d08e0edfbf13734b00ce024150030b2be5f.json" file could not be downloaded (HTTP/1.1 404 Not Found)

` (this error was ~2 min ago)

Ping output:
[xxxx@xxxx ~]$ ping packagist.org PING packagist.org (51.77.109.217) 56(84) bytes of data. 64 bytes from 51.77.109.217 (51.77.109.217): icmp_seq=1 ttl=48 time=11.0 ms 64 bytes from 51.77.109.217 (51.77.109.217): icmp_seq=2 ttl=48 time=11.1 ms ^C

[Seldaek]

Ah yeah sorry a couple more nodes were having the same issue, I rejoiced too quickly, checked all of them now..

[Sidlegionair]

@Seldaek Seems resolved, thanks!

[dilshandesh]

@Seldaek

[Seldaek]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

[dilshandesh]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

This one also the same issue which people have faced up there. and I solved it. Thanks!

[Sidlegionair]

@dilshandesh The issue reported here was HTTP status code 404. Yours is reporting HTTP status code 500 with a different message.

[devdadco]

Wonder how many production workloads this broke / is breaking that are based on the public packagist - private packagist to the rescue? Or are there issues too?

[devdadco]

Still ongoing... @Seldaek can you please help

[Composer\Downloader\TransportException]                                     
  The "https://packagist.org/p/symfony/polyfill-php56%2461f926ceed20b48df6e5c  
  02f19b8874cf3b3bebf6fbf202bac3d52279f33d36d.json" file could not be downloa  
  ded (HTTP/2 404 )

ping packagist.org
PING packagist.org (54.37.131.18): 56 data bytes
64 bytes from 54.37.131.18: icmp_seq=0 ttl=44 time=57.359 ms
64 bytes from 54.37.131.18: icmp_seq=1 ttl=44 time=54.724 ms
64 bytes from 54.37.131.18: icmp_seq=2 ttl=44 time=53.596 ms
64 bytes from 54.37.131.18: icmp_seq=3 ttl=44 time=53.011 ms
64 bytes from 54.37.131.18: icmp_seq=4 ttl=44 time=52.556 ms
64 bytes from 54.37.131.18: icmp_seq=5 ttl=44 time=52.534 ms

[devdadco]

Moved on from 404 to 302 now:

  [Composer\Downloader\TransportException]                                     
  The "http://repo.packagist.org/p/symfony/polyfill-php56%246256af99f55d49470  
  e29994e51758c5b472305e322b30b72e26b6956ecdb7c26.json" file could not be dow  
  nloaded (HTTP/1.1 302 Found)

[Seldaek]

@ptrm04 maybe try using composer 2 preview with composer self-update --preview. It might help with network trouble too.

[Sidlegionair]

@ptrm04 You can also run your own mirror if packagist.org is too unstable, see https://github.com/Webysther/packagist-mirror

[devdadco]

Thanks guys. Also, why on Earth would it want to install "polyfill-php56" - I've searched the entire solution (including vendors) but "polyfill-php56" was not found anywhere. Symfony 5.1 and composer 1.10.10.

Are any of you getting those? I want to work on a project but heck can't get it to run due to all these errors.

[ossycodes]

@dilshandesh that seems unrelated, please don't pile up random stuff in this old issue, rather open a new one if you have an issue you can reliably reproduce.

This one also the same issue which people have faced up there. and I solved it. Thanks!

Please how did you fix?