Crud: [Kind of bug] Huge backpack bundle sometimes exhaust memory or time allowed to download the package

[srigurubyo]

Bug report

composer update

What I expected to happen

Update to latest version of backpack

What happened

./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 7 installs, 0 updates, 0 removals

• Installing ocramius/package-versions (1.9.0): Loading from cache
• Installing doctrine/event-manager (1.1.0): Loading from cache
• Installing doctrine/cache (1.10.1): Loading from cache
• Installing doctrine/dbal (2.10.2): Loading from cache
• Installing creativeorange/gravatar (v1.0.14): Loading from cache
• Installing prologue/alerts (0.4.7): Loading from cache
• Installing backpack/crud (4.1.11): Downloading (failed)
  Downloading (100%)
  Failed to execute (9) unzip -qq '/home/rajan/saeWork/laravel/v2/saeusip/vendor/backpack/crud/d1a71b925eefae93a814fac371406a81' -d '/home/rajan/saeWork/laravel/v2/saeusip/vendor/composer/792f4eff'

[/home/rajan/saeWork/laravel/v2/saeusip/vendor/backpack/crud/d1a71b925eefae93a814fac371406a81]
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of /home/rajan/saeWork/laravel/v2/saeusip/vendor/backpack/crud/d1a71b925eefae93a814fac371406a81 or
/home/rajan/saeWork/laravel/v2/saeusip/vendor/backpack/crud/d1a71b925eefae93a814fac371406a81.zip, and cannot find /home/rajan/saeWork/laravel/v2/saeusip/vendor/backpack/crud/d1a71b925eefae93a814fac371406a81.ZIP, period.
The archive may contain identical file names with different capitalization (which fails on case insensitive filesystems)
Unzip with unzip command failed, falling back to ZipArchive class

[promatik]

I just tried the update and it worked, i'm on windows.
It appears your download failed, have you tried another time?

[srigurubyo]

Yes, tried multiple times.
I am on Ubuntu 20.04
Btw, my bandwidth is > 50Mbps

[tabacitu]

Hmm that's very weird. I don't have a way to test on Ubuntu right away, but I just updated to 4.1.11 on Mac OS (still unix) and it worked fine.

@srigurubyo what about getting version 4.1.10 - does that work for you? Can you do composer require backpack/crud:"4.1.10" and let us know if this one installs fine?

[srigurubyo]

@tabacitu I had been using 4.1.10 for a while and there was no problem with it. Let me try with a new server and keep you posted.

[srigurubyo]

It worked in the same machine now and gave me nightmare yesterday whole day while updating.

I am not sure what was the problem the though.

[LemarinelNet]

Just to report I also have the problem.
I've already noticed that with 4.0, composer require backpack/crud would also sometimes take ages to start downloading.
Now starting a new project, I face this issue again. Installing dependencies works correctly, but as soon as composer arrives on backpack, it takes a long time before starting to download, and it sometimes fails to download !

[tabacitu]

Ouch! Thanks for reporting this @LemarinelNet . Any idea what the error is when it fails to download? What does composer say?

[LemarinelNet]

Don't know how composer works : does it use a "master mirror" or does it download the package from "your" server ?

In facts, when it has to download backpack, it stays stuck on "connecting..." for a while. Then it either fails or start to download, and the download seems to be at normal speed.

I can't remember the exact error, but last time it has failed, it had to wait for a couple minutes on "connecting..." then after the timeout, it fell back to sources download to try to build it. I didn't let him finish since I knew it was just a matter of new try to make it work, and it did at the next try....

Really act like the server from which composer tries to donwload the package is down. I suppose after the connection timeout it make a couple of new tries before giving up, so this would explain why it finally works ... ?

[tabacitu]

Got it, thanks!

Don't know how composer works : does it use a "master mirror" or does it download the package from "your" server ?

I'm not exactly an expert in this either, but the way I understand it is that:

• you do composer require backpack/crud
• you Composer looks on packagist.org (the main Composer repository) for the package, finds it;
• packagist.org tells you Composer where to download the package from: directly from source on Github, basically points to the right zip file depending on what version you want to install;
• Composer then downloads the file directly from Github and puts it in your vendor folder;

We can move from Github to our own servers, but that won't help much. When Github is down so are most other things you'd want to install using Composer. We kind of have to accept that _we can no longer work if Github is down_. That we're all completely dependant on a free Microsoft service... 👀

I think/hope that's what happened there for you. I know for sure that Github had problems a few days ago - I wasn't even able to access github.com - then came the new UI design. Maybe composer require got everything else from cache, but the latest Backpack...

I'm eager to fix this so no more people encounter this problem but... I don't see a way I can, unfortunately... Open to more information & solutions if anybody has it!

[LemarinelNet]

This is an issue I frequently encounter with backpack (each time I have to install my composer dependencies on a freshly installed PC, I had to do it many times this past weeks...). That's not a "new issue" for me, I started with backpack on the begining of 2020 and had encountered this as the begining .

What is strange is that all the other dependencies are correctly downloaded, but not backpack !

I think we should make a try for a fresh install, while enabling debugging info on composer (if there is any !) to see what differs from backpack versus the other packages that are downloaded correctly ?

[pxpm]

Hum.. from what you describe what I think is happening is that you hit the memory limit alocated to php or the max execution time.

When you run composer update it will build a tree list of dependencies and version correlations. That sometimes takes up to 1/2GB of memory. Also some versions of composer are bugged so I suggest you run the latest one.

I had that problem several times in shared environments, my solution was:

• do the composer update in local
• push the composer.lock to the server
• in server delete the vendor folder and run composer install instead of composer update.

Later I updated to a VPS with 4GB a no more composer errors.

[LemarinelNet]

My dev machine has 12GB RAM and I have the exact same issue as on my server which is 16GB ram.

And I don't face this issue on composer update, but on composer install...

Nonetheless, doesn't hitting max executing time or memory limit would result on an instant crash !?
Also, my web server is way more faster than my dev machine and also is the network. It should so complete the composer install quickly than my machine, but still hangs up on backpack !

Finaly, if it was a execution time issue, running "composer install" the second time should be quicker because it reuse all the downloaded files from cache, and so should make the deal.. But it's not.
Every single time I run composer install, it hangs on downloading (stuck on "connecting...") backpack, whatever the time it took to achieve the other packages !?

REAAAAALLY strange !

[LemarinelNet]

Interresting, this issue happens with other composer packages :

https://github.com/mpdf/mpdf/issues/880

[pxpm]

Indeed it is strange. The only thing that I can remember of maybe it's the size of our package it is 34MB+ while I am pretty sure other packages are way way way less than this. We have lot's of assets js/css to make all field types work and provide the functionalities.

The thing I see is common between us and mpdf is the big bundle size.

The place where you download the other packages is the same where you download backpack. But maybe, and it's just a maybe, the way composer deals with packages bigger than X MB might be different, like downloading in chuncks or something like that.

Like @tabacitu said I think we can't do much more than trying to reduce the bundle size.

I am using: Composer version 1.8.6 2019-06-11 15:03:05 in a machine with 4GB RAM, just increase PHP timeout limit and it works.

So sorry I can't be much more help.

[LemarinelNet]

Made a test a few minutes ago on my home machine with verbose composer (-vvv) :
Downloading (connecting...) Following redirect (2) https://codeload.github.com/Laravel-Backpack/CRUD/legacy.zip/acc4df836505a612d1dd15a39ce02be87e36577d Downloading https://codeload.github.com/Laravel-Backpack/CRUD/legacy.zip/acc4df836505a612d1dd15a39ce02be87e36577d
It did in fact passed quickly from "connecting..." to "downloading", but did stuck a while on "downloading" without any progress for long minutes. Then the download did happened pretty quickly.
I wonder if github is not "building" the zip live, each time a user ask for it ?
Perhaps it's kept in cache then for a short while, then serve it from cache if present, else rebuild the zip ??

Relaunching from my work machine (another location) and downloading was almost instantaneous (file in cache on server, had deleted it from my user cache)...

PS : that's true that backpack is 10x bigger than the biggest of my packages in my composer cache folder... If github is truely building zip on demand, it could explain the issue !

[tabacitu]

If github is truely building zip on demand, it could explain the issue !

I hope that's not it, because in that case there's not much we can do about it 😆 🙄

I can confirm this has been happening to me too, in the past few weeks. Then again, Github has had entire hours of downtime these past few weeks, soo... maybe it's not us it's them?! For me, it's been the PHP memory limit exhausted:

PHP Fatal error:  Allowed memory size of 1610612736 bytes exhausted (tried to allocate 4096 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/Solver.php on line 223

Fatal error: Allowed memory size of 1610612736 bytes exhausted (tried to allocate 4096 bytes) in phar:///usr/local/bin/composer/src/Composer/DependencyResolver/Solver.php on line 223

Check https://getcomposer.org/doc/articles/troubleshooting.md#memory-limit-errors for more info on how to handle out of memory errors.%

I have no idea what we can do to fix this... Other than separate Backpack's asset into a different package, so that they're each smaller. I'm totally open to suggestions. I guess in localhost this isn't a huge deal, since you can do COMPOSER_MEMORY_LIMIT=-1 composer require backpack/crud and any other command really, but if this happens in production, and basically prevents auto-deploy... this will really suck... But screw that, no, it's not ok on localhost either... COMPOSER_MEMORY_LIMIT=-1 is just a workaround. Damn you huge feature set! 🤣 🎉 🥳 😞

[robertotcestari]

I also had this memory limit issue with composer.
Using composer v2.0 (which is still in beta)(https://php.watch/articles/composer-2) seemed to resolve for me, and it is way faster.

This was really annoying and happened with composer _update_, _install_ and _require_.

[tabacitu]

That's great to know! Thanks a lot for the info @robertotcestari !

[srigurubyo]

Hitting the issue again today... Even after multiple tries same status.

$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 0 installs, 3 updates, 0 removals
  - Updating backpack/crud (4.1.13 => 4.1.14): Downloading (100%)    Update failed (Failed to execute (9) unzip -qq -o '/home/sribharathi/saeWork/laravel/gosafe/vendor/backpack/crud/e71c5bab156d2863e4e3ed74c7304500' -d '/home/sribharathi/saeWork/laravel/gosafe/vendor/composer/e225c817'

[/home/sribharathi/saeWork/laravel/gosafe/vendor/backpack/crud/e71c5bab156d2863e4e3ed74c7304500]
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of /home/sribharathi/saeWork/laravel/gosafe/vendor/backpack/crud/e71c5bab156d2863e4e3ed74c7304500 or
        /home/sribharathi/saeWork/laravel/gosafe/vendor/backpack/crud/e71c5bab156d2863e4e3ed74c7304500.zip, and cannot find /home/sribharathi/saeWork/laravel/gosafe/vendor/backpack/crud/e71c5bab156d2863e4e3ed74c7304500.ZIP, period.
)
    Would you like to try reinstalling the package instead [yes]? 

[pxpm]

Can you clear composer cache, or try to update composer to version 2.x like @robertotcestari suggested?

[tabacitu]

Wait - is anybody/everybody having this problem using hirak/prestissimo by any chance? I am.

[LemarinelNet]

I don't use this plugin.

[srigurubyo]

I neither use that package.

[tabacitu]

In other packages online, it seems that composer clearcache (sometimes with an additional composer global update) fixes the problem for some people - when the error is the unzipping. Could you please try that and let me know if it does for you? Mine is a little different.

[tabacitu]

Update: Personally, I haven't been getting this error in 2 weeks. Soo... I'm kind of hoping it wasn't our fault, and it was a Github glitch. If anybody is still getting this error, please reply here and let us know. If nobody confirms it's still happening, we'll close it after 2 more weeks.

Cheers!

[srigurubyo]

I could upgrade to 4.1.15 without any issues. As @tabacitu mentioned, the issue at github end would have caused this.

[tabacitu]

Pfiew. Let's close this then. Thanks a lot for the help and patience everyone!