Composer: cleanup inherited dependencies when removing a root dependency

[djmattyg007]

If I install a package that requires various other packages, and some of those packages require other packages, etc etc, and then run 'composer remove' to remove the original package I installed, none of its dependencies (that I never specifically required) are removed. There doesn't appear to be any way of keeping track of which packages were installed automatically as dependencies, nor does there appear to be any way of automatically cleaning these up.

Various Linux distro package management tools such as apt-get and pacman do support this kind of functionality if you're looking for example

[djmattyg007]

Even worse, I just discovered that I can't use composer remove to remove these dependencies at all. It looks like I have to manually edit the composer.lock file or first add them to the composer.json file before I can remove them with composer remove.

[curry684]

This is a known shortcoming indeed. Easiest way to fix it is rm -rf vendor composer.lock && composer install.

[alcohol]

Did you run composer remove --update-with-dependencies? Or just composer remove?

;-)

[curry684]

Why is that not default actually?

[alcohol]

Because then someone else will ask, "why is it default?", and then we get into an endless discussion about who has the biggest penis, and nobody wins in the end. Because, internet :d

[Seldaek]

@curry684 it's not default because it's generally not what you meant to do, and in many cases ends up updating way more than what people expect. The option is there though because sometimes without it you are stuck.

[curry684]

Not to get into the penis discussion, but I think that may have some unfortunate unintended by-effects. For example I include composer/composer. I then use the Symfony\Console classes in my code. I remove composer/composer, but its dependency on symfony\console is not removed, so my IDE continues to approve and autocomplete all my code. At runtime stuff starts to crash though because the autoloader is missing, confusion arises.

I don't really see the use case for leaving them around actually, as far as I can see it can only cause confusion like illustrated above. At the very least I think a message would be helpful like:

The following packages are no longer included because they are not referenced
by an active dependency:
  justinrainbow/json-schema:1.6.2
  symfony/console:3.0.2

Either remove them manually, or incude them explicitly if they are still required.

[Seldaek]

AFAIK they will still be in the autoloader as they are still seen as installed. The removal case is a bit special though and maybe for remove we could default to include deps yes, but then it also creates an inconsistency with the update cmd.

[curry684]

Isn't that even worse? That means that my code will continue to work in dev, and then when I deploy to live and composer install all hell breaks loose in production?

That would mean I'm completely dependent on the survival of composer.lock throughout the process, and if it ever gets trashed nobody has a clue what happened. Whilst it should at any point be possible to trash vendor and composer.lock as general cleanup before retrieving updated dependencies, as the project definition itself in composer.json should always install a complete system, it may just have unverified updated dependencies.

[Seldaek]

Well if it is still installed it will still be in the lock file so it'll
be in prod too.. If you lose the lock file and update then those
packages get removed yes. But you were using things without declaring
them as dependencies which is anyway subject to _all hell breaking lose_
at any time. n-th degree dependencies are _not_ your dependencies, and
you shouldn't assume they are. I think you are just arguing on principle
here, this isn't a real world issue IMO.

[curry684]

It's sort of hypothetical, fully agreed. We all know however that during development the IDE autocomplete is leading - if PhpStorm can find the class and auto-insert a use, we continue coding without bothering to check why the class is available.

I'm just arguing the point that IF I accidentally break my codebase because I screwed up on using nth-degree dependencies and removed the 1st-degree citizen, I would like to know it 3 minutes after I read a warning on the subject explicitly listing the orphans, not 2 months later during server deployment with no hint where to start. It makes the git rollbacks and the debugging just so much easier.

Plus, I don't see any valid reason for defaulting not to clean them up. And people are opening issues on the subject, that makes it a real world issue.

[alcohol]

Plus, I don't see any valid reason for defaulting not to clean them up.

Please, no penis debate.

[curry684]

It's not a penis debate, I'm asking the question as I'm not seeing any advantage at all to leaving them around whilst not referenced. It's a question, to which I think the answer is _"none whatsoever"_ right now, but I'm willing to be proven wrong, otherwise I wouldn't ask. It would become a penis debate if both options are defendable for equally valid reasons and I would continue to argue for a change.

As you may have noticed over the past few weeks I'm trying to help improve this project. To do so I must understand fundamental choices, and perhaps ask the odd dumb question in the process as I'm not immediately aware of how and why every choice was made over the past 4 years. Right now I think this behavior is broken in Composer and should be repaired. I'm asking the questions to find out if a PR will be accepted if I spend my costly time on it.

[barryvdh]

I have noticed the behaviour also when removing packages. To make it also seems that the --update-with-dependencies is expected behaviour. You may be right that is can remove more packages then expected, but currently it _is_ going to break sometime, eg. the next composer update, which is even more confusing.

Current behaviour:

$ composer require symfony/console
    - Installing symfony/polyfill-mbstring (v1.1.0)
    - Installing symfony/console (v3.0.2)

$ composer remove symfony/console
    - Removing symfony/console (v3.0.2)

$ composer update
    - Removing symfony/polyfill-mbstring (v1.1.0)

Expected behaviour would be what --update-with-dependencies does now:

$ composer remove symfony/console
    - Removing symfony/console (v3.0.2)
    - Removing symfony/polyfill-mbstring (v1.1.0)

@Seldaek Could you please re-open this issue for discussion.
@alcohol Could you please keep the discussion civilised, without immediately closing as a 'penis discussion'

[Seldaek]

Yeah I tend to agree that for the case of remove defaulting to include deps probably makes sense. I don't like the inconsistency but given that it would mostly do what people mean by default it's probably alright. I generally am against changing this for the update command though.

[Seldaek]

If you'd like to send a PR to switch the flag to an --update-without-dependencies, I think that is fine.

[stof]

This is a bit more complex though: compose remove should by default remove useless dependant packages IMO, but should avoid updating other dependant packages (which can be unexpected)

[Seldaek]

That's what I thought as well @stof but it won't update them if they are root requirements, only if they are n-th requirements that two of your packages require. That limits the scope of problems quite a bit. It's still a possibility of course.

[curry684]

Yep, I think the correct behaviour would be to remove the flag completely (or rather for now issue a deprecation message that it's become default), but only do the directly related work.

So to continue on @barryvdh 's example, the output would be:

$ composer remove symfony/console
    - Removing symfony/console (v3.0.2)

Removing orphaned packages no longer explicitly required:
    - symfony/polyfill-mbstring (v1.1.0) was required by symfony/console (v3.0.2)

The extra information helps the developer judge whether he needs to intervene.

As @stof correctly points out it should not update any packages in the process, that's for the developer to decide when he wants to composer update explicitly.

I may have time this weekend to look into this.

[Seldaek]

Ok that's gonna be a slightly more involved patch though especially as the Installer class has no knowledge of it running in "remove" mode, but if you want to spend the time it would definitely be a better solution.

[barryvdh]

That would probably be the cleanest solution, but imho just showing what packages will be removed will be enough for most people (who just remove 1 package at the time).

[curry684]

To be honest that's part of a bigger pet peeve of mine - it's nigh on impossible at times to even determine why a package was ever installed in the first place. install and remove should be more verbose about that, at least with -v.

[...rest of post trashed as I didn't know about a feature already existing...]

[Seldaek]

$ c depend paragonie/random_compat
symfony/polyfill-php70 requires paragonie/random_compat (~1.0)

[curry684]

See, I sometimes make dumb remarks :innocent: could use recursive lookup though.

[Seldaek]

For recursively figuring things out the tree view helps, with a grep context:

$ c show -it | grep -B40 paragonie
│  └──php >=5.5.0
├──guzzlehttp/psr7 ~1.0
│  ├──php >=5.4.0
│  └──psr/http-message ~1.0
│     └──php >=5.3.0
├──mtdowling/jmespath.php ~2.2
│  └──php >=5.4.0
└──php >=5.5
symfony/symfony v2.8.0 The Symfony PHP framework
├──doctrine/common ~2.4
│  ├──doctrine/annotations 1.*
│  │  ├──doctrine/lexer 1.*
│  │  │  └──php >=5.3.2
│  │  └──php >=5.3.2
│  ├──doctrine/cache 1.*
│  │  └──php >=5.3.2
│  ├──doctrine/collections 1.*
│  │  └──php >=5.3.2
│  ├──doctrine/inflector 1.*
│  │  └──php >=5.3.2
│  ├──doctrine/lexer 1.*
│  │  └──php >=5.3.2
│  └──php >=5.3.2
├──php >=5.3.9
├──psr/log ~1.0
├──symfony/polyfill-intl-icu ~1.0
│  ├──php >=5.3.3
│  └──symfony/intl ~2.3|~3.0
├──symfony/polyfill-mbstring ~1.0
│  └──php >=5.3.3
├──symfony/polyfill-php54 ~1.0
│  └──php >=5.3.3
├──symfony/polyfill-php55 ~1.0
│  ├──ircmaxell/password-compat ~1.0
│  └──php >=5.3.3
├──symfony/polyfill-php56 ~1.0
│  ├──php >=5.3.3
│  └──symfony/polyfill-util ~1.0
│     └──php >=5.3.3
├──symfony/polyfill-php70 ~1.0
│  ├──paragonie/random_compat ~1.0

[djmattyg007]

Did you run composer remove --update-with-dependencies? Or just composer remove?

The problem with this is that if I forget to include the --update-with-dependencies flag when removing a package, I have no sensible way of later removing the packages that would be removed had I included that flag to begin with. There is also no reminder that it's even necessary if I forget.

[curry684]

@djmattyg007 the issue is acknowledged and marked as a bug, no more need to argue the point.

[alcohol]

@djmattyg007 no that is not true? If you run composer update it will remove all of those dependencies.

Quick example:

rob@macbookpro /tmp $ composer require phpunit/phpunit symfony/yaml
Using version ^5.2 for phpunit/phpunit
Using version ^3.0 for symfony/yaml
./composer.json has been created
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Installing myclabs/deep-copy (1.5.0)
    Loading from cache

  - Installing sebastian/version (2.0.0)
    Loading from cache

  - Installing sebastian/resource-operations (1.0.0)
    Loading from cache

  - Installing sebastian/global-state (1.1.1)
    Loading from cache

  - Installing sebastian/recursion-context (1.0.2)
    Loading from cache

  - Installing sebastian/exporter (1.2.1)
    Loading from cache

  - Installing sebastian/environment (1.3.3)
    Loading from cache

  - Installing sebastian/diff (1.4.1)
    Loading from cache

  - Installing sebastian/comparator (1.2.0)
    Loading from cache

  - Installing symfony/yaml (v3.0.2)
    Loading from cache

  - Installing doctrine/instantiator (1.0.5)
    Loading from cache

  - Installing phpdocumentor/reflection-docblock (2.0.4)
    Loading from cache

  - Installing phpspec/prophecy (v1.5.0)
    Loading from cache

  - Installing phpunit/php-text-template (1.2.1)
    Loading from cache

  - Installing phpunit/phpunit-mock-objects (3.0.6)
    Loading from cache

  - Installing phpunit/php-timer (1.0.7)
    Loading from cache

  - Installing phpunit/php-token-stream (1.4.8)
    Loading from cache

  - Installing phpunit/php-file-iterator (1.4.1)
    Loading from cache

  - Installing phpunit/php-code-coverage (3.1.1)
    Loading from cache

  - Installing phpunit/phpunit (5.2.4)
    Loading from cache

sebastian/global-state suggests installing ext-uopz (*)
phpdocumentor/reflection-docblock suggests installing dflydev/markdown (~1.0)
phpdocumentor/reflection-docblock suggests installing erusev/parsedown (~1.0)
phpunit/php-code-coverage suggests installing ext-xdebug (>=2.2.1)
phpunit/phpunit suggests installing phpunit/php-invoker (~1.1)
Writing lock file
Generating autoload files

rob@macbookpro /tmp $ composer remove phpunit/phpunit
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing phpunit/phpunit (5.2.4)
Writing lock file
Generating autoload files

rob@macbookpro /tmp $ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing phpunit/php-code-coverage (3.1.1)
  - Removing phpunit/php-file-iterator (1.4.1)
  - Removing phpunit/php-token-stream (1.4.8)
  - Removing phpunit/php-timer (1.0.7)
  - Removing phpunit/phpunit-mock-objects (3.0.6)
  - Removing phpunit/php-text-template (1.2.1)
  - Removing phpspec/prophecy (v1.5.0)
  - Removing phpdocumentor/reflection-docblock (2.0.4)
  - Removing doctrine/instantiator (1.0.5)
  - Removing sebastian/comparator (1.2.0)
  - Removing sebastian/diff (1.4.1)
  - Removing sebastian/environment (1.3.3)
  - Removing sebastian/exporter (1.2.1)
  - Removing sebastian/recursion-context (1.0.2)
  - Removing sebastian/global-state (1.1.1)
  - Removing sebastian/resource-operations (1.0.0)
  - Removing sebastian/version (2.0.0)
  - Removing myclabs/deep-copy (1.5.0)
Writing lock file
Generating autoload files

[curry684]

He said:

I have no sensible way of later removing the packages

There is a way, through composer update, but that may also update 50 other packages and thus inadvertently break your project, hence not sensible.

[alcohol]

If a composer update breaks your project, then it is most likely that your requirements are ill-defined and need more thought and consideration. While I am not saying that composer update is a perfect solution, I would definitely consider it a sensible one.

[curry684]

I had a 2 year old project break last week because it depended on dompdf/dompdf:dev-master and we ran composer update. Turned out back when it was developed there was no stable available and we were tied to it, and the API at the time was marked 0.6 later and then overhauled for 0.7.

Shit like that happens, and it's fine that it happens in a controlled fashion when you hit update. We wouldn't have appreciated it happening as a side effect of just purging an unused package as it took over an hour to properly repair :smile:

Whether that makes it sensible or not - now that's a discussion noone's going to win.

[djmattyg007]

In most developers' minds, you'll find that they expect to use composer update to upgrade packages, not to remove them.

[curry684]

I was wrecking my brain with this issue tonight, and realized it could have far more consequences, possibly even breaking projects. The good news is it doesn't right away. The bad news is it may later on. Let's assert some preconditions:

1. symfony/symfony replaces symfony/console
2. composer/composer requires symfony/console

Now starting from a fresh empty directory I go:

$ composer require symfony/symfony composer/composer:@dev

This installs fine, as symfony/symfony already provides symfony/console it is not installed separately. Completely correct. Now I issue the following command:

$ composer remove symfony/symfony
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Writing lock file
Generating autoload files
$

This is very VERY wrong. In this case the remove decided not to remove anything, as symfony/symfony was still needed to fulfill the console dependency. It was however _removed_ from composer.json as requested. The correct action would have been to remove symfony/symfony and install symfony/console instead, to mirror the action a fresh install would do. So does update fix this as it does with the orphaned packages?

$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Generating autoload files
$

Nope, doesn't do anything. So I now have a project in which the entire Symfony stack is loaded, including autoloaders, including IDE autocompletion and everything, without a valid configuration for them to be. My dev, test and prod will even function fine if I use them from the same composer.lock that I already have.

Then one day in the future someone decides to clean up, removes lockfile and vendor, and hits composer install. Which only installs symfony/console, and hell breaks loose.

So yeah, the valid solution to this issue can only be to resolve the full tree following the removal, and diff it to the installed tree, removing orphans and installing newly discovered dependencies as it goes.

[Seldaek]

@curry684 yes, that's just how it is.. and I don't think it's really fixable as things are as we need to take into consideration what's installed. Maybe the root cause is that we prefer what is installed than what isn't. Maybe. But if we don't I'm not sure what the consequences are, probably in some cases every time you update it would cycle between remove A, install B and then remove B, install A. SAT solving is a hard problem without a perfect solution :/

[curry684]

Yep, SAT solving is complex, but I think we can implement this without touching that part.

In the end the golden rule should be that the lock should always be something that matches the requirements in the composer.json. Keeping the lock is a good idea for QA, but you should always be able to remove it, hit install, and still have an identically functioning system (barring package provider screwups). Essentially in an ideal world in any situation rm -rf vendor composer.lock && composer install should always have the same final result as composer update. Currently that rule is violated in the cases mentioned above.

The solution I think is to let the solver do what it's already good at - reducing a dependency tree to a flat install map. And after any remove operation create one from the install file, and compare it to the current install map per the lockfile. Uninstall/install any differences and you're done.

In the case of the replaced package this can have unintended by-effects. If I installed symfony/symfony:3.0.0 with Composer, and then removed symfony/symfony it would install symfony/console:3.0.x-dev as the best match, as Composer is in dev itself. This is an implicit unintended upgrade that should only happen when hitting update manually. The solution to that is simple though - when a package is removed, insert its replaced packages back in the install stack before the diff. It would do this for 40+ packages, discover all of them but one are neither installed nor needed so silently dropping, and install symfony/console:3.0.0 per the diff.

What surprises me is that this logic would be identical to what require should do after inserting a new root requirement, but apparently isn't coded like this. I assume it just explodes the new subtree on its own and acts on that, but I think the generic implementation would be better.

we need to take into consideration what's installed. Maybe the root cause is that we prefer what is installed than what isn't.

This is indeed the real question. I think Composer shouldn't give a shit about what's currently installed outside of update on the _package_ level, it should stick to its requirements in all modification operations, it should just prefer as much as possible to stick with current _versions_.

[djmattyg007]

Essentially in an ideal world in any situation rm -rf vendor composer.lock && composer install should always have the same final result as composer update.

If you're using wildcards in your dependencies, or any of your dependencies are using wildcards in their dependencies (etc), there isn't any guarantee that that will hold true.

EDIT: Actually I think I misunderstood, nevermind.

[davidsteinsland]

TL;DR: You are all talking about the update command, but what about remove? Why doesn't composer remove vendor/package remove the package _with_ its dependencies? I don't know the inner-workings of composer, but I bet that it's a no-brainer to figure out whether or not a dependency is still needed, or if it was installed as a second-degree dep.

Today I just removed a deprecated package. Later on, I was made aware that the build system still installed its 5-6 dependencies. Why? I don't know, but I had to do some weird remove stuff for it to work (#4947)

[alcohol]

composer remove --update-with-dependencies

Maybe if people bothered to actually learn about the commands they were using, none of this discussion would be necessary. There is plenty of documentation. Assumptions get you intro trouble.

[curry684]

Why do you complain about a TL;DR and then add more info to it that's just repeating what's already been said, adding to the TL;DR level for those after you?

The title of the issue is clear enough: "when removing a root dependency its indirect dependencies should be cleaned up". We know that, but it's a complex issue that's not going to be solved in a day. There are workarounds in this issue.

[davidsteinsland]

@alcohol

--update-with-dependencies Allows inherited dependencies to be updated with explicit dependencies.

Well, it's not super clear the flag will _remove_ packages. All I see is _update_.

[alcohol]

So clarify the documentation and submit a PR? :-)

[davidsteinsland]

Yes, I see your points @alcohol and @curry684. It was not my intention to repeat what was already said, and I understand that the problem may be more complex than what I originally thought.

Having over 4000 issues, I'm sorry I did not fully use the search function to my advantage (as I reported a duplicated issue today).

Would like to contribute to composer but I guess one reaches a limit of how many projects one can contribute to.. All the best, good luck, and sorry!

[barryvdh]

The problem is not that people don't read the documentation correctly, it's that the behaviour is currently not intuitive enough, so a documentation PR doesn't help much imho..

[curry684]

We could still do what @Seldaek suggested in https://github.com/composer/composer/issues/4895#issuecomment-182855867 until a proper solution is implemented.

[barryvdh]

Ok, did that in https://github.com/composer/composer/pull/5002

[Seldaek]

@curry684 if you really care about fixing this proper, would you mind creating a new issue referencing this with a summary of what should still be done? Too much backlog here so I'll leave it closed :)

[curry684]

:+1:

[asgardia7]

Hello, TLHR.

Hello could someone find solution about what to do if there is old dependencies in the lockfile (I don't know when but they where there for years). I don't want to use composer update 'cause it will change all the versions of my packages.

Is there another solution than manually remove the extra dependencies in the lockfile and do composer composer install ?

Lastly, how could I know which dependencies should be removed (not used by the packages listed in my composer.json?)

Thanks please,

PS :

• I think all of this must be possible 'cause when we do composer update, the orphans are well removed. Or should we run composer udpate and when it start to removing and install new version, we use CTRL+C then repairing the last broken package?
• Or re-add the package in composer.json then use composer remove. :D

[connorjclark]

Would like to know if there has been any progress on this feature in the last couple years. Running composer update can be quite daunting on large projects - doubly so for removing the lock file.