Composer: local-global repositories (composer link command)

[fprochazka]

Hi,
I'm writting simulatenously few libraries and it's realy pain to commit everything and for every few lines write $ composer update. So it would be nice to have new kind of local repository or something, that wouldn't copy the dependency into project, but rather link to it. So when I change something in one project, it affects the other one immediately and I don't have to commit stuff and run update on composer.

Some command like "link this library, but don't touch it, just autoload it".

Or is there any suggested worflow on this problem?

[Seldaek]

There is no way to do it with composer at the moment, obviously you can rm -rf vendor/ and then symlink your repo back at this place manually. This should work as long as you don't update, if you udpate, I'm honestly not sure what can happen. This voids your warranty :)

[fprochazka]

I'm pretty sure it wouldn't do anything catastrophic, as long as there are uncommited changes in the repository. I've just bulk replaced something and the IDE replaced it even in the vendor directory. The composer complained than, about the uncommited changes.

I think that is the best behaviour possible! I had my own vendors script inspired by Symfony's vendor script and before this https://github.com/Kdyby/sandbox/commit/37248738f9d7717750fb31e9ad3c28d5e62e24f4 commit, it managed to destroy few days of work :)

So now the question is, could there be added some "create symlink only" respository? I'm not sure if I find time to create a pull request, but I will try.

[Seldaek]

I think the way npm does it (see docs: http://npmjs.org/doc/link.html) sounds pretty awesome. If you can copy that entirely, go for it:)

[Seldaek]

Then the only thing that should be done is that the installers should check if the package dir is a link, and if so they should leave it alone or warn user or something smart, but not delete it and overwrite with something else, and especially not recurse into the symlink and delete everything.

[pavelkucera]

I would welcome this type of repository too because of the same reasons as @HosipLan. It's always "type a couple of lines, commit them with message á la 'update', run $ composer update... Oh, there's a typo! So, again..." and that is a way too long for development :).

[Seldaek]

I agree it can be useful, but do you guys realize you can just go in your vendor directory's clone and just develop in there directly? Install with --prefer-source to make sure you get a clone, and then you just go in vendor/foo/bar, and eventually change the git remote with git remote set-url --push origin <your fork url>.

[fprochazka]

Well, yes I do, but I find it really paintfull. I tried something like that long ago, when I was using git submodules. It's annoying for so many reasons. I will try to implement the linking into composer and send pull as soon as I can :)

[Seldaek]

Alright, I find it quite practical to work directly in the vendor dir for testing changes easily in my application. That said, I'm happy if you work on a link command either way :)

[dantleech]

This would be a nice feature..

[minutephp]

Yes, this is a long sought after feature.. right now I'm doing this to get a similar result (see the "Brave new Local Development" part)

http://gos.si/blog/composer-development-with-local-dependencies

[rrajkomar]

@minutephp : What you suggest in your post (which I had already read prior to creating issue #4152) is nice but would only work for the current version you have on your machine (if you switch branches dependencies content get switched as well).
Using the repositories option in the ~/.composer/config.json file, you can achieve the same result, you would also be able to handle branches and tags without issues (when using git, AFAIK it would not work with other VCS, i.e. with svn you can't do it because there's no local repository) but as mentionned in the issue it is not very intuitive.

Maybe a solution would be to have another configuration option that would work the other way around : the global config could override values defined in the project's composer.json

For the moment, since I'm working mainly with git, I can work around the issue with the reposiotires option set only in each developer's config.json (and using the same method on the different servers I use).

[ahuszko]

This may help you to improve local package development: http://ahuszko.ghost.io/compact/

[staabm]

@Seldaek I think we can close this one, because we have PathRepository now?
https://getcomposer.org/doc/05-repositories.md#path

[Seldaek]

No I'd still like to investigate a better solution.. path repo has several issues IMO it's a good simple solution but not really the best.

[Petah]

+1

[Seldaek]

@Petah there is a subscribe button on the right to avoid spamming people with useless +1s.

[Petah]

Thanks :+1:

[mreschke]

The path repo can get tripped up sometimes for odd reasons with versioning...not a bug, just logic I can't figure out. But for me the path repo works amazingly for package development. @Seldaek what are some of the issues you are seeing?

[rudiedirkx]

As I see it (I'm new) _path repo_ only works for the owner/primary developer, since the location is in the shared composer.json. This should be a dev config IMO, which means another (optional) file that is not shared, like composer-dev.json.

I don't understand at all why the package is symlinked or copied. Why not just read from the source?

1299 is full of good ideas IMO:

• separate file
• no symlinks
• $autoloader->setPsr4() as a back-up, (which helps me now)

Sorry if I'm repeating discarded/irrelevant ideas.

[rrajkomar]

@mreschke : Actually the path repository type does not match the requirement at all since the idea is not to add another source for the dependencies but simply to be able to tell that on dev machines private packages dependencies should be retrieved locally instead of through the otherwise used composer typed repository. Therefore the project composer.json file structure should not have to suffer changes for this feature support.

@rudiedirkx : Using another file that would not be commited (much in the same way as symfony's parameters.yml file) could be a solution if Composer could prioritize it as the first place it has to look into for packages before the usual (/comitted) composer.json file. Although there would be the composer.lock (if present) mismatch issue to handle as well.

[Seldaek]

Another util by @franzliedke that seems to be a good fit for this use case: https://github.com/franzliedke/studio

[franzliedke]

Thanks for mentioning me, @Seldaek. This is, in fact, precisely the use case that Studio makes possible.

With the latest version of Composer, I can finally prepare the new release and will post a proper announcement on my blog.

[Basster]

Thanks, @Seldaek and @franzliedke. Exactly was I was looking for in #4125. I'm happy now. :)

[b01]

@Seldaek I'd been having problems with using ~/.composer/config.json "path" method on my Macbook Pro for some strange reason. Using @francisbesset Studio worked with no issue.

[francisbesset]

@b01 not me but @franzliedke ;)

[fboutin-pmc]

I was looking for a composer link command as well. Studio looked good but I think it's pretty bloated for what it does. So we finally ended-up with manually make use of Symbolic Links combined with this handy script that re-deploy linked components,

composer run-script post-install-cmd -vvv -- --redeploy

Simply put,

1. You obviously git clone the COMPONENT repo you want to link to
2. In the project where you make use of the component, you composer require/install as usual, followed by rm -R vendor/COMPONENT
3. You link with ln -s FULL_PATH_OF_COMPONENT_REPO vendor/COMPONENT
4. You run the aforementioned script and voilà!

[rudiedirkx]

@fboutin-pmc That's what I do too, but with a global package that maintains the links per project.

[franzliedke]

@fboutin-pmc Care to explain why you think Studio is bloated? I would obviously love to know how, and if I can change that. I'd say feature-completeness is a bigger problem. ;)

[atrauzzi]

Up front, the requirement of having to define anything in a consuming repository to support link functionality is a little inconvenient. I think that's the beauty of npm link really -- it's part of the ecosystem.

[nerdbeere]

the requirement of having to define anything in a consuming repository to support link functionality is a little inconvenient

I can't stress this enough. I have to admit I'm pretty new to the whole PHP / Composer Ecosystem, but coming from npm, developing packages locally without a proper link command is a real pain for me.

The beauty about npm link is, that it doesn't require any changes to your code. Maybe that's just me, but when I work on a feature I only want to see changes in files that actually belong to this feature.

[iamandrewluca]

https://johannespichler.com/developing-composer-packages-locally/

[atrauzzi]

@iamandrewluca - While this symptomatically gets the desired outcome, structurally, it's not great. We need a way to do this without having to alter the composer.json file and risk accidentally committing these overrides.

This is why the major comparison here is to npm link.

[dave-newson]

As it's been about 10 months and this thread has no conclusion, so I'd like to give a bit of an overview, especially for anyone else coming to visit this thread, as I did, looking for an equivalent to npm link

TL;DR

If you want a swiss-army-knife and fine-grained control of your package development, look into Studio.

If you want an npm link equivalent, look into composer-localdev-plugin

Edit: continuously edited because I got the information wrong multiple times.

Overview

The official Composer mechanism

https://johannespichler.com/developing-composer-packages-locally/
https://gos.si/blog/composer-development-with-local-dependencies/

As pointed out by @iamandrewluca - the official mechanism involves modifying the project's composer.json to establish a local link, including specifying the path there the library can be found.

People feel this approach is flawed because:

• It requires each project to opt-in to this linked version of the library
• It requires an edit of a file tracked by VCS

Do it manually

You could also jump into the dirs and set up the symlinks yourself in the vendors dir, but that's a whole bunch of effort and I _believe_ we're programmers and against the idea of making humans do the work of machines.

composer-localdev-plugin

https://github.com/gossi/composer-localdev-plugin

This one is like the infant-version of what most of us are after - it provides a behavior similar to npm link.

This installs as a composer plugin, and you need to manually maintain a localdev section in your ~/.composer/config.json file.

When you call composer install your home-directory-based override file will control if a library is symlinked rather than installed, providing a direct equivalent to npm link.

Issues include:

• It's an external plugin people need to discover exists (boo)
• Setting up the config is a little fiddly.

composer-localoader-plugin

https://github.com/rudiedirkx/composer-localoader-plugin

Similar to the above, however rather than maintain a global reference file in your home directory, a single composer-locaload.json file will be added to your project directory (which can be gitignored).

This lets you opt-in on a project-by-project basis what you want to override, however if you actually want an equivalent to npm link (with overrides applying globally) this isn't a helpful feature.

Issues include:

• Overrides are per-project, not global. Depends on your use case.
• It's a global plugin for Composer, a little stale, has old dependencies which cause conflicts.

studio

https://github.com/franzliedke/studio

This is kind of a swiss-army-knife of package development, and provides a good way to create/link/unlink/destroy your packages on a similar case-by-case basis to composer-localoader-plugin

Using studio load <libdir> will add a studio.json file to your project (which you can add to gitignore), and this file controls the symlinking on composer install, and Studio itself can do other fun nifty things.

Problems:

• Overrides are per-project, not global. Depends on your use case.

My Use Case

Just to clarify why i'm here, and add to the voices that think composer should include this kind of functionality out of the box:

I'm writing a library that's framework-agnostic (ala Doctrine).

To plug this library into various frameworks (Symfony2, Symfony3, Symfony4, Symfony Flex, Laravel) I need to write various Bundles/libs to wrap my framework-agnostic lib, and I also need to set up mini-applications so I can test these wrappers against 5+ different frameworks.

In all, this results in ~6 applications on my local machine which I want to use my local version of code, rather than pull the lib every time.

With a feature like npm link this is attained in a matter of minutes - simply link the dependency, running composer install for my various test apps (all automated, thanks) and I can continue with my day-job.

With the current composer-suggested method I have to change 6+ composer.json files and remember not to commit them - as does everyone on my team. If this method were suggested by another dev in my department I would laugh and reply with "ew no there must be a better way".

What I'm going to do now

I really want an npm link equivalent, and composer-localdev gives me this capability, albeit in a slightly clunky way, so I'll be looking to contribute by efforts to improve this library so I don't have to maintain edits to the ~/.composer/config.json file by hand (eg. a composer link and composer unlink command extension).

I hope this post will serve to save people the 40 minutes I've just spent on this, and maybe as a nudge to the Composer team to investigate this feature further, nearly 7 years after it was originally proposed.

[Seldaek]

@dave-newson as far as I can tell, your use case would be very well served by loading your local version of your lib through a path repository as a symlink. I see that studio and co can do more, and maybe Composer will one day too, but for that particular use case path repos are more than enough.

[franzliedke]

@Seldaek I think his point was that path repositories require modifications to versioned files, which can be undesirable when using this setup only locally to e.g. test new versions of the linked library. Studio does quite a bit of work to avoid that.

@dave-newson Thanks for the kind words. A blog post has been on my to-do list for far too long. :see_no_evil:

[Seldaek]

@franzliedke I understand that's the main benefit yes, but it sounds to me like he has a few local test apps that wouldn't really suffer from that problem.

[dave-newson]

@Seldaek For me it's really a question of scale.

Yes - you're not mistaken - I can go into each of these libraries and I can modify VCS-tracked files (ew) to point to specific paths (ew) on my local machine, and it _will_ work.

However, the more libraries (2) and apps (4) that I have to do this with increases the likelihood of mistakes, human error, and wasted time. If you then scale this up by the number of developers I have in the office who also need to work on this code (3) you get.. 2x4x3 = 24 _manual edits_

That.. well that's _silly_ . We're not ever doing that, and we'll find an alternative route (composer-localdev-plugin) if Composer won't support our use case.

And while I've fixed this problem by going to a third party, this question of scale expands to all libraries in the wild who want the same kind of behavior - that, to me, looks like a serious black-hole of man-hours.

Rather than continue to argue in a circle, can I ask:

• Are you still open to supporting a global library-symlinking feature, similar to that of composer-localdev-plugin with a command like composer link (see comment from 2012)?
• If yes - as I'm not overly familiar with composer internals - can you think of any pitfalls to implementing this?

[hacktivista]

FYI https://getcomposer.org/doc/03-cli.md#composer-home-config-json

[atrauzzi]

@dave-newson - 💯 The truth is that based on what's needed here, the workarounds and alternatives suggested don't meet the need and are not an undertaking I'd share as a best practice with my team.

I recently aliased a package using yarn -- it was such a cinch and didn't get in my way at all. There was virtually no setup, just one command in the source repo and another to bind it in the consuming project.

The truth is, this is a feature that would serve the community far more than is being initially gauged by reducing friction when working with packages in-context. Think of how much easier it will be to tell people how to boot a local checkout of a project to submit bug fixes from within the projects they're experiencing the issues in! Heck, it might even help prevent bugs by encouraging people to understand the libraries they consume better.

Those two reasons alone are enough to make this worth having as an official feature in composer.

[esynaps]

The solution proposed by @hacktivista is not sufficient.

This file allows you to set repositories and configuration for the user's projects.

In case global configuration matches local configuration, the local configuration in the project's composer.json always wins.

You can not overwrite a VCS repository (used by the remote) and defined in composer.json with a path repository defined in config.json

[hacktivista]

@esynaps... you could define a global VCS repo in your local and a global VCS repo in your remote... as in https://getcomposer.org/doc/05-repositories.md#using-private-repositories

A VCS repo can be a path (for your local repo).

[atrauzzi]

@hacktivista - That still doesn't solve the fact that it requires a lot of extra machine-specific setup.

This approach isn't a _solution_ to the original problem. Symptomatically obtaining an outcome is very different than ascertaining the spirit of a desired feature and not creating extra work or scalability problems along the way.

[hacktivista]

I wouldn't say it's "a lot". However it might not be the best fit for your use-case.

Still it's a valid solution for some use-cases, my intention for posting it here was to inform people looking for global-local repositories (as I was), so they can do it as explained in the posted link.

[atrauzzi]

I'll put here what I usually say in response to _"workaround"_ answers: Be careful you aren't hijacking the discussion or giving maintainers a bad signal (that the issue is not valid).

It would be more on topic to not reiterate the same workaround more than once. In fact, that's why I'm replying still -- to assert that what's been brought up so far (including workarounds) doesn't resolve this ticket. That way nothing gets misinterpreted on a skim (it happens 😢).

[gnutix]

This issue is still a huge PITA, and neither https://github.com/gossi/composer-localdev-plugin nor https://github.com/franzliedke/studio (v0.13 and v0.14-alpha) currently work for me : I spent two days trying to get a proper setup with a working autoloading, unspoiled composer.json/lock files, and no mess with composer branches in the vendors... no success. And don't tell me about the dozen of blog posts with "workarounds", they don't do much good.

[lkraav]

I landed on this issue when my "just work on vendor clone" dependency package defined a new dependency.

Running composer update to install the new sub-dependency in main app can't go forward, because it looks for dev-master updates on GitHub, which I don't want to push vendor clone wip code to (yet).

Fortunately composer detected unpushed changes and at least didn't unexpectedly destroy work :+1:

Like others before, I'm also thinking some kind of a "silent local link" mechanism to avoid changing version-controlled composer.{json,lock} files would be useful.

Dec 2019: Studio still looks like the most advanced package available to take care of this problem.

[lkraav]

https://github.com/Letudiant/composer-shared-package-plugin doesn't seem to be presented in this issue before.

[pawawat]

any update on this problem?

[henzeb]

In case anyone lands here, None of the above mentioned packages worked for me the way I wanted it, so I wrote my own. Should work for composer 1 and 2. Currently at Alpha, but I've been testing it for a week, and it seems stable.

It goes back to the very basics, just replace a package with a symlink (it means the package should already exist and required in your project), without touching composer.json or lock files.

https://packagist.org/packages/henzeb/composer-link

[gmazzap]

Hi @henzeb thanks for your work.

There's the discussion about to bring "link" functionality natively in Composer.

The discussion started 8+ years ago in this issue and recently I'm trying to move it forward. I've written a requirements document on how it could work, but I wrote no code yet waiting for maintainers feedback.

Considering the experience you gained on the topic, if you could find the time to give some feedback on https://gist.github.com/gmazzap/b71e0d4eb99e0657d939bbc8d856bc7d that would be much appreciated.