Yarn: Allow subdirectories within git repos in yarn install

[akshetpandey]

Do you want to request a feature or report a bug?
Feature

What is the current behavior?
Unable to install packages that are inside subdirectories in git repos. This is a very common case for monorepos and just repositories that can't have the node package in the root repo.

If the current behavior is a bug, please provide the steps to reproduce.
Not a but

What is the expected behavior?
There should be an availably syntax for yarn add and yarn global add that allows to add packages that are inside subdirectories in a git repo.

It make it much easier to use unreleased code when testing unreleased versions for issues/regressions or fixes that aren't live yet like this ticket https://github.com/Microsoft/code-push/issues/428

ex. packages affected by this issue
https://github.com/Microsoft/code-push/tree/master/cli
https://github.com/Microsoft/BotBuilder/tree/master/Node
https://github.com/facebook/react-native/tree/master/react-native-cli
https://github.com/babel/babel/tree/master/packages/babel-cli

Related npm issue: https://github.com/npm/npm/issues/2974

Please mention your node.js, yarn and operating system version.
yarn versions v1.2.1
node version v8.7.0
macOS Sierra 10.12.6

[akshetpandey]

There are a couple of good ideas for syntax to use to specify subdirectories in the linked issue (npm/npm#2974)

Use pip's way of using a query parameter
git+https://git.repo/some_repo.git?subdirectory=subdir_path#branch
or
git+https://git.repo/some_repo.git#branch?path=subdir_path
Although I would assume most url parsers won't parse this one.

Use :: as it is not a valid token for a branch name
git+https://git.repo/some_repo.git#branch::subdir_path
git+https://git.repo/some_repo.git#::subdir_path

[lukescott]

Another idea: git+https://git.repo/some_repo.git/subdir_path#branch. If I were to assume this feature existed, that's what I would try first. The other options would require someone to look up the syntax.

[akshetpandey]

That might be ambiguous depending on how the git server urls are setup.

The url might be something like

git+https://github.com/yarnpkg/yarn/<subdir_path>#branch

Then it become hard to say what is part of the repo path and what is the subdir_path

[lukescott]

Not sure if this applies to every repo, but with Github both git+https://github.com/yarnpkg/yarn and git+https://github.com/yarnpkg/yarn.git work. You could assume anything after .git/ is a sub-directory.

Regardless of syntax it would be nice to have this feature either way. It would be cool to make the syntax intuitive though :).

[akshetpandey]

Oh yea, I was mostly talking about other website, not really github. Sorry my example wasn't the best one to explain it.

[stealthybox]

Long discussion about supporting this in npm:
https://github.com/npm/npm/issues/2974

There are some suggested separators in the thread.

[joshcartme]

I would also like to be able to do this to facilitate working with a monorepo that contains a lot of sub modules. I also found #1570 which discusses this. That was closed in favor of https://github.com/yarnpkg/rfcs/issues/15 but https://github.com/yarnpkg/rfcs/issues/15 doesn't exist anymore so it's hard to tell what if anything was done or decided there.

[matheo]

so yarn doesn't support this yet?
all right, no simpler option than publish and use npm packages :(

[erykpiast]

Yeah, rising popularity of monorepo architectures makes it a crucial feature. Is there any workaround for this, for now? Discussion about public API tend to be years long...

[lukescott]

My # 1 use-case for this is pulling in bug-fixes from monorepo's that haven't been merged or released yet. I often want to keep most of the original packages, but pull in one of the sub-repos with a bug-fix. Currently I am unable to do this, and the workarounds are complex. Being able to reference a Github branch would greatly lower the resistance.

I know there are some people at npm that are highly resistant to the idea of this, and monorepos in general. However, many of the contributors here work on projects that use monorepos. I would suspect that if yarn were to implement this feature and it became popular enough npm would follow suit. A lot of yarn ideas have been implemented in npm later on. yarn to me is the babel of npm.

[sdhull]

@erykpiast couple workarounds that I can think of:

1. clone the target repo somewhere and then use yarn add /path/to/repo/subdirectory
2. or possibly (if it's in your own monorepo) use yarn workspaces?

Or -- this is maybe even crazier -- add the remote in question as a submodule and (optionally move your project files into a subdirectory) _then_ use yarn workspaces?

Disclaimer: I haven't tried either of these approaches and they both add overhead for new developers & your CI builds...

[jwdunne]

Or -- this is maybe even crazier -- add the remote in question as a submodule and (optionally move your project files into a subdirectory) then use yarn workspaces?

I can confirm this works.

Upgraded to webpack 4. Create React App doesn't have support yet but there's a PR that's waiting to be merged - unfortunately webpack 4 has changed it's plugin API. Doubly unfortunate: create-react-app is repo containing multiple packages. Adding the forked PR as a submodule and adding the package using 'file:' syntax worked a charm.

Literally saved an entire day or more of work with this work around.

[micimize]

It's alarming that this isn't on the roadmap for either npm or yarn due to the rise in popularity of monorepos. The git+... syntax makes contributing to single-package projects a breeze, and by contrast not having it for subdirectories makes contributing to monorepo projects a total headache.

[jwdunne]

I agree. After a few months down the git submodule, file: route, it's not ideal. Onboarding other developers is more difficult. My colleagues have to worry about initialising and updating the submodules. In their defence, it used to be as simple as running yarn to get themselves going.

Monorepo support would be a dream come true.

[Frikki]

yarn + lerna (using yarn’s workspaces) works to manage monorepos.

[jwdunne]

A second tool to install from a monorepo isn't ideal. It should be as
simple as installing a package from a single package repo, using yarn add ...

In general, it doesn't help JS' 'little problem'. A use case comes up. We
use a new tool when an existing tool is the perfect home.

On Sun, 1 Jul 2018, 11:34 p.m. Frederik Krautwald, [e-mail]
wrote:

yarn + lerna (using yarn’s workspaces) works to manage monorepos.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/yarnpkg/yarn/issues/4725#issuecomment-401638034, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAjP2lWrm2u8tOiIZQ5D-oKre9BqTj5-ks5uCU6SgaJpZM4P7hjl
.

[micimize]

@Frikki I don't believe lerna has much submodule support, and isn't a "consumption tool" anyways.

@jwdunne For now I just npm publish a namespaced version of the fork and depend on it

[ramasilveyra]

Hello! I built https://github.com/ramasilveyra/gitpkg to "publish" packages from monorepos as git tags, it may help.

[stealthybox]

@ramasilveyra that looks super useful -- thanks for the addition 👍

[robermorales]

Some news in supporting git subdirs as install sources?

Thanks,

[nickiaconis]

I encountered this issue today when needing to yarn add a yet unaccepted/unpublished fix I wrote for a monorepo. +1 from me.

[UrielCh]

Foking DefinitelyTyped cause the same king of issue,
Each time I fork this git, to prepare a new pull request, I need to depended on a sub directory of my forked git.
Is file:// is currently the only solution ?

[nickiaconis]

@UrielCh I was able to use https://github.com/ramasilveyra/gitpkg as a workaround. It might work for your situation too.

[UrielCh]

Thx for this suggestion, creating a new git for this fork will be simple.

[t-benze]

Hello! I built https://github.com/ramasilveyra/gitpkg to "publish" packages from monorepos as git tags, it may help.

This may look a bit hacky at the beginning, but after some reasoning it seems to be a legitimate solution to the problem 👍

[yanickrochon]

Seriously, gitpkg works! It creates tags inside the project repository, and can be used with repositories containing multiple packages.

And this prompted me to ditch npm because only yarn can fetch nicely from Git.

[Preen]

+1

[micimize]

Hello! I built https://github.com/ramasilveyra/gitpkg to "publish" packages from monorepos as git tags, it may help.

Just used gitpkg and I think it's such a good solution that it should be added as a note in the docs for this use case

[EqualMa]

I built a service https://gitpkg.now.sh/ to download the sub folder of a git repo as a .tgz file, so that it can be installed by yarn and npm. With this service, package users can use the subdir directly and there is no need for package developers to publish the subdirs.

For example, if you want to add babel-core in branch next-8-dev as dependency,
just use:

yarn add https://gitpkg.now.sh/babel/babel/packages/babel-core?next-8-dev

also works for npm install.

Note that I must say sorry to @ramasilveyra because my service's name conflicts with the tool he develops but I didn't know that before.

[dword-design]

@EqualMa This looks pretty amazing. I'll have a look at this soo.

[erykpiast]

@EqualMa It'd be even more amazing if it works for projects with a build step. But anyway, great job, thank you so much!

[uriel-sf]

Do we still need gitpkg? Running latest lerna and when run in independent-mode it will publish git tags of each component.

[EqualMa]

@erykpiast Finally GitPkg service supports custom scripts, allowing you to run a build step in postinstall or other scripts! You can checkout the docs or just try adding custom scripts at https://gitpkg.now.sh

[erykpiast]

@EqualMa that's even more than awesome! Thank you!

[Acidic9]

Any updates to this? Is it planned or anything?

[artemjackson]

@EqualMa awesome! great job!

[jason-krypton]

I'm having an issue trying to fork a repo (https://github.com/ianstormtaylor/slate) that uses workspaces and has multiple packages in the same repo. I only want to modify and import one of the packages. Trying to yarn add pointing to the directory that holds the individual package.json file gives an error...

yarn add https://github.com/kr-project/slate/tree/master/packages/slate#e73db650a332fc5ded3fd72cffb1fcd7279b85d7
error https://github.com/kr-project/slate/tree/master/packages/slate: Extracting tar content of undefined failed, the file appears to be corrupt: "Invalid tar header. Maybe the tar is corrupted or it needs to be gunzipped?"

I've also tried to check in the published files, but my understanding is this is a problem with yarn looking a tar only found at the root directory of a github repo?

So is the only way to fork a project that uses workspaces to use something like gitpkg? I'm a bit surprised that there doesn't seem to be more native support for this.

[LucidDan]

Just ran into exactly the same issue as @jason-krypton ... trying to work with a fork of a repo that uses workspaces, normally I'd just yarn add the forked git repository ... that's impossible now I guess? Wasting a decent bunch of time on this in order to change a couple of lines of code. :-(

[robclancy]

I have similar issues with ckeditor5. I'm trying out GitPkg now.

[bedney]

So @LucidDan and others.

I've found the following to work:

• You must be running yarn 2.1.X for this to work.
• See this page for the newer syntax that allows this: https://yarnpkg.com/features/protocols
• You have to add individual entries for each package in the monorepo to your package.json under either dependencies or, in my case, devDependencies:

"package-1": "my-org-name/mono-repo-name#head=my-git-branch&workspace=package-1"

It's disappointing that there isn't a way to just have yarn install the top-level monorepo and unroll all of the attendant packages into their proper place. Maybe there is and I'm just being stupid here? If so, someone enlighten me ;-)