Yarn: Install from private github url

[jamiebuilds]

Do you want to request a _feature_ or report a _bug_? Bug

What is the current behavior?

You can't install a package like github:yarnpkg/yarn because it is private even though if

[2/4] 🚚  Fetching packages...
error [email protected]:yarnpkg/yarn.git: Tarball is not in network and can not be located in cache (/Users/thejameskyle/Projects/yarn-perf/[email protected]:yarnpkg/yarn.git)

What is the expected behavior?

It should be able to install the package like npm currently can.

[doot0]

This is also the case for private URLs that require basic authentication, e.g. git+https://gitusername:gitpassword@server/path/to/repo.git.

[milosivanovic]

I've provided a simple workaround for authenticating via SSH in yarn at https://github.com/yarnpkg/yarn/issues/513#issuecomment-253010084 that you can use until this issue is fixed.

[martynchamberlin]

Looks like @ramasilveyra closed https://github.com/yarnpkg/yarn/pull/971 in deference to https://github.com/yarnpkg/yarn/pull/1081, but that PR didn't actually fix the problem, at least for me. I'm manually grabbing a non-published bleeding-edge version of yarn so that it will have https://github.com/yarnpkg/yarn/pull/1081 in it, but I'm still getting this error:

Tarball is not in network and can not be located in cache [...]

Any suggestions what to change to get this to work?

[martynchamberlin]

Eh, ignore my comment. I had universally installed a stable release version of yarn and due to how my VM is setup, this was the one it was referencing instead of the bleeding-edge local version. So that's why I was getting that error. I've confirmed I no longer get that error on the latest version.

[stiang]

I’ll just add that I initially had some problems getting this to work _after_ it was reportedly fixed. It turned out that we had been using URLs like the following in package.json:

git+ssh://[email protected]:org/repo.git

Note the ":" between the hostname and the org name. When I replaced the ":" with a "/" it worked perfectly:

git+ssh://[email protected]/org/repo.git

I’m not sure if the first URL scheme is supposed to work, but I’m pretty sure we copied it from someone at some point, so we’re probably not alone in using it.

[BryanCrotaz]

We are working on a PR that will handle this URL format as detailed above by @stiang

[szimek]

@stiang Thanks for the tip! I changed:

github:org/repo#1.0.1

(which worked with npm) to

git+ssh://[email protected]/org/repo.git#1.0.1

and it works with yarn as well now.

[martynchamberlin]

I’m not sure it the first URL scheme is supposed to work, but I’m pretty sure we copied it from someone at some point, so we’re probably not alone in using it.

Given that a colon is the "Clone with SSH" Github syntax for cloning a repository, it's definitely good that we're now supporting this. Thanks!

[robmoorman]

Hard to see these issues still apply while this package is been put in the OSS as 'the package installer'. @szimek solution to git+ssh://[email protected]/org/repo.git#1.0.1 works though.

[BryanCrotaz]

@robmoorman do you mean that someone else has fixed this issue? Changing the URL format that is supported by npm is not a valid workaround for someone (like us) that has 100+ repos.

[robmoorman]

Hi @BryanCrotaz no it hasn't a fix yet, I see many people are struggling with the same use case you describe (too many repo's to change the url format as a workaround). But the 'workaround' as described above does actually solves the this issues in particularly. It it a fix for all issues; no. In my opinion yarn still needs to support at least all the url formats npm works with to make yarn a drop-in replacement.

[BryanCrotaz]

@robmoorman what did you mean by "Hard to see these issues still apply"? Not sure I see your point

[robmoorman]

Let me rephrase that; hard to see there is no solution yet (to support these url formats) :)

[BryanCrotaz]

@robmoorman ahhh I see now. Well we've been working on a fix for 4 days now - biggest problem is that the architecture of yarn is not documented. We've been forensically working out how it works.

[jonaskello]

If we could get support for semantic versioning also that would be awesome! What I mean is that something like this could work:

git+ssh://[email protected]/org/repo.git^1.0.1

jspm already has this feature but npm does not. Here is the relevant npm issue.

[jamiebuilds]

I agree with @isaacs here, I don't think it makes sense to apply semver to something that isn't explicitly semver. It also breaks compatibility with npm without a huge benefit.

[jonaskello]

@thejameskyle Actually the issue I referenced states that it will be added to npm (check the discussion at the end of the issue):

The CLI team still intends to ship this as part of giving hosted Git dependencies as much parity with registry dependencies as is practicable.

So I wouldn't see it as breaking compatibility with npm but rather keeping feature parity with npm, jspm and bower.

To me this feature makes perfect sense for organisations that are too small to maintain a private package server, but still need to share some packages between projects that (unfortunately) is not open sourcable. I think this would be a pretty common scenario. But maybe yarn has a different solution for this scenario?

[kaicataldo]

Is this currently being worked on? I ask because I'm happy to try to implement (would most likely need some guidance once I start digging in).

[BryanCrotaz]

@kaicataldo you could help work out why the test is failing on PR https://github.com/yarnpkg/yarn/pull/1816

[balthazar]

@thejameskyle @chrisirhc

After using the master version including #2384, I was still experiencing the issue could not resolve hostname, fatal: Could not read from remote repository.

What I had to do was to create a new resolver fitting my (awesome) company configuration and add it to the list of resolvers. You can see the change here. Although it should be quite simple to create a fork of yarn for this, I have a feeling it would interest more people, thus my investigation to give a way for users to provide custom resolvers based on their environment.

Since the resolver is retrieved here, and a TODO is already there acknowledging this should be refactored, maybe it would be a good time to do it.

I'm quite ensure of what would be best in your opinion, since I don't know where to put this kind of class configuration. Maybe a new file and a path reference in the project package.json or .yarn-metadata.json? I think the latter is not really used anymore, but I might be wrong.

If you give me a path to follow, I'm more than willing to create a PR 🚀
Thanks!

[BryanCrotaz]

It feels like this shouldn't be in the yarn repo, but should instead be a plugin registered with yarn somehow. Otherwise you'd need to keep updating your personal private yarn fork every time there's an improvement.

[balthazar]

That's exactly what I'm saying, glad we agree.

[chrisirhc]

Any thoughts on supporting custom resolvers / plugin architecture? Is there perhaps some .yarnrc type of file where users can configure custom resolvers for their environment?

Other alternatives:

• yarn can read the .gitconfig and create custom resolvers based on it.
• yarn can fallback to using the git client for resolution (with the raw url) when it doesn't know how to resolve a git url by itself. (Currently, it falls back to git client but uses its own parsed hostname, port numbers, username, instead of using values from gitconfig / git client)

[deecewan]

@chrisirhc why not just include the resolver for this particular case? it seems like it's pretty common. npm supports this format of package.

I do like your second option, tho.

[pleunv]

I'm not entirely sure what the outcome of these issues is, but I just noticed that the bitbucket shorthand ("myLib": "bitbucket:myOrg/myLib") is also not supported right now, while supported out of the box by npm. Any chance this could be added? We've been using this as a protocol-agnostic way to reference internal git repos, as we have both SSL & HTTPS usage internally and don't want people having to mess around with rewrites.

[vandamm]

Also, GitHub shorthand "user/repo" is not supported for private repositories, while works fine for public ones.

[balthazar]

Sorry for the close, I don't think this is resolved by my PR

[ramasilveyra]

https://github.com/yarnpkg/yarn/pull/2992 added support for private git repos when using shorted repo syntax (this landed on v0.24.0).

The generation for the first time of a yarn.lock file works well, but then with a yarn.lock file if you try rm -rf node_modules && yarn fails with this error:

error An unexpected error occurred: "Invalid protocol: git+ssh:".

That error comes from https://github.com/request/request/blob/b12a6245d9acdb1e13c6486d427801e123fdafae/request.js#L455 and I think (im not sure, still looking to the code trying to understand) it's because yarn tries to use the tarball fetcher instead of the git fetcher.

Update

The offline feature/fix added recently makes the git resolver defaults to the tarball to avoid http calls https://github.com/yarnpkg/yarn/pull/3000/files
Returning the git shrunk if is possible seems to fix the issue look the last commit at https://github.com/ramasilveyra/yarn/tree/fix-private-deps

[bestander]

@ramasilveyra, do you have a PR in mind that could fix it?

[ramasilveyra]

@bestander Hey I just created a pr with the fix https://github.com/yarnpkg/yarn/pull/3425, but I'm not sure if it's the best solution.

[chamini2]

This could be closed now, right?

[bilby91]

Last version fixes the problem.

[gsklee]

Just tried it and it doesn't work (github repo shorthand with private repo); it prompted me to enter my ssh key's passphrase but I cannot actually enter anything at all.

[gsklee]

Answering myself: https://github.com/yarnpkg/yarn/pull/2992#issuecomment-308024798

[onury]

This still doesn't work with yarn 1.3.2

Works —> npm i --save github:org/repo
Doesn't Work —> yarn add github:org/repo Got error: Permission denied (publickey)

macOS v10.13.1
Node v8.9.1
Yarn v1.3.2
npm v5.5.1

[jherdman]

This is definitely broken still, even with long form Git URLs.

[JoshuaCWebDeveloper]

I can confirm that the following _do not_ work on Yarn 1.3.2 but _do_ work on NPM:
yarn add github:org/repo
yarn add ssh://github.com/org/repo.git
yarn add ssh://[email protected]/org/repo.git
yarn add git+ssh://[email protected]/org/repo.git

However, using HTTPS _does_ work on yarn:
yarn add https://github.com/org/repo.git

Unfortunately, in my case, there are dependencies that use some of the above formats, so it looks like I cannot use yarn until this issue is fixed.

[Schlammsuhler]

if you added you public key to your github account this works:
yarn add ssh://[email protected]:yarnpkg/yarn.git