yarn install fails for unpublished and linked-in dependency

[viliusl]

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
Given I have a module listed in dependencies/devDependencies that is not yet published and is linked-in to my module, yarn install fails.

If the current behavior is a bug, please provide the steps to reproduce.
https://github.com/viliusl/yarn-link-issue

What is the expected behavior?
For yarn to not fail during install when yet-unpublished dependency is linked-in.

Please mention your node.js, yarn and operating system version.

• node: v6.9.4
• yarn: 0.19.1
• os: macOS 10.12

[alexdohmen]

+1 same here
error: error Couldn't find package "the-next-big-module" on the "npm" registry.

[npiguet]

I have to add my voice to this one. Tested with yarn 0.18.1 and node 5.10.1.

The output from yarn install looks like:

Running 'yarn install' in /home/npiguet/workspace/epb/epb-web-components/epb-charts
yarn install v0.18.1
info No lockfile found.
warning [email protected]: No license field
[1/4] Resolving packages...
error An unexpected error occurred: "https://private.reposiroty/repository/npm/private-module: Not found".
info If you think this is a bug, please open a bug report with the information provided in "/path/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

yarn-error.log contains the following info:

Arguments: 
  /projectpath/node/node /projectpath/node/yarn/dist/bin/yarn.js install

PATH: 
  /projectpath/node:/projectpath/node/yarn/dist/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:

Yarn version: 
  0.18.1

Node version: 
  5.10.1

Platform: 
  linux x64

npm manifest: 
  {
    "name": "private-module",
    "version": "1.0.0",
    "description": "Description",
    "main": "./dist/private-module.js",
    "typings": "./dist/index",
    "globalDependencies": {
      "react": "0.14.8",
      "react-dom": "0.14.8"
    },
    "devDependencies": {
      "gulp": "3.9.1"
    },
    "scripts": {
      "typings-install": "typings install"
    },
    "dependencies": {
      "ts-loader" : "1.3.3",
      "process-nextick-args": "1.0.7",
      "private-module": "1.0.0"

    }
  }

yarn manifest: 
  No manifest

Lockfile: 
  No lockfile

Trace: 
  Error: https://private-registry/repository/npm/private-module: Not found
      at Request.params.callback [as _callback] (/projectpath/node/yarn/dist/lib/util/request-manager.js:309:18)
      at Request.self.callback (/projectpath/node/yarn/dist/node_modules/request/request.js:186:22)
      at emitTwo (events.js:100:13)
      at Request.emit (events.js:185:7)
      at Request.<anonymous> (/projectpath/node/yarn/dist/node_modules/request/request.js:1081:10)
      at emitOne (events.js:90:13)
      at Request.emit (events.js:182:7)
      at IncomingMessage.<anonymous> (/projectpath/node/yarn/dist/node_modules/request/request.js:1001:12)
      at IncomingMessage.g (events.js:273:16)
      at emitNone (events.js:85:20)

[MikeTaylor]

I have this problem, too, and it's a killer. I means that before I can start development work on a new package linked in via yarn link, I first have to release a null version -- which will then be ignored, but whose presence will allow the link to work. I am keen to see this fixed.

[bestander]

As reported in #1214 (comment)

I'm getting this issue on version 0.24.5.
I have some packages that i developed and they are not published on any server, they are just linked.
My projects uses this linked packages and some published packages like express or bluebird.
when I try to run "yarn" to install all or a specific package it returns the error:

error Couldn't find package "package-name" on the "npm" registry
steps to reproduce:

cd path/to/a-dev-package
yarn link
cd path/to/a/project
yarn link a-dev-package
yarn add bluebird
it returns :
error Couldn't find package "a-dev-package" on the "npm" registry

[dandv]

Same problem with 0.27.5. The only workaround was to not list my linked package among the dependencies in package.json, but then I get a linter warning on the require/import line that the dependent module is not listed in the dependencies.

I'd rather not look into para-package-management workarounds like yalc.

[arcanis]

I'm not sure this is a bug - even if you're using yarn link to install the dependency, we still need to resolve it according to its range in order to put it inside the lockfile 🤔

[arcanis]

I'm going to remove this one from the 1.0 milestone - it's not a bug strictly speaking, so we will want to think about this more in depth before changing the behavior. I agree it could be clearer.

[nadav-dav]

@aracarie
It is possible to enforce a linked-module:"*" version on unpublish+linked dependencies.

It makes a lot of sense to connect the latest version of a local linked module, and removes the need to lock a specific version in the yarn.lock file.

[bestander]

From a user point of view this still looks like a high(er) priority issue because it prevents people from using yarn link command.
We should find some sort of compromise here.

[bestander]

After thinking about it a bit more I agree with @arcanis.
This happens for an edge case situation and can be mitigated with using 'link:' protocol.

Are we missing anything?

[nadav-dav]

@bestander Are you referring to yarn link? Because that is not really working..

[bestander]

@nadav-dav, you can actually do this in package.json

"dependencies": {
  "my-local-dep": "link:../../local/path"
}

It will create a symlink automatically.

[nadav-dav]

Cool:) I didn't know that!

Well it is a step in the right direction, but still means that deploying / publishing the module will not be possible.. Actually it only helps if you experiment for local development.

[bestander]

Another alternative is to use workspaces https://github.com/yarnpkg/website/pull/580 if you can structure them in one folder tree.
If anyone can send a PR with a fix to this task that would be great

[arcanis]

Well it is a step in the right direction, but still means that deploying / publishing the module will not be possible.. Actually it only helps if you experiment for local development.

If you're not in the case of the local development, it means your package has to be available by some way, right? Maybe not even on the npm registry, but at least as a github repository or something. In such a case, you can use yarn link (even if I don't completely see the point for production environments).

[apranovich]

@bestander thanks so much for posting the example

"dependencies": {
  "my-local-dep": "link:../../local/path"
}

it actually solved my issue.

I tried to use unpublished local package which in its turn uses few unpublished local deps as well.
So I have the following picture:

// Main proj
// package.json
"dependencies": {
   "local-dep": "../../local/path"
}

// local-dep proj (this is a part of monorepo, btw, with workspaces turned on)
// local-dep and dep1_2 are also packages of the monorepo
// package.json
"name": "local-dep",
"version": "1.0.0",
"dependencies": {
   "dep1": "../local_path/dep1",
   "dep2": "../local_path/dep2"
}

When I run yarn inside Main repo it loaded local-dep but failed with the error that yarn couldn't find package dep1 and dep2 on npm registry.

But as soon as I added link prefix "dep1": "link:../local_path/dep1" to the local path it got back to work.

yarn version is 0.27.5

[BYK]

For workspaces to work, you need to define which folders should be treated as workspaces for yarn to automatically link them: https://yarnpkg.com/en/docs/workspaces/

Also you mentioned using Yarn 0.27.5 which is old. There were many improvements over that for workspaces and other things in 1.0.0. The latest is 1.0.2 and I strongly suggest trying that.

[BYK]

I'd like to close this issue since this can be addressed with multiple ways now:

• Using the link: protocol as @bestander suggested above
• Using workspaces: https://yarnpkg.com/en/docs/workspaces/
• Using selective version resolutions: https://yarnpkg.com/en/docs/selective-version-resolutions/

Objections?

[MikeTaylor]

Sorry, but I do object really. It's great that there are multiple workarounds, but that's not the same thing as the bug being _fixed_. Isn't this something that should Just Work?

[BYK]

@MikeTaylor

Sorry, but I do object really. It's great that there are multiple workarounds, but that's not the same thing as the bug being fixed. Isn't this something that should Just Work?

"Just Work" means there is a single, agreed-upon way of doing this and currently that sounds like one of the three options above depending on project size, structure, and some preference. Why do you think those solutions are not enough and should be complemented with changes in one of the core pieces of Yarn?

Is this an npm-compatibility thing that we aren't aware of?

[snajjar]

Sorry, I beg to differ too.

I'm working on a "core library" and my customers owns separate repositories in which i develop plugins, using links to the core library.
None of the above solutions are working in my use case:

• local path in package.json aren't ok to use if want/need to publish the packages (I do)
• workspace is a great feature but doesn't work either with separate file hierarchies
• selective dependency resolution (if I understand it correctly) is basically a dependency bypass and doesn't help in my case either.

However, I understand the issue: the yarn.lock file should be kept up to date.

Couldn't we just have a simple behavior like this:

• Check for the dependency existence in registry (say the dependency is name "my-depency")
• If not present (or not with a semver-compatible version), try require.resolve('my-dependency/package.json') and read the version from it

I may be missing something here, but for now I don't see any reason why it shouldn't work like this

[snajjar]

Any news on this @BYK ?

[BYK]

@snajjar

workspace is a great feature but doesn't work either with separate file hierarchies

Can you elaborate on this? Sounds like your use case should be handled by workspaces.

[snajjar]

Hello, sorry for the delay @BYK.

I have a use case (which i believe is not uncommon):

• I develop some core library (which is aimed to be opensource once stabilised).
• I sale some work to customers that make use of this core library

Here's something very similar to my work directory structure:

work/
    core-lib/
        package1/
        package2/
        package3/
            package4/ 
    customer1/  # using core-lib
        package5/
    customer2/ # using core-lib
        package6/
        package7/
            package8/
                package9/

My customers want to have their intellectual property handled on separate githubs and npm published separately, but referencing the right version of core-lib.
I need to be able to work on core-lib (adding feature and bugfixing) and appreciate the changes inside customer1 or customer2 folder.

I can work with workspaces inside core-lib, customer1 and customer2 packages, but I still need to have a "link" working between my customer folders and core-lib, which i was previously handling simply with npm link.

I also do not wish to have a "common" workspace in the whole work directory, since i need to be absolutely sure that dependencies are settled correctly inside each customer packages when I publish it.

Is there something more I could do with workspaces that would avoid the need of a global link to core-lib?

[DaAitch]

Hi, in my opinion it should be possible to develop multiple modules at the same time, each of them having dependencies to published modules or to local linked modules (not published). With "it should be possible" I mean that we need a solid workflow that we don't need to change package.json and replace links with published packages and so on, which is quite ugly I think.

I would expect:

_Module A_

• yarn link: I can add module-a to some local registry

_Module B_

• yarn add --dev [email protected]: I add a dev dependency. Before looking at remote repositories, yarn inspects local registry going through each linked module, checking version and name and if it matches, then it is resolved.

That means:

• for me yarn link <package> does not makes sense. It's just a hack, that "something" works, but in the end we want a correct package.json and no link hacks.
• before publishing a module, it would be possible, that yarn checks, that every dependency is also published with that version, so that at first I have to publish Module A.

[dualbox]

Interesting idea @DaAitch
@BYK How could we advance on this discussion? Is this something that is currently discussed in the dev team?

[roytz]

That's not only a link issue. As written in lots of comments above - it happens with regular packages (like winston, eslint etc.).
I user workspaces and when I execute yarn I always get the Couldn't find package "[PACKAGE_NAME]" on the "npm" registry.
When I delete that package, I get the same error for the next listed package on the package.json file.
Still happening using latest yarn version (1.7.0) and laters node version (10.5.0).
@dualbox @BYK @snajjar , any idea to when it'll be fixed?
@viliusl did you find any solution/workaround?

[jwater7]

I encountered this today. Only way around it is to upload an initial module to the NPM repo and then yarn would continue (using my linked version instead).

[rodrigoyoshida]

The "link:" workaround saved me after hours trying to figure out what was happening. I am impressed that this bug is known for almost a year and a half and it is still there.

[graup]

@snajjar suggested a fix more than 1.5 years ago and nothing has happened here. To me this makes sense and would solve the issue. If we get consensus on this I'm sure we can find someone to implement it (I'll gladly give it a shot).

Couldn't we just have a simple behavior like this:
Check for the dependency existence in registry (say the dependency is name "my-depency")
If not present (or not with a semver-compatible version), try require.resolve('my-dependency/package.json') and read the version from it

In the meantime, the most simple workaround seems to be to just publish an empty package with the same name.

[BYK]

I am going to close this issue in favor of existing solutions like workspaces and yarn link as the proposed solutions are not general enough. Even more, Yarn v2 will be supporting nested workspaces which should be the ultimate solution here. You can learn more about v2 here:

• https://2019.jsconf.eu/mael-nison/yarn-2019-and-beyond.html
• https://yarnpkg.github.io/berry/features/workspaces

[BYK]

Oh, and selective resolutions may also help: https://yarnpkg.com/en/docs/selective-version-resolutions

[MikeTaylor]

Most fundamentally, the thing that worked for me was: never use yarn link, use workspaces instead.

[arcanis]

There was one case where yarn link was useful: when you wanted to use a package stored in another project (although in this case I think you could create a symlink into your project that would point to the second and it might have worked).

This soon won't be the case anymore, as the v2 introduces the portal: protocol which allows you to add a dependency to a different location on the filesystem tree (like link:) while also following the transitive dependencies (unlike link:). And peer dependencies will work just fine (something that just wasn't possible previously).

[graup]

Thanks for the replies. Sounds like good solutions are being worked on! Nested workspaces would indeed solve my issue. Looking forward to that!

[brainkim]

Hello from 2020. Ran into this problem today and cannot understand why this is not considered an issue.

[remorses]

Still an issue, yarn install should not try to install linked dependencies

[roncodes]

Same issue as well, also the link:../../dir workaround doesn't work for my specific case