Yarn: `yarn install` ignores `files` from local dependencies

[phtrivier]

What is the current behavior?

It seems that yarn install ignores files from package.json of projects that are locally depended on.

If the current behavior is a bug, please provide the steps to reproduce.

Using two simple projects that depends on each other locally:

○ → tree
.
|____project1
| |____bad.js
| |____good.js
| |____package.json
|____project2
| |____package.json

project1 specifies which files should be copied by dependent projects:

// project1/package.json
{
    "name" : "project1",
    "version" : "0.0.1",
    "files" : ["good.js"]
}

project2 depends on project1 locally:

// project2/package.json
{
    "name" : "project2",
    "version" : "0.0.1",
    "dependencies" : {
        "project1" : "file:../project1"
    }
}

npm install in project2 only copies the good file:

 2017-03-02 12:43:41 ⌚  turus in ~/tmp/yarn-issue/project2
○ → npm install
[email protected] /home/phtrivier/tmp/yarn-issue/project2
└── [email protected] 

npm WARN [email protected] No description
npm WARN [email protected] No repository field.
npm WARN [email protected] No license field.

 2017-03-02 12:43:45 ⌚  turus in ~/tmp/yarn-issue/project2
○ → ll node_modules/project1/
total 32
drwxrwxr-x 2 phtrivier phtrivier 4096 mars   2 12:43 .
drwxrwxr-x 3 phtrivier phtrivier 4096 mars   2 12:43 ..
-rw-rw-r-- 1 phtrivier phtrivier   37 mars   2 12:37 good.js
-rw-rw-r-- 1 phtrivier phtrivier 1171 mars   2 12:43 package.json

However, yarn install copies everything, ignoring files:

  2017-03-02 12:43:48 ⌚  turus in ~/tmp/yarn-issue/project2
○ → rm -rf node_modules

 2017-03-02 12:44:55 ⌚  turus in ~/tmp/yarn-issue/project2
○ → yarn install
yarn install v0.21.3
warning [email protected]: No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 0.26s.

 2017-03-02 12:44:58 ⌚  turus in ~/tmp/yarn-issue/project2
○ → ll node_modules/project1/
total 44
drwxrwxr-x 2 phtrivier phtrivier 4096 mars   2 12:44 .
drwxrwxr-x 3 phtrivier phtrivier 4096 mars   2 12:44 ..
-rw-rw-r-- 1 phtrivier phtrivier   57 mars   2 12:37 bad.js
-rw-rw-r-- 1 phtrivier phtrivier   37 mars   2 12:37 good.js
-rw-rw-r-- 1 phtrivier phtrivier   80 mars   2 12:38 package.json

 2017-03-02 12:45:03 ⌚  turus in ~/tmp/yarn-issue/project2
○ → 

What is the expected behavior?

Same behavior as npm.

Please mention your node.js, yarn and operating system version.

○ → npm --version
3.10.10
○ → yarn --version
0.21.3
○ → uname -a
Linux turus 4.8.0-39-generic #42-Ubuntu SMP Mon Feb 20 11:47:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[rally25rs]

I suspect this may be a common error across other resolvers as well.

Normally when you npm publish a package, only the items listed in files are included in the .tgz, so for normal NPM packages, the cli tool (npm or yarn) doesn't have to really worry about it.

(you can verify this by setting up an offline cache directory and yarn add bootstrap, then compare the downloaded .tgz to their GitHub repo. Only the items from files are in the downloaded/cached .tgz, not every file in the Github repo)

For other resolvers, it's going to have to filter on its own. I did a quick search through the yarn source and didn't see anything that would be doing that check (though I could have missed it).

[SimenB]

@rally25rs yarn pack does this filtering to avoid bundling them before publish.

[kohlmannj]

Confirming @rally25rs's suspicion: this issue appears to also affect installation from GitHub using the git+https:// protocol. I've seen Yarn install a private GitHub dependency with its src _and_ build folders, despite its package.json:files array containing only the build folder.

I suppose publishing with yarn pack is the workaround, but that's not feasible for all projects. Ultimately it'd be better to honor the files property.

[foxymoron]

For private git repositories, if npm is able to filter packages by reading the files:[] property, shouldn't yarn be doing the same ?

[MonsieurBon]

We face exactly the same problem with a dependency we're pulling directly out of our private corporate git repository. Any idea when this will be fixed?

[chead2344]

Has there been any progress made on this?

[ZaLiTHkA]

just out of curiosity, is there any plan to change this behaviour?

I've just run into a situation where I need one specific file from private repo A as an import in private repo B, but the file I need is created as part of repo A's build step, so I don't need the other 5MB of stuff that it contains...

I've created a package.json in repo A with the relevant files key, but as other people in this thread have discovered, npm install limits the files that are retrieved, while yarn add does not. looking through the output from yarn add --help, it seems there are no command flags to control this, so is this the behaviour we're stuck with now?

[skidding]

Not sure if it's related, but I'm experiencing a situation where yarn ignores package.json:files from a workspace package.

You can see the src dir being included when running yarn pack (as well as when installing the dep from npm), even though it's not included in package.json:files.

cat package.json

{
  "name": "react-cosmos-test",
  "version": "4.0.0-beta.0",
  "description": "APIs for testing Cosmos fixtures in headless environments",
  "repository": "https://github.com/react-cosmos/react-cosmos/tree/master/packages/react-cosmos-test",
  "license": "MIT",
+  "files": [
+    "lib",
+    "enzyme.js",
+    "generic.js"
+  ],
  "dependencies": {
    "react-cosmos-config": "^4.0.0-beta.0",
    "react-cosmos-loader": "^4.0.0-beta.0",
    "react-cosmos-shared": "^4.0.0-beta.0",
    "traverse": "^0.6.6"
  },
  "xo": false
}

yarn pack
tar -tf react-cosmos-test-v4.0.0-beta.0.tgz

package
package/enzyme.js
package/generic.js
package/lib
package/package.json
+package/src
package/lib/enzyme.js
package/lib/generic.js
+package/src/__tests__
+package/src/enzyme.js
+package/src/generic.js
+package/src/__tests__/enzyme.js
+package/src/__tests__/generic.js

I appreciate any help with this one as it's annoying for react-cosmos users to download unnecessary source files, which among other things can cause Flow errors in the dependent project. Thanks!

[zastavnitskiy]

Hi everyone, I'm going to fix it.

After the initial research I have two questions.

I plan to modify https://github.com/yarnpkg/yarn/blob/master/src/fetchers/copy-fetcher.js#L10, and then I have several options:

Option one:

• read files field from package.json
• delete all the files that do not match regexp (using the logic from https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/pack.js#L75)

Option two:

• run pack command for this directory

Which one is better?

And the second question – does doing it in the fetcher make sense?

@BYK @kittens I will appreciate your feedback on how to proceed here.

[tommedema]

This problem occurs to me when depending on local packages using Yarn workspaces, too.

Could someone please confirm if this is the same issue? Or should I create a new issue for yarn workspaces specifically?

E.g.

workspace/packages/a has files: ['good.js]
workspace/packages/b depends on package a through yarn workspaces

yarn install --production results in package b's node_modules/a having the full contents of package a (even when package b has been set as nohoist), while it should only include good.js

If this is a separate issue please let me know so I can create it.

[Grey2k]

As I can see the problem in that yarn using git archive
https://github.com/yarnpkg/yarn/blob/master/src/util/git.js#L225

command for fetch packages and its not included '.git' folder and no ignore checks at all here
https://github.com/yarnpkg/yarn/blob/master/src/fetchers/git-fetcher.js

If I added files: ['.git'] (for fetching with submodules also) - its will be another behavior than npm :(
very sad

Have the yarn team any plans for this?

[Meleipe]

I would love to see this issue solved. if npm had this issue, it's solved already... and this one is marked as high priority since 2017 and nothing untill now... This is blocking my development :/

[mayteio]

+1, I am building a component library with workspaces and lerna. Imports only work when I import @scope/package/**dist**/Component. This obviously doesn't look good or translate when when it's build, as the published package will be available at @scope/package/Component.

[yinonov]

strange that this is still an issue, it's like the first issues I ran in straight when using lerna and yarn, got me all confused

[sveyret]

Actually today (version 1.19.1), at least in my project, even the workaround yarn pack does not work and includes all files.

[rcmorano]

JFTR, I just came across with this after upgrading yarn from 1.12.3 to 1.15. Surprisingly, it worked with old version w/o any workaround.

[mtford90]

This is exceptionally annoying as far as using certain IDEs is concerned e.g. auto imports can be barfed when using typescript

[kitfit-dave]

Any word on this? I am running into this when developing a package the specifies the babel transpiled dist directory in files. So when someone who gets it from the repository uses it they will do import something from 'mypackage/somefolder'. But as the yarn workspaces link is to the root of the package not the dist folder, I have to do import something from 'mypackage/dist/somefolder'

[mattvb91]

experiencing the same issue