Do you want to request a feature or report a bug?
Bug
What is the current behavior?
The more package through yarn are installed, the longer the subsequent installing
If the current behavior is a bug, please provide the steps to reproduce.
Create project and add 20 packages step by step.
What is the expected behavior?
Faster installing.
Please mention your node.js, yarn and operating system version.
Node v9.3.0, Manjaro, yarn 1.3.2
dimensi
All 22 comments
137 seconds for one package with 2.4k size.
dimensi
on 26 Dec 2017
Please run with --verbose and pipe the output to a file. The verbose log contains the seconds durations for the steps.
For example:
...
verbose 0.319 Checking for configuration file "/Users/.yarnrc".
verbose 0.321 current time: 2017-12-28T03:39:14.759Z
[1/5] 馃攳 Validating package.json...
[2/5] 馃攳 Resolving packages...
[3/5] 馃殮 Fetching packages...
verbose 0.974 Copying "/Users/jvalore/Projects/yarn/scripts/eslint-rules/index.js" to "/Users/jvalore/Library/Caches/Yarn/v1/npm-eslint-plugin-yarn-internal-0.0.0-6961c602-d274-4709-bd19-7ad81a04ede5-1514432355041/index.js".
verbose 0.974 Copying "/Users/jvalore/Projects/yarn/scripts/eslint-rules/package.json" to "/Users/jvalore/Library/Caches/Yarn/v1/npm-eslint-plugin-yarn-internal-0.0.0-6961c602-d274-4709-bd19-7ad81a04ede5-1514432355041/package.json".
...
the 0.319, 0.321, 0.974 are the seconds since the start (specifically it is node.js process.uptime() so you can use that to maybe narrow down what is taking so much time.
rally25rs
on 28 Dec 2017
Hm, i get again slow intall. What should I look at? @rally25rs
logs.txt
dimensi
on 30 Dec 2017
Looks like it's taking about 50 seconds to copy files from the cache to node_modules/, even though most of those look like it skips the copy since the file is already there.
If you delete node_modules/ directory, does it become fast again, then slow the 2nd time?
rally25rs
on 2 Jan 2018
I remove node_modules, then try install new package, it's was long. After i remove package and install it again and it was fast.
dimensi
on 2 Jan 2018
@rally25rs it's really sad speed, I ask many people and i not one that monster low speed.
dimensi
on 10 Jan 2018
Hi! I'm having basically the same problem. Fetching isn't that slow though (I mean it does take some time but nothing abnormal). The linking, though, has gotten very very slow. For the same packages, the total time went from 53s to 193s. I can't tell exactly when it happened but it was fairly recent (maybe has something with upgrading to High Sierra?).
My setup:
Macbook Pro 2017
Node: 8.9.2
Yarn: 1.3.2
As a benchmark, npm takes 66s to install the same packages from scratch.
I tried piping the verbose output to a file and saw nothing in particular that would turn the blame to a single package. It's just slow from A to Z in the linking step.
etiennetalbot
on 11 Jan 2018
Welp... I just realized that the "slowness" was because of my antivirus (Avira with Real-Time Protection toggled on). So... yeah, my bad. All is fine!
etiennetalbot
on 16 Jan 2018
@etiennetalbot I'm also seeing a massive slow down when linking packages. There has been consistent feedback from colleagues experiencing the same issue.
@rally25rs is this worth taking out of triage? This is causing a lot of pain at the minute.
afenton90
on 19 Apr 2018
@afenton90 Sorry for the inconvenience. We need the community to assist in issues like these that are tied to I/O on individual machines.
I use yarn daily on a project with >1000 packages and haven't experienced this slowdown, so I am unable to debug into the issue.
I know that Docker has some slowness issues (filesystem i/o between container and host is slow), and I've heard some say that it's antivirus scanning the tens of thousands of files being touched.
Make sure you are on Node 8.5 or newer, which added fs.copyFile() which is much faster than the old way of copying files in node (in other words, yarn is much faster on node >8.5).
Running with --verbose will print log lines with timestamps for every file copy. However in the couple logs that I've seen, it seems like it just takes a long time to copy or stat a file, which is probably going to be a filesystem i/o issue.
You can debug yarn and take performance snapshots in chrome devtools by cloning the yarn source and running it with node --inspect-brk yarn/bin/yarn.jswhich might lead to something.
If you have any packages with install scripts, those can certainly take time and would be outside of yarn's control, so I would also run with --ignore-scripts for the purpose of debugging the slowness.
rally25rs
on 19 Apr 2018
@rally25rs I ran a test on my machine and several Ubuntu servers and found that the copying from the cache is taking all the time.
- Directories are created for
node_moduleswithin 3secs. - Copying from local cache 73secs.
Has there been any changes to cache creation/copying recently that may cause this decrease in performance?
Info
Machine: MacBook Pro (Mid 2015)
Processor: 2.2 GHz Intel Core i7
Memory: 16 GB 1600 MHz DDR3
Node: v8.10.0
Yarn: 1.6.0
afenton90
on 23 Apr 2018
@rally25rs @dimensi the issue for me was the version of node I was using an update to the latest LTS 8.11.1 solved the issue for me. No more slow copying.
afenton90
on 23 Apr 2018
@afenton90 If you moved from an older <8.5 version of node, then yes that would make a huge difference. Node 8.5 introduced fs.copyFile() which runs a native filesystem copy. Yarn tries to detect that function and uses it when it is available.
rally25rs
on 23 Apr 2018
I use 9.3 node then i post this issue...
dimensi
on 26 Jul 2018
I run into a similar issue with macOS 10.14, yarn 1.10.1 and AVG Antivirus 18.6. Linking dependencies take forever... When I disable the antivirus, everything works fine. But that is not safe. Any way to work around this?
QuentinRoy
on 5 Oct 2018
@QuentinRoy That is an AVG issue then.
Yarn has to create thousands of files to build the node_modules directory correctly.
benpoulson
on 2 Nov 2018
@benpoulson I switched to Sophos, same issue. It seems yarn is intrinsically incompatible with anti-viruses.
QuentinRoy
on 17 Nov 2018
@QuentinRoy @benpoulson It's not that yarn is intrinsically "incompatible" with antivirus software. It's just the nature of antivirus software. yarn install will likely perform 10's of 1000's of I/O operations. Any reasonable antivirus software is likely going to inspect each and every one of those. It's the nature of how they work.
Similarly, if you are using Docker to run yarn in any environment other than linux, it also adds a significant amount of overhead to I/O operations in the name of file consistency between the host machine and the container. Take a look at https://docs.docker.com/docker-for-mac/osxfs-caching/ for more info. Unfortunately, in my own testing even using the fastest 'delegated' setting, it's still not nearly as fast as running yarn install directly on the host system. For my current project it takes ~3x longer in container than directly on host.
jamesdh
on 28 Nov 2018
@jamesdh this may be true but idk why, npm is running lightyears faster for me (and my fellow employees) with Sophos, and the slowness is inconsistent between yarn versions in our experience.
I'm not going to delve too deep into providing logs because our company decided to migrate to npm since we have to use an antivirus.
Also, these issues have been going on for a pretty long time now and there is an alternative that does run a lot faster with this antivirus situation although they are supposedly subjected to do similar operations to yours (as far as I understand).
I hope you solve it, but someone's going to really want to make it work with an antivirus and put in work to figure out what's making yarn slower than npm in that scenario.
motiazu
on 10 Dec 2018
It's strange. I have two projects using yarn. One is fast, the another is too slow. Take forever to link dependencies.
revskill10
on 31 Mar 2019
I have same issue!
yarn install
yarn install v1.16.0
[1/5] Validating package.json...
[2/5] Resolving packages...
[3/5] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[4/5] Linking dependencies...
[###############################################---------------------------------------------------------------------------------------------] 22778/67399
Done in 179.59s.
MacOS / Docker
reduardo7
on 7 Jun 2019
I used to have the same issue. It used to take 30+ Minutes to just install one package. I had my projects on an external drive and network drive and I tried adding scan virus exception path, opening firewall etc. None enhanced the speed. However, as soon as I git clone to, let say C:/New Folder/my repo. It took 1.5 seconds. I hope this resolves the issue for some developers.
micsel
on 31 Oct 2019
Related issues
ocolot
路
3Comments
mnpenner
路
3Comments
davidmaxwaterman
路
3Comments
catkins
路
3Comments
danez
路
3Comments
Most helpful comment
@jamesdh this may be true but idk why, npm is running lightyears faster for me (and my fellow employees) with Sophos, and the slowness is inconsistent between yarn versions in our experience.
I'm not going to delve too deep into providing logs because our company decided to migrate to npm since we have to use an antivirus.
Also, these issues have been going on for a pretty long time now and there is an alternative that does run a lot faster with this antivirus situation although they are supposedly subjected to do similar operations to yours (as far as I understand).
I hope you solve it, but someone's going to really want to make it work with an antivirus and put in work to figure out what's making yarn slower than npm in that scenario.