Sp-dev-docs: Document primary dependencies

[sympmarc]

There's been a lot of chatter about the size of the node_modules dependencies required simply to create a Hello World example. Some have pegged it at ~100k files and .5Gb.

One thing that would probably help a lot with skepticism is for you to outline what your main dependencies are (not all of their dependencies), why you chose them, and what they do.

As Chris Johnson said in one thread on FB, "Personally I don't make tools or platform choices based on the number of files it takes to create a solution with it." He also pointed out that few people questioned what tooling was loaded with a .NET-based Visual Studio project. But that "black box" mentality doesn't work as well here in the open source world, nor should it.

Also, many of the dependencies can probably be loaded globally rather than per solution. Do you have a plan for that which you could add to the dependency discussion?

@chakkaradeep : Here's a "continue in whatever official feedback mechanism you guys post this in"...

[johnnliu]

I actually see this discussion in the open source world a lot recently. And I think this response to SPFx is just us not used to the number of dependencies that NPM does, naturally.

The main reason for this dependency chain is that people reuse modules - a lot - and that lets them step on shoulders of others to build bigger modules. There is no one person crazy enough to step in and say, hey lets combine everything into a common framework (like Microsoft did with .NET) and then make sure it all work together seamlessly.

Each open source author can get burnt out, or a project can fall out of active maintenance, so project dependencies are kept that way without been tweaked over time, and that contributes to the overall 'bloat'. NPM modules may have dependency on different versions so there can be dupes with different versions too.

Regardless, the modules are split between build modules and dist modules. Modules that aren't used on the client side aren't deployed. WebPack modular bundling also ensures only used modules are actually packaged and deployed. Unused components aren't deployed.

This recent project by @nolanlawson can be useful. I've wanted to try it on something and SPFx seems like a perfect test case :-D

• https://twitter.com/nolanlawson/status/766275330403205122
• https://www.npmjs.com/package/slow-deps

c:\Projects\dep-test>slow-deps
Analyzing 9 dependencies...
[====================] 100% 0.0s
----------------------------------------------------------------
| Dependency                      | Time     | Size   | # Deps |
----------------------------------------------------------------
| @microsoft/sp-build-web         | 3m 58s   | 173 MB | 798    |
| @microsoft/sp-webpart-workbench | 2m 45.5s | 158 MB | 678    |
| @microsoft/sp-client-preview    | 31.4s    | 54 MB  | 130    |
| @microsoft/sp-client-base       | 30s      | 27 MB  | 123    |
| @microsoft/sp-module-interfaces | 15.2s    | 6.5 MB | 70     |
| gulp                            | 13.9s    | 3.4 MB | 196    |
| office-ui-fabric-react          | 7.2s     | 11 MB  | 2      |
| react                           | 6.9s     | 9.5 MB | 41     |
| react-dom                       | 1.2s     | 11 KB  | 0      |
----------------------------------------------------------------
Total time (non-deduped): 8m 29.5s
Total size (non-deduped): 442 MB

[sympmarc]

Good fodder for what I think needs to be in the docs, @johnnliu. Even explaining the difference between build and dist modules would be a great thing for the team to do.

[johnnliu]

798 dependencies seems a bit high...

build and dist modules are just what I call them. In hindsight they probably aren't the correct terms. In package.json you see both dependencies (for distribute) and devDependencies (for build and never deployed).

[sympmarc]

I noticed issue #59 sort of gets at some of this. Some rationalization of the existing dependencies would certainly be a good idea. But I also think some docs or even just opinion pieces from the team building #SPFx would be a REALLY good idea.

[patmill]

Are you interested in the runtime dependencies, or the build system dependencies? The vast majority of the files are purely for the build tool chain. .

[chakkaradeep]

And also note that we are trying our best to bring down the SPFx locally for development. You could easily remove a lot of dependencies if the only way to test web parts is to deploy to SharePoint. As @patmill this corresponds to a build tool chain.

[sympmarc]

@patmill I think it would be useful for you guys to spell this all out. Not for me, but for the many people who are going to look at the huge number of files in the Hello World Web Part and wonder what the heck is going on.

@chakkaradeep "You could easily remove a lot of dependencies" isn't going to be an option for people as they get started here because working in nodejs land will be new for them.

[chakkaradeep]

@sympmarc I think you misread :) - I was saying that one of the reasons you are seeing lot of build dependencies is because it is actually bringing down SharePoint Framework for local development. Not suggesting developers remove dependencies :) - Again, these are not runtime, build time.

[sympmarc]

Ah. Well, are you planning an option to not do that? Right now , it's "yo @microsoft/sharepoint", and I don't think that option is there.

Like AC keeps saying, I think you guys need to get out in front of these questions BEFORE people ask them, not wonder why they are asking later. Explain to the market why what you are doing is good, and how it works.

[chakkaradeep]

If we remove SharePoint Framework, then you don't get workbench and developers basically have to go back to testing everything in SharePoint.

What I am trying to understand is that and please correct me if I am wrong, why would the dependency chain prove to be a problem when development and tooling is supported and is abiding by the web stack? Like, for example, if the total files were just 100 - what does it really say/mean?

On the other hand, if there are performance, de-dup issues, we will sure handle those.

[sympmarc]

I just think you should explain what dependencies you are loading and why. You guys are too close to all of this. Think of the people coming to this method of development and starting from square one. They have no idea what you've done here or why it works. Throw them some bones or you are going to lose them.

The fact that you end up with a folder structure with ~100k files at .5Gb is going to look ludicrous to people who haven't done this type of dev. (And I'm a bit flabbergasted myself!) Don't you want them to feel comfortable with your how and why? This isn't black box mumbo jumbo anymore. "Trust us" has never worked well for you, anyway.

You're not getting out in front with a good story. There's been very little story without some of us blogging about it. Notice the patterns in the comments and respond to them early, not later or never.

[andrewconnell]

@chakkaradeep If a big part of the dependencies being pulled down are the SharePoint Framework & workbench that support local development, is there any reason why they have to be in every project?

While you _can_ include Mocha, Jasmine, Webpack, TypeScript, Gulp, etc in every project's devDependencies, couldn't you provide instructions to install them globally and then you only need one copy of them on your machine to do local dev?

Sure, a lot of time Node dev's include these in our devDependencies so you can just do git clone BLA && npm i && npm start, but you've already got some global requirements anyway (Yeoman & the generator). This would let developers elect to include them locally or globally.

[chakkaradeep]

@andrewconnell Good points. I think it has to do how systemjs, webpack and other key runtime libraries - that is easier if installed locally in your project than say installing globally which could result in unusual behaviors. This is how NPM works today as well. This is the same problem with gulp - doesn't matter if you have gulp installed globally, you need to still install in the project you are working on.

That said, we are still looking into this. So, there is time :)

[andrewconnell]

@chakkaradeep Eh... not so sure I agree with that. It's not uncommon to see folks install Express once globally and then just assume it's on the machine when it's built. Same goes for TypeScript, TSLint, etc.

Some tools don't behave well when installed globally, like Gulp, but if _you own_ (where you = MSFT) the SPFx & workbench, shouldn't you be able to address this?

[sympmarc]

Even if you address the technical parts of these questions, @chakkaradeep, you need to be able to explain how it all works in a way people can get their heads around - and appreciate. So far I don't see that story coming. Since people are reacting to the dependency chain, why not do a little work and explain that part for them?

[chakkaradeep]

I am currently trying out something - installing the build tools globally and check to see if that works. There are ways to optimize but we also need to better job in documentation.

[chakkaradeep]

Here is official blog post entry from NPM on this issue as well. Its old but I dont see things have changed:
npm 1.0: Global vs Local installation

[chakkaradeep]

@andrewconnell While SPFx packages are indeed MSFT, we still need to play in the rules of NPM. Installing project dependencies and devDependencies globally has lot of repercussions across many different projects. Not to mention, each project's intention could be different to other projects and installing/managing dependencies globally will lead to a lot of mayhem.

Installing node, npm globally makes sense as they are required to basically run/start one or more projects. And npm's own require() function doesn't account to searching global npm folder. At this point you are crossing npm's territory and I would be cautious not to change that behavior.

[mike-morr]

@andrewconnell Installing node packages globally is generally considered a bad practice. It causes people to start logging issues with "XXXX is broken" Followed by "Oh my bad, hadn't updated my global install of XXXX in a while". I think the reason this pattern is so prevalent is because it is easier in quickstart scenarios to have the user install something globally, then explain the pathing to node_modules/.bin. I have seen a few training videos and conference sessions warn about it.

[andrewconnell]

@chakkaradeep & @mike-morrison I get both of those points and don't disagree with either. I'm simply bringing up that since a huge part of the dependencies is the plumbing that's required to simply test projects locally is the SPFx built by MSFT, there's an op to avoid that.

Seems the SPFx part of the project is what's so big... I just poked around a few of the Node projects I'm working on that include Express and a bunch of other popular tools in my devDependencies (webpack, gulp with 15+ plugins, typescript, tslint, mocha, chai, sinon, conventional-changelog, etc) and that's only creating a node_modules folder of 90-120MB.

Just trying to help address some people's concerns with the big footprint. I get how Node/NPM works... I get that the vast majority of the people playing with the dev preview are noobs to Node.js... but I also get you need to win these people over so just trying to help :)

[SPDoctor]

If you are seeing folder sizes of 500MB + and deeply nested folders, please check your version of npm - should be something like 3.10.7. If npm -v reports 2.something you need to upgrade: npm -g install npm@next. Node modules should all be listed under node_modules folder (all 800+ of 'em, and not nested in sub-folders called node_modules - that's the v2 npm behaviour).

While I think the number of dependencies is an issue and needs looking at, I suspect a lot of pain is because people are installing using npm v2.

[andrewconnell]

Totally agree with @SPDoctor - VS 2015 installed its own copies of Node & TypeScript within its install path and this causes havoc as it also includes it in the PATH envvar... this is a nightmare to figure out (not a Node thing... it's what VS installer does).

If you have huge nested paths or file path length issues, it's not an NPM3 thing... NPM3 removed the nested folders & put them all at one level.

The only file path length issues folks should run into are the same that you'd run into with ASP.NET projects & if you use the stupid default VS uses, burying projects in the MyDocs folder.

[andrewconnell]

I don't know how people are coming up with some of their numbers, but on MacOS with Node v5.11.1 & NPM v3.10.3, my node_modules folder is 275MB & 32,893 files.

[SPDoctor]

[nyn3x]

Just like @SPDoctor my helloworld-webpart counts about 220 megs

node -v && npm -v
v6.3.1
3.10.6

But I see the confusion about the huge amount of packages. Maybe adding some more choices to the generator could help, like selecting the testing-framework (mocha vs jasmine vs none) or maybe even providing the option to not install the local workbench.

This would help to create "lite" projects.

[patmill]

BTW - turns out that if you have npm 2._, you get 500 megs of node_modules, and with npm 3_ you get 250 megs. There's a whole discussion going on in https://github.com/SharePoint/sp-dev-docs/issues/59 about how to get lower.

[GreyMana-git]

As you wrote, switching to NPM3 reduces the size from 590MB to 210MB, so that really helps. The versioning is a little bit confusing, but if I get it right the latest LTS of Node is 4.5.0 which includes NPM 2.0, while the current version 6.5.0 uses NPM 3.10.3. I missed the "npm -g install npm@next" in the Setup your Machine documentation which left me with the 2.0 version.

Regarding the size, I'm developing sharepoint in multiple VMs (depending on customer) with multiple project per customer. They all take space on my SSD and I am already juggling the VMs to the normal HDD when I don't use them. So increasing the project size by 0,5 GB per webpart set isn't something I really like.

[waldekmastykarz]

@GreyMana-git Out of curiosity, given that SharePoint Framework doesn't require SharePoint to be running on the development machine, would you consider developing on your host instead of in a VM or is there anything else that requires you to be using a VM?

[andrewconnell]

would you consider developing on your host instead of in a VM or is there anything else that requires you to be using a VM

@waldekmastykarz Not really an issue today since SPFx will only work in O365, but when it does make it to on-prem one day, real data / customer isolation would be a valid use case. Also consider when will create one component siloed from everything else and not multiple within a bigger solution for a customer. Workbench is nice, but at some point you need a full working environment to work against.

[waldekmastykarz]

@andrewconnell how would we deal with the same requirements with O365 (access to real data, customer isolation, representative environment)? Couldn't we use the same approach with on-prem (like having a central location used by the whole team as opposite to each dev having their own machine)? It will be interesting to see how the story evolves when we will be able to use SPFx for building complete solutions rather than only Web Parts.

[andrewconnell]

@waldekmastykarz A shared environment is always nice in theory, but in the past has never been a good option with multiple dev's on the same project... what you describe is an integration environment, NOT a developer environment.

You want to develop in isolation so you know what you are working on will not affect / be affected by others. When you have shared environments, that's not true.

Today, with the capabilities of SPFx used to build weather web parts or isolated customizations like that, it's not much of an issue. However, I'm assuming the vision of SPFx is much bigger (haven't seen it discussed publically) to support much more complex scenarios so this will become an issue over time.

[waldekmastykarz]

I know: shared environment didn't work for numerous reasons like overwriting each others work or apppool recycles on every deploy.
In the past Chris O'Brien ran a series of posts how they did team development on O365 and I've been wondering if SPFx changes any of that or not. As you said, it might come down to what we can use SPFx for in the future: standalone customizations or complete portals, that will determine the dev story.

[GreyMana-git]

@waldekmastykarz It's like @andrewconnell wrote this is mainly an issue with on-premise development. For example you can't reorganize a global taxonomy termset when it's bound to some code logic (e.g. in the custom properties) in a shared environment.
And even with client side solutions (we only use SSOM via powershell and disallow farm solutions) there are situations where you avoid testing a in-development code snapshot in a shared enviroment. For example when you use feature branches and didn't have merged them.

Right now the majority of my projects are on premise, but if I only had O365 I could easily use my host system and keeping the files on the regular HDD.

[nickpape]

We did some work to dedupe the node_modules folder, which should be around 165mb at the moment.. However, the discussion about which part of the build tools should be opt-in or opt-out is still ongoing. We haven't unveiled the black box yet, as we are still working on developing a story that: A) gives you powerful OOB tooling, B) is stable and solid, C) that can easily be extended with either gulp-core-build or plain gulp tasks, and D) which gives you more leverage over the dependency graph.

We have been discussing several ideas in this vein, and will continue to update the community with our thinking.

[andrewconnell]

When you say “black box”, are you referring to the dynamic generation of the webpack config & not exposing the gulp tasks?

[chakkaradeep]

Apologies for the late reply, this should have been updated long back. We published a doc on the basic toolchain packages that we use. However, it would be impossible to go through and document each of those packages along with their dependencies and further down to its individual package dependencies.

I think over the year, the toolchain has optimized a lot of its dependencies and package sizes.

I am closing this now. Please create a new thread if you have any specific issue.

[msft-github-bot]

Issues that have been closed & had no follow-up activity for at least 7 days are automatically locked. Please refer to our wiki for more details, including how to remediate this action if you feel this was done prematurely or in error: Issue List: Our approach to locked issues