Angular-cli: bug: all RxJS operators pulled into prod bundle when using lettable imports from 'rxjs/operators'

[Splaktar]

Versions

Angular CLI: 1.6.3
Node: 8.9.0
OS: darwin x64
Angular: 5.1.2
... animations, common, compiler, compiler-cli, core, forms
... http, language-service, platform-browser
... platform-browser-dynamic, platform-server, router
... service-worker

@angular/cdk: 5.0.2
@angular/cli: 1.6.3
@angular/flex-layout: 2.0.0-beta.12
@angular/material: 5.0.2
@angular-devkit/build-optimizer: 0.0.36
@angular-devkit/core: 0.0.22
@angular-devkit/schematics: 0.0.42
@ngtools/json-schema: 1.1.0
@ngtools/webpack: 1.9.3
@schematics/angular: 0.1.11
@schematics/schematics: 0.0.11
typescript: 2.5.3
webpack: 3.10.0

Repro steps

https://github.com/Toxicable/issue-lettable-bundle-size

Observed behavior

From https://github.com/angular/angular-cli/issues/8854#issuecomment-354532989, "There appears to be a bug in Webpack that is causing the CLI to import all of the RXJS operators if you use any lettable operator imports without doing each as a deep import individually.

Upgrading to lettable operators in CLI 1.6.3 caused RxJS to go from 60 KB to 122 KB for me. It sounds like a fix is 'coming soon', but I haven't been able to track down the issue to follow."

Another user mentioned a bundle size increase when moving to lettable operators in the CLI: https://github.com/angular/angular-cli/issues/8720#issuecomment-348804558

Desired behavior

From the 5.0 blog post:
"These new operators eliminate the side effects and the code splitting / tree shaking problems that existed with the previous ‘patch’ method of importing operators."
"RxJS now distributes a version using ECMAScript Modules. The new Angular CLI will pull in this version by default, saving considerably on bundle size."

I desire a considerable reduction in bundle size, not an increase.

Mention any other details that might be useful (optional)

I tracked this down across Slacks, GitHub, and Twitter as I was told that there was a known Webpack problem that would be fixed soon and then pulled into the CLI. As far as I can tell, any similar Webpack issue was solved in 3.9.0 and the current CLI already includes it as it is using 3.10.0 now.

I can provide source map explorer screenshots if needed, but the linked repo above should enable you to better reproduce and create your own source map explorer output.

RxJS Docs for lettable operators and builds: https://github.com/ReactiveX/rxjs/blob/master/doc/lettable-operators.md#build-and-treeshaking

Workaround

Breaking out all of your imports into individual (one import per line) deep imports seems to avoid this issue, but it is very painful and not desirable to have these extra lines of imports in projects of any significant size or scope.

[filipesilva]

/cc @IgorMinar @jasonaden @benlesh @Toxicable

[IgorMinar]

I'll look into this

[IgorMinar]

The problem is that you have -vc turned on. Vendor chunking results terrible size regressions. This is because webpack 3 generates a ton of cross-chunk module-stitching code that prevents uglify from dead-code-eliminating the unused operators. Once you turn off vendor chunking everything will work correctly.

If things go as planned webpack 4 should significantly improve this situation, but until then definitely always always turn off vendor chunking if you care about payload size. This is why -vc is disabled in cli by default.

with vendor chunking:

without vendor chunking:

[Splaktar]

I'm not using -vc in my project and I'm still seeing this issue. Here's the command that I'm running:
ng build --target=production --env=staging

My env for staging is the following:

export const environment = {
  production: true,
  firebaseConfig: { ... },
  googleAnalyticsTrackingId: '...',
  mapsApiKey:  '...'
}

I'll post the source map explorer screenshots in a minute.

[Splaktar]

According to https://github.com/angular/angular-cli/wiki/build, --vendor-chunk (aliases: -vc) default value: true.

Perhaps the issue is that the default should now be false?

[Splaktar]

ng build --target=production -sm
Pre-lettable operators (60.63 KB for RxJS):

ng build --target=production -sm
With lettable operators (122.39 KB for RxJS):

ng build --target=production -sm --vc=false
With lettable operators (122.39 KB for RxJS):

ng build --target=production -sm --vc false
With lettable operators (122.39 KB for RxJS):

ng build --target=production -sm -vc false
With lettable operators (122.39 KB for RxJS):

ng build --target=production -sm -vc=false
With lettable operators (122.39 KB for RxJS):

ng build --target=production -sm -vc=false -cc=false
With lettable operators (122.39 KB for RxJS):

[filipesilva]

@Splaktar is your project public maybe? That way I could take a look.

Are you using lazy bundles as well? Can you try removing them and checking if there's a difference in rxjs?

[Splaktar]

@filipesilva the source isn't public. The staging and production sites are.

There is currently one major lazy loaded module. The other routes just go to simple components that are fairly static.

const routes: Routes = [
  {path: '', pathMatch: 'full', component: LandingComponent, children: []},
  {path: 'agents', loadChildren: './agents/agents.module#AgentsModule'},
  {path: 'profile/:id', component: ProfileComponent, canActivate: [UsersService], children: []},
  {path: 'terms', component: TermsComponent, children: []},
  {path: 'privacy', component: PrivacyComponent, children: []},
  {path: '**', component: PageNotFoundComponent, children: []}
];

@NgModule({
  imports: [RouterModule.forRoot(routes, {useHash: false})],
  exports: [RouterModule],
  providers: []
})
export class AppRoutingModule {}

[filipesilva]

If you turn that lazy route into a normal route, or remove it altogether, does the rxjs size in the main bundle change?

[Splaktar]

Removing my lazy loaded route and re-running with ng build --target=production -sm:

RxJS size is smaller, but I still see tons of operators that I am not using (especially w/o the primary AgentsModule in the build).

Here's the full source map explorer:

[devoto13]

Note that some operators come from libraries. So they are still needed in the bundle even if you don't use them directly.

[Splaktar]

In the AppModule, I use shareReplay, switchMap, filter, debounceTime.
In the AgentsModule (lazy loaded), I use those plus distinctUntilChanged, take, map.

[IgorMinar]

@Splaktar can you create a minimal reproduction similar to how @Toxicable created the one earlier?

I see that you are using firebase in your app and you are using the cjs distro of that library. I'm wondering if that's why all the operators are retained. A minimal reproduction would confirm or reject that theory.

[Splaktar]

I'm using "firebase": "4.7.0", and including it via .angular-cli.json like this:

      "scripts": [
        "../node_modules/hammerjs/hammer.js",
        "../node_modules/firebase/firebase.js"
      ],

It looks like I should be doing this instead:

      "scripts": [
        "../node_modules/hammerjs/hammer.js",
        "../node_modules/firebase/firebase-app.js",
        "../node_modules/firebase/firebase-auth.js",
        "../node_modules/firebase/firebase-database.js"
      ],

But I'm not sure how much that will help. I looked at their web setup page and Firebase JS SDK GitHub and I don't see any way to pick different types of modules.

I'll dig into that a bit more, try out firebase 4.8.0, and then work on building a new repo for a public reproduction.

Update: Firebase 4.8.0 and deeper imports did not change anything.

[filipesilva]

Hm... importing "../node_modules/firebase/firebase.js" in scripts shouldn't result in dependencies insofar as importing is concerned, because importing should not work at all in scripts.

[Splaktar]

OK, I've created a reproduction repo here. Time to get to sleep now...

Using ng build --target=production --env=prod -sm:

[filipesilva]

@Splaktar awesome, I was just putting up a repro myself. This way it's much easier. Will take a look.

[filipesilva]

Running some baseline tests, measuring size of node_modules in main.*.bundle.js and where it comes from via ng build --target=production --env=prod -sm and source-map-explorer:

• normal build: 1.09 MB node_modules (@angular/* 575 KB, rxjs 122 KB)
• commenting out lazy route: 0.987 MB node_modules (@angular/* 542 KB, rxjs 56KB)
• forcing lazy module to be non-lazy: 1.01 MB (@angular/* 543 KB, rxjs 56 KB) (lazy bundle used to be ~32kb)

I think having a lazy bundle is having the same effect on bundle size as the vendor chunk (described in https://github.com/angular/angular-cli/issues/9069#issuecomment-354896045). This makes sense because vendor chunk is not special, it's just another chunk, just like a lazy chunk is just another chunk.

This makes me think that the best hope to address this problem is Webpack 4. Support is being tracked in https://github.com/angular/angular-cli/pull/8611.

Note: regarding https://github.com/angular/angular-cli/issues/9069#issuecomment-354904486, when Build Optimizer is used then vendor chunk will default to false (https://github.com/angular/angular-cli/wiki/build#--build-optimizer-and---vendor-chunk)

[Splaktar]

@filipesilva thank you very much for looking into this!

Oh that's really interesting about Build Optimizer and Vendor Chunk options. Perhaps https://github.com/angular/angular-cli/wiki/build can be updated with that information?

I guess at this point, we should recommend that teams do not use lettable operators if they are using the Angular CLI with lazy loaded routes (this is every developer as we have been telling them for a year+ to lazy load every possible route). Do you see any alternative to this?

I need to advise multiple active clients on this atm in addition to my own projects.

[devoto13]

@Splaktar I remember facing this problem some time ago when importing from rxjs/operators, so I switched to using rxjs/operators/<name> imports everywhere and haven't had problem with RxJS bundle size ever since. It's not a big deal, since imports are added by IDE automatically and I never have to write them manually anyway.

Maybe one option for you is to advice using deep imports (at least as a temporary solution). I've updated your reproduction from https://github.com/angular/angular-cli/issues/9069#issuecomment-354976634 to use this approach and RxJS bundle size is back to normal.

[Splaktar]

@devoto13 thank you very much for looking into that! It was a question that I was just asking myself.

What you have shown though is that using lettable operators with deep imports is not a solution for "saving considerably on bundle size". I can't advise my clients to add a considerable amount of boilerplate (yes it can be auto filled by IDEs in most cases) and change to a new syntax if there is no significant impact on bundle size.

I'm going to push up a branch w/o lettable operators for comparison.

[devoto13]

@Splaktar Thanks! Please do. Will be very interesting to see how much better it gets without lettable operators.

[Splaktar]

I don't expect it to be better w/o lettable operators other than that it matches most of the existing docs and apps that are out there today. The issue is that it's not clear how lettable operators are "considerably better" in Angular w/ the Angular CLI today.

Non-lettable operator branch (i.e. old fashioned): https://github.com/DevIntent/angular-example/tree/withoutLettableOperators

The only difference that I can see is the savings of the add operator when using lettable operators. This saves 612 B only.

[songawee]

I'm not sure if this bug is specific to the CLI, but I've reproduced the issue of increasing bundle size with RxJS without using the CLI - https://github.com/songawee/angular-example.

Perhaps there's something more at play here? 🤔

[filipesilva]

Had a go with the latest webpack4 alpha and only saw a 0.4MB difference between a normal build (1.28MB) and one with the lazy route commented (1.24MB). There seems to be a bug with the sourcemaps so I couldn't look at them.

Also unsure why the main bundle got bigger overall but I wouldn't read much into that. I can't really say if uglify and everything else is working correctly, and to run webpack4 at all some stuff is disabled as well. Once webpack4 goes into beta and I can get a full build setup I'll look into size parity.

I also tried using http://chrisbateman.github.io/webpack-visualizer/ but that seemed to show that now all of the rxjs operators were retained in all cases. I have to look into this better in the webpack 4 betas.

[IgorMinar]

@songawee the issue is really in webpack v3 and the whole bundling stack outside of the cli, so it's expected that you can reproduce it without the cli.

@filipesilva keep in mind that I don't expect that webpack v4 will just make things work out of the box. we'll need to flag the rxjs package as side-effect-free and possibly do even some repackaging of that npm package. all of that is planned for rxjs v6

@Splaktar the current workaround is to use module id rxjs/operators/<operator-name> instead of rxjs/operators to import operators. You definitely want lettable operators even with this flaw because the original operators that you had to bring in via /add/operator/<operator-name> always end up in the main bundle (critical path) even if they are used only in lazy-loaded code, and more importantly are headache due to far-away effects that can make your code work without /add/operator/foo import if you rely on an operator that was brought in by another part of your code or a 3rd party library (if this far-away code changes and removes the operator import, your code will suddenly break and you will wonder why).

[Splaktar]

Thank you for the feedback and information.

Is there a Webpack issue or issues to track for this? Is the Webpack team aware of this?
Is there an RxJS issue to track for the supporting work coming in v6?

[TheLarkInn]

Is there a Webpack issue or issues to track for this? Is the Webpack team aware of this?

Do we know whether or not this is still an issue with v4? Would anyone be willing to try it in latest commits on our next branch? (We can assist if needed). We are getting pretty darn close to merging next into master and want to make sure if this is not fixed in 4 that we can get an issue to track and resolve. ( cc @sokra for visibility )

[TheLarkInn]

Also, just as beneficial: could someone create a _webpack minimal_ reproduction of this issue.

(Example: no angular deps, but just a few modules, minimal config with new RxJS version, that we can clone and test).

[filipesilva]

@TheLarkInn I tried with [email protected] and it still seemed to happen. I will setup a bare webpack repro tomorrow so we can look at this sort of effect in more detail.

It boils down to this:

• currently unused side-effect free exports are dropped via uglify (via the PURE comment)
• uglify can only drop code that is in the same chunk+module (via ModuleConcatenationPlugin)
• when there are multiple chunks wired via webpack modules, uglify cannot drop these exports

@IgorMinar am I missing something here?

I'll setup a repro where webpack3+uglify is used on code using imports from rxjs with code splitting. Then I'll try to use webpack4 and mark rxjs as being side effect free by editing package.json directly.

This should let us see the exports being dropped in webpack 4.

[songawee]

@TheLarkInn I haven't tried v4 yet, but I did try and create a minimal webpack repo - https://github.com/songawee/webpack-example. The results are a bit different from what I expected, but the bundle size does grow when not using deep imports. Let me know if you need anything updated in the repo to make it a better example.

If we did want to test v4, would we have to specify RxJS to be side-effect: false?

[IgorMinar]

@TheLarkInn @filipesilva as I mentioned before the rxjs package needs to be opted into the side-effect free treatment for webpack v4 optimizations to have any effect. Currently you'll have to patch rxjs's package.json to do that.

[IgorMinar]

If we did want to test v4, would we have to specify RxJS to be side-effect: false?

@songawee yes

[IgorMinar]

@Splaktar there is no webpack 4 issue specifically for this, it's an optimization that was baked into the v4 branch already, so if there was an issue, it should be closed now.

I'm also not aware of a particular rxjs issue for turning on the side-effect free flag so I created one: https://github.com/ReactiveX/rxjs/issues/3212

[IgorMinar]

The webpack example demonstrating this feature is here: https://github.com/webpack/webpack/tree/next/examples/side-effects

[filipesilva]

Here is a repo showing RxJs sizes with webpack 3 and 4: https://github.com/filipesilva/webpack-rxjs-operators.

Only the used operators are retained with Webpack 4.

---

Webpack side-effects flag comparison on RxJs operators

This repo compares how using "sideEffects": false reduces bundle size for RxJs operators.

It includes a lazy chunk to add indirection to module loading. This prevents Webpack 3 with UglifyJS from dropping the unused RxJs imports but Webpack 4 side effect detection can still remove used code.

cd webpack-3
npm i
npm start 
# source-map-explorer will open on your browser, with all rxjs operators in the main bundle
cd ../webpack-4
npm i
npm start
# source-map-explorer will open on your browser, with only the used rxjs operators in the main bundle

Webpack 3 bundle sizes:

-rw-r--r-- 1 kamik 197609  159 Jan  5 16:37 0.js
-rw-r--r-- 1 kamik 197609 125K Jan  5 16:37 main.js

Webpack 4 bundle sizes:

-rw-r--r-- 1 kamik 197609 5.7K Jan  5 16:40 0.js
-rw-r--r-- 1 kamik 197609 8.7K Jan  5 16:40 main.js

License

MIT

[filipesilva]

@IgorMinar I created a similar repro with a basic Angular app (https://github.com/filipesilva/webpack-angular) and the results weren't so good. There was no real size difference.

I think this is because Angular 5 uses flat ESM.

According to doc in https://github.com/webpack/webpack/tree/next/examples/side-effects:

The "sideEffects": false flag in big-module's package.json indicates that the package's modules have no side effects (on evaluation) and only expose exports. This allows tools like webpack to optimize re-exports. In the case import { a, b } from "big-module-with-flag" is rewritten to import { a } from "big-module-with-flag/a"; import { b } from "big-module-with-flag/b".

Since Angular 5 is using flat ESM, it is not possible to rewrite import { something } from '@angular/core'; to import { something } from '@angular/core/something'; because there is no @angular/core/something dir/file.

We should run some tests with a non-flat ESM build of Angular to see the impact.

[IgorMinar]

Since Angular 5 is using flat ESM, it is not possible to rewrite import { something } from '@angular/core'; to import { something } from '@angular/core/something'; because there is no @angular/core/something dir/file.

I'm aware. There is a possibility of us changing the packaging strategy again in v6 to be webpack v4 compatible. We have yet to analyze the trade offs though.

[IgorMinar]

It includes a lazy chunk to add indirection to module loading. This prevents Webpack 3 with UglifyJS from dropping the unused RxJs imports but Webpack 4 side effect detection can still remove used code.

@filipesilva 125K vs 8.7K? that's actually a bit better than what I expected. Does the app still work?

[filipesilva]

It wasn't a real app to be fair. Just imports and console logs. Other rxjs imports, like Observable, were still there so that looks as it should.

[blackholegalaxy]

any news on that? Rxjs documentation states bundle size can be fixed through Webpack configuration. Is that intended to be fixed through webpack 4 upgrade?

Ref: https://github.com/ReactiveX/rxjs/blob/master/doc/pipeable-operators.md#build-and-treeshaking

[Lakston]

I just found this issue (should have looked for it earlier...) after spending my morning trying to figure out why I had all of rxjs being bundled on our app (@angular/cli": "1.6.7")

Also I haven't checked thouroughly yet but it seems like we have the same tree shaking problem with date-fns/esm pulling everything into the prod bundle.

I'll second @blackholegalaxy with his/her question, or if anyone can point us to an issue where we can track / fix this ?

Thanks !

[filipesilva]

@blackholegalaxy @Lakston this the the correct issue to track resolution.

The only way to truly achieve tree shaking of RxJs (in the current context) is to use Webpack 4 and have the package marked as being side effect free.

Without these two preconditions, it is only possible to treeshake RxJs operators in the main bundle, and only if the vendor bundle is disabled (as it is by default on prod builds).

[blackholegalaxy]

Sorry about my lack of knowledge of the "under the hood angular CLI logic", but could the solution on the link I provided above, by Rxjs team, work in the CLI context? It seems CLI relies on webpack 3 right?

[filipesilva]

@blackholegalaxy we already implement that in the CLI actually.

[blackholegalaxy]

Ok nice to know. We will then wait for webpack 4 improvements. Thanks for precision @filipesilva :)

[benlesh]

If you have Observable and an operator or two, that's a working app as far as RxJS is concerned.

Most applications use between 8-20 operators. I'd estimate the average is around 12-15.

[tonny008]

still have this issue in Angular CLI 6 with rxjs6 with no rxjs-compact.

[alinnert]

I found this issue because I was also experiencing this issue with a custom Webpack 4 + Babel 6 setup. I then tried Rollup just for fun to see if it makes any difference. Using RxJS and one/two pipeable operators resulted in a ~10 kb bundle with Rollup. Webpack on the other hand created a 120 kb bundle. But I noticed that Rollup requires the following line in .babelrc:

~ json
{
"presets": [
[
"env",
{
"modules": false // <- this
}
]
]
}
~

After adding this line in the .babelrc file of my Webpack setup as well its created bundle file shrank by ~110 kb to about the same size that Rollup produces.

Now I guess the Angular CLI uses TypeScript instead of Babel. From reading the documentation it seems to have a similar option in tsconfig.json called compilerOptions.module and a matching value of "None".

I didn't try this (and I don't have any Angular projects right now) but maybe this helps you finding a solution? But at least to me this also means that this is not necessarily a Webpack issue, right?

[Lakston]

Having this issue as well with all rxjs in my bundle, as well as all lodash-es when I only use 2 imports

Rx: 6.3.3
CLI: 7.0.6

[juarezpaf]

I have an Angular app using v6.1.8, CLI v6.2.3 and rxjs v6.3.2. Not importing anything from the internal path and it's still appearing inside of the bundle, in addition to that I am seeing bunch of operators that we are not using at all.

[alex-okrushko]

@juarezpaf One of the possibilities why this could be happening is that one of the deps libraries that you use also has rxjs in their deps/devDeps. At the same time that library could be configured to use compilerOptions.module: commonjs in their tsconfig.json file.
That's what was happening in my case.

[Splaktar]

I'm not sure when the exact fix went in, but this is no longer an issue in Angular CLI 7.3.8 as you can verify in my updated fork of the original reproduction.

[KishoreBarik]

Facing similar issue

Environment:

Note
Earlier I was seeing the left panel (900+ KB) was occupied by polyfills(at that time Size of Rx as nearly 53KB). So to solve this I found out I had to mention "es5BrowserSupport": true in angular.json and commented all imports in polyfills.ts except import "zone.js/dist/zone";.
Now I am seeing similar amount of size is being taken by RxJs. If I uncomment the polyfill imports the RxJs is back to around 53 KB but polyfills take 900+KB.

I just want to get rid of this 900+ KB sized thing.

[Splaktar]

@KishoreBarik this issue is closed. Please open a new issue and link to this issue from your new issue.

[angular-automatic-lock-bot[bot]]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{_This action has been performed automatically by a bot._}