Webpack-encore: [Performances] Slow builds

[slince]

C:\www\lookbook>yarn watch
yarn watch v0.27.5
$ encore dev-server --port 8090 --disable-host-check --hot
Running webpack-dev-server ...

  WARNING   Passing an absolute URL to setPublicPath() *and* using the dev-server can cause issues. Your assets will load from the publicPath (http://127.0.0.1:8090/) instead of from the dev server URL (http://localhost:8090/).
 DONE  Compiled successfully in 75262ms                                                                         21:33:25

 WAIT  Compiling...                                                                                             21:33:25

 DONE  Compiled successfully in 214ms                                                                           21:33:25

 WAIT  Compiling...                                                                                             21:34:12

[slince]

here is my webpack.config.js

const Encore = require('@symfony/webpack-encore');
const path = require('path');
const glob = require('glob');


const buildPath = 'public/build';

const config = {
    buildPath: path.resolve(buildPath),
    assetsPath: path.resolve("./assets"),
    baseScssPath: path.resolve("./assets/base/scss"),
    baseJsPath: path.resolve("./assets/base/js"),
    lookbookScssPath: path.resolve("./assets/lookbook/scss"),
    lookbookJsPath: path.resolve("./assets/lookbook/js"),
    mapPath: path.resolve(buildPath + '/manifest.json')
};

Encore
    .setOutputPath(buildPath)
    .setPublicPath('/build')
    .cleanupOutputBeforeBuild()
    .autoProvidejQuery()
    .enableSassLoader()
    .enableVersioning(true)
    .enableSourceMaps(!Encore.isProduction())
    .configureTerserPlugin(function(options){
        "use strict";
        options.comments = false;
    })
    .autoProvideVariables({
        '$': 'jquery',
        'jQuery': 'jquery',
        "window.jQuery": "jquery",
        '_': 'lodash',
        "window.lodash": "lodash"
    })
    .addExternals({
        'jquery': 'window.$',
        'jQuery': 'window.$',
        'lodash': 'window._',
    })
    .addRule({
        test: /\.njk$/,
        loader: 'nunjucks-loader'
    })
    .enableSingleRuntimeChunk();

if (!Encore.isProduction()) {
    Encore.enableVersioning(false);
    Encore.setPublicPath('http://127.0.0.1:8090')
        .setManifestKeyPrefix('build/');
}

// 创建共享
Encore.createSharedEntry('base/js/vendor', config.baseJsPath + '/vendor.js');
Encore.addEntry('base/js/base', config.baseJsPath + '/base.js');
Encore.addStyleEntry('base/css/base', config.baseScssPath + '/base.scss');

Encore.addEntry('lookbook/js/app', config.lookbookJsPath + '/app.js');
Encore.addStyleEntry('lookbook/css/app', config.lookbookScssPath + '/app.scss');

//final webpack config
module.exports = Encore.getWebpackConfig();

[Kocal]

I don't think Compiled successfully in 214ms is too slow.

By the way, don't hesitate to use words and sentences the next time you want to communicate with humans.

Merry Xmas!! :tada:

[slince]

@Kocal hi, 200ms is only one of them, and each compilation after it takes almost 70,000+ms. At least 70000ms is required for each recompilation process; There is no such problem in the old version;

don't hesitate to use words and sentences the next time you want to communicate with humans.

I don't know how to organize the language to describe my problem because of my bad English. i'm sorry.

[weaverryan]

There is no such problem in the old version

Do you mean that the performance became much worse after upgrading from an older version of Encore? What version of Encore were you using before? And approximately how fast was the build time?

[slince]

@weaverryan yes, the old version is "@symfony/webpack-encore": "^0.19" and nodejs version is 8.x;
now version is "@symfony/webpack-encore": "^0.22.2" and nodejs is 10.x

[slince]

Encore will compile twice each time you start compiling：

[nickpolet]

I can't recreate your issue. Are you using the --watch flag when running webpack? It looks like something is causing it to re-run but I don't think that is your issue.

Have you read through the warnings that you see. The DEPRECATION warning is telling you to use .enableSingleRuntimeChunk(). Have you tried that?

Also, it's really tricky to try and debug an issue if you don't describe what is happening in any detail. Use google translate and state that your description has been translated by google translate.

[slince]

@nickpolet @weaverryan I found that it may be because css has too many lines (more than 5000 lines) causing the compilation to become slow

[pcarballeda]

I have a website with 114 entries and with my curent webpack configurarion, dev-server expenses about 50 seconds to compile all this assets. But with symfony encore I've just added 6 entries and expense about 100 seconds. So, definitely it's to slow and documentation is really poor.

[L3tum]

I have a website with 131 files apparently, not very big though and mostly very small module files. It takes Encore 4 minutes to compile it.

[Lyrkan]

@pcarballeda Were the 50 seconds also with Encore and using the same command (the optimizations are not the same between environments)? If that's the case then the extra 50s are probably related to the content of those 6 files. You could maybe pinpoint what's causing your issue by adding/removing things (either from the entry list or from the files themselves)?

@L3tum By "131 files", do you mean entrypoints? In both cases, are you importing things in them? Don't forget that Webpack has to process those imports as well.

---

That being said, having a build time of X minutes is definitely not normal.
Documenting how to improve that is a bit hard because the issue will probably not be the same between two projects... but here are some things you could try:

• Run Webpack with the --profile option, you should be able to see a lot of metrics and maybe something will stand out
• Try disabling source maps... if it reduces your build time drastically, maybe you could try another source-map type
• If you are using SASS/SCSS the resolveUrlLoader option is known for slowing down some builds. It is enabled by default since imports in CSS files feel more natural this way (it makes the paths relative to the file you are editing), but you can disable it by using the second parameter of Encore.enableSassLoader()
• If you are using TypeScript you could try calling Encore.enableForkedTypeScriptTypesChecking() to make the type-checking part of the build faster
• If the external libraries you are using allow it, only import the things you need (for instance import { Observable, Subject } from 'rxjs' instead of import * as rx from 'rxjs').
• More generally, keep in mind that importing hundreds of files into a single entrypoint is not the same than having hundreds of entrypoints since each entrypoint will have its own dependency graph that'll need to be computed, transformed and eventually compared to other graphs (to extract common code).... So if you can and that's not already the case, avoid having multiple entries for a single page, create an unique file and import everything your page needs in it.
• If you can, try compiling on another computer... builds can be resource-intensive and it could be an hardware/software issue (for instance if your source files are stored on a network drive).
• Try updating your build-related dependencies (Encore itself, loaders, plugins, etc.)

[L3tum]

@Lyrkan No, I have 2 entrypoints with a total of 131 files being processed by webpack. I already tried disabling the source map and disabling resolveUrlLoader doesn't seem to help either.

Profile revealed that the majority of the time is spent in "building" (up to 300 seconds with a quick glance?) but nothing else.
Could it maybe be that Webpack is only using one core but is executed multithreaded so a lot of context switching is happening? I haven't found anything regarding Encore enabling multithreading for Webpack but that'd be the only thing I could think of.

[Lyrkan]

@L3tum Weird then... --profile should've given you the build times file per file, were they equally distributed?

AFAIK Webpack doesn't handle anything related to threading/running things in parallel unless you use plugins that add this kind of feature (HappyPack/Fork TS checker/parallel-webpack/...). I think Encore only does something like that by default for the Terser plugin (see #497).

I added two points to my previous answer about hardware/software issues and dependencies update.

[L3tum]

@Lyrkan Thanks for all the tips!

Yeah, it gave me these metrics per file, but most of them were around 200-300 seconds with the occasional few milliseconds in between. Which is why I asked about multithreading because 200-300 seconds per file would mean a lot more than 4 minutes....

We don't use any extra plugins though, really Encore only with Sass, Versioning and Source Maps enabled.

It is executed in a VM, but that VM has 4 cores and it'd seem weird if Hyper-V-mounted filedrives would be that slow. Is there any way to get some output out of webpack about what it's doing at that moment? Maybe it hangs at some file due to some obscure issue or so.

Oh, I also had a page fault yesterday which corrupted my entire workspace, but the build time was already pretty abysmal before that. I hadn't thought about that though, maybe I'll just do an npm install again

[jfcherng]

As https://github.com/symfony/webpack-encore/pull/497 has been shipped with v0.23.0. The performance for compiling JS would be better now?

[Kocal]

For minimizing JS, yes

[slince]

@jfcherng I have upgraded to 0.23, but nothing has changed.

 DONE  Compiled successfully in 101958ms

[slince]

@Lyrkan @weaverryan
There is a http background image in my css file, the compilation speed is faster when I comment this line.

  .bottom-banner {
    width: 100%;
    height: 160px;
    text-align: center;
    margin-top: 30px;
    margin-bottom: 100px;
    //background: url(//pucenter.dataoke.com/images/pro_index_bot_bg.jpg?t=201811301045);

In fact, this shouldn't take too much time to download this image; the network speed is fast enough.

[Lyrkan]

@slince Does it also do it when you specify the full scheme? For instance:

.bottom-banner {
   // (...)
   background: url(https://pucenter.dataoke.com/images/pro_index_bot_bg.jpg?t=201811301045);
}

I suspect this is related to the resolve-url-loader that tries to resolve things like that.
They fixed something similar a while ago (https://github.com/bholloway/resolve-url-loader/pull/68) but it only works if you specify the protocol.

In fact, this shouldn't take too much time to download this image; the network speed is fast enough.

It doesn't really matter since nothing should be downloaded in this case anyway.

[slince]

@Lyrkan Yes, everything is getting fine, when I specify the protocol as https.

[slince]

Hi, i find that when i remove Encore.createSharedEntry , it won't compile twice.

// Encore.createSharedEntry('base/js/vendor', config.baseJsPath + '/vendor.js');

[L3tum]

I don't know if it's worth anything but compile times are now pretty fast at usually around 10 seconds for me. We didn't change all that much about it to my knowledge aside from splitting up the files a bit more.

My guess is that either the file was too large hitting some limit in my setup or some change here (I didn't follow this repo too closely, sorry) actually fixed the performance issues. It may just have been some weird Windows issue that they've fixed, there have been like 4 updates for my machine since my last comment on this issue.

Anyway, thank you for your time!

[pscheit]

i'm sure this is all the time the resolve-url-loader, because it will scan your WHOLE project for single font/import files from sass.

its a bit hacky, but you can enable debug for the resolve-url loader, when you modify your webpack.config:

  .configureLoaderRule('sass', loaderRule => {
    // enable debug for resolve-url-loader
    // check here which indexes, or run a "find" here
    console.log(loaderRule.oneOf)

    loaderRule.oneOf[0].use[2].options.debug = true
    loaderRule.oneOf[1].use[2].options.debug = true    
})

it will then tell you which files it searches and why

[Lyrkan]

@pscheit The resolve-url-loader should be less of an issue nowadays since we are now using v3 that removes the "magic" resolving behavior of the previous versions.