Webpacker: Inconsistent file names when building on multiple servers

[scottrobertson]

Hey

Apologies if this is the wrong place for this type of issue.

We have 2 web servers, each running behind a load balancer. During deployment we run the following on each server:

bundle exec rake assets:precompile

This has worked totally fine until we have added webpacker into the mix. Now each of the manifest.json files have different file hashes inside them:

{
  "header.css": "/packs/header-436715b1b24e023140032846f44ebb5c.css",
  "header.js": "/packs/header-39a13196ce8946d39988.js",
}

{
  "header.css": "/packs/header-436715b1b24e023140032846f44ebb5c.css",
  "header.js": "/packs/header-9c53e3fbb9e8b6312123.js",
}

Note that the CSS file hashes are the same, but the JS are not.

Where do these hashes come from? Shouldn't they be consistent like all other assets?

[renchap]

It comes from here: https://github.com/rails/webpacker/blob/master/package/environment.js#L68

[contenthash] is based on the hash of the file content, so it seems that running webpacker on your 2 servers does not generate the same result. Can you have a look at what is different between the 2? Its probably due to a version mismatch, or environment variables.

The best way to deal with this is to generate your assets on one server, and copy them to the other one once its done.

[scottrobertson]

Hey @renchap, thanks for the reply.

I don't see how that can be possible. Same Ruby version, same Node version, same Yarn version, all the same bundle versions and packages installed via yarn.

What env vars would impact this?

Copying from 1 server to the other is not really going to be possible for us. We use cloud66, and they handle all of the deployment side of things (basically just capistrano under the hood from what i understand). Will reach out to them and see what they can find too.

[renchap]

Can you look at the md5 or sha1 hash of the 2 versions? Or can you make them accessible somewhere?

[scottrobertson]

f9cdbd74f889cd4c9c18b5f8bcb8651b  header-9c53e3fbb9e8b6312123.js
ddd7592a56b4d33f4fd62109c973c994  header-39a13196ce8946d39988.js

So the content must be different.... urgh

File sizes are identical though:

31689 Jan 24 18:59 header-39a13196ce8946d39988.js
31689 Jan 24 18:59 header-9c53e3fbb9e8b6312123.js

Flipping between the files in CLI seems to suggest they are the same (very scientific...)

[scottrobertson]

Spinning up a third server now to try and see if that one is different yet again. Should help us determine what is going on.

[scottrobertson]

Added a third server, and the hash is different yet again. Will do some more debugging in the morning.

[pvande]

@scottrobertson You're not crazy. [chunkhash] isn't stable by default.

As you can see the bundle's name now reflects its content (via the hash). If we run another build without making any changes, we'd expect that filename to stay the same. However, if we were to run it again, we may find that this is not the case [...] This is because webpack includes certain boilerplate, specifically the runtime and manifest, in the entry chunk.

[scottrobertson]

Thanks @pvande

So... in terms of webpacker, is there a clean way to fix this? Seems like a pretty fundamental flaw in the whole system (webpack)...

[rossta]

@scottrobertson My interpretation of the docs above is a little different. The example described explains how the [chunkhash] of several chunks can change even if you've only modified only one. If you're not building multiple chunks, or haven't modified any files, then [chunkhash] should be stable across builds and across servers.

I tend to agree with @renchap — that there may be some differences within or across the environments on your servers causing hashes to be inconsistent. Curious—do the hashes change across successive builds on the same server (without changes in file contents)?

[scottrobertson]

@rossta Hey

If i just run rake assets:precompile on the production server, then no they do not change. They only change when it's going through the deploy process.

Does chunkhash only take the contents into consideration, or does it use things like timestamp of the files etc? (cannot find that info anywhere).

[scottrobertson]

Scratch that. Compiling on demand does not seem to actually be doing anything. Will try some other things.

[rossta]

More info would be helpful to, like Webpack/Node/Rails/Ruby/etc versions, any customizations you're doing to your Webpack config if you're willing to share...

[scottrobertson]

Our webpack/production.js

const environment = require('./environment')

const config = environment.toWebpackConfig();
config.devtool = 'sourcemap';

module.exports = config;

Webpack: 3.2.0
Webpacker: 3.0.2
Rails: 4.2.x
Ruby: 2.3.x
Node: v6.12.3

[scottrobertson]

We got it!

Thank you very much to @tyvdh, everyone in here and Cloud66!

const environment = require('./environment')
const WebpackMd5Hash = require('webpack-md5-hash')

environment.plugins.append(
  'WebpackMd5Hash',
  new WebpackMd5Hash()
)

environment.config.set('output.filename', '[name]-[chunkhash].js')
environment.config.set('devtool', 'sourcemap')

module.exports = environment.toWebpackConfig()

Switching to use https://github.com/erm0l0v/webpack-md5-hash seems to be the key here.

[rossta]

I haven't tried that plugin, but I'd also be curious if upgrading to the latest Webpacker, 3.2.1 resolves your issue independently. The base Webpacker configuration went through some implementation changes that introduced some non-deterministic behavior in earlier versions of 3.x.x though not sure whether that would explain the behavior you're seeing.

[p0wl]

you wrote the files have the same size but have a different chunkhash, what is the diff between the files? The only thing that I can think of is a different order in the modules list that webpack maintains. But it wouldn't make any sense that the md5 is the same then...

Are you using yarn or npm in production?

Please have a look at the diff and share it here (if it does not contain anything secret).

[scottrobertson]

@p0wl sorry, i missed this. I will try and get that for you.

So, the fix we did worked, but it has a very big flaw. It's not getting a new hash each time it builds (even if the content has changed).

[kjleitz]

@scottrobertson did you ever get this to work for you? We have the same issue. We have multiple app servers under a load balancer, and having different fingerprints on the packed files on each server has thrown an _enormous_ wrench into the works.

• can't use [contenthash] instead of [chunkhash], since it seems to be only available in webpack 4.x (latest webpacker uses 3.x and we can't rely on the 4.0 beta)
• can't supply a function to output.filename to roll our own, since webpack-dev-server _had_ a bug which disallows it (this was fixed in webpack-dev-server 3.x but webpacker uses 2.x). The dev server is kind of necessary for us, because the bin/webpack compile time currently takes five whole minutes for our app, whereas the dev server cuts that down to next to nothing
• the webpack-md5-hash doesn't help us because it has a bug where consecutive runs with changes don't actually change the hash

Without deterministic hashes on packed files between the servers, we're getting random 404s for script tags on our pages.

I've resorted to storing the hash of the last commit as an environment variable when our deployment scripts run, and then doing:

const lastCommitHash = process.env.LAST_COMMIT_HASH;

// ...

  output: {
    filename: lastCommitHash ? `[name]-${lastCommitHash}.js` : '[name]-[chunkhash].js',
  },

// ...

...but that's really kind of terrible. Solves the problem of ensuring all the app servers have the same fingerprint on their packed files, which is the most important bit, but it invalidates the cached filenames on every deploy.

[scottrobertson]

@kjleitz We ended up using something similar to your commit hash solution.

However, we have since moved to Kubernetes, and therefor just build the image once so it always has the same ID.

[kjleitz]

@scottrobertson sounds like we need to move to Kubernetes 😉

[scottrobertson]

@kjleitz ha, well that is a rather drastic change to fix 1 issue :D

[grantbdev]

@kjleitz I was recently able to resolve this by isolating the lines that were causing different hashes in a codebase. Bisect the imports via commenting out and building the Webpack bundle, and then compare on two different machines or the same machine with the codebase copied in two different folders.

It turned out that the default handlebars-loader configuration loads the runtime with an absolute path which was the cause of the slight difference that made the hashes inconsistent across servers.

[kjleitz]

@grantbdev interesting—I don't know that your solution would have solved our problem specifically, because I believe the build-unique hash appended to the filenames might have been unique regardless since they aren't based on the md5 of the constituent files... so there were no specific lines causing the issue; it was simply that rake assets:precompile was being run on four independent machines with different fingerprint outputs despite the otherwise identical setup of those servers. Maybe I'm misunderstanding?

Using the last commit hash as the fingerprint (and setting CI=true to prevent unnecessary compilation) has solved our problems, for now.

It turned out that the default handlebars-loader configuration loads the runtime with an absolute path which was the cause of the slight difference that made the hashes inconsistent across servers.

Is the absolute path different between your servers? Why? (just curious, because in our setup the servers are all effectively clones of each other)

[kpheasey]

I encountered this issue after upgrading from webpacker 3.5.5 to 4.0.2. The [chunkhash] naming was inconsistent between servers on my OpsWorks stack, even after creating new instances. Using [contenthash] fixed my problem.

// config/webpack/production.js

process.env.NODE_ENV = process.env.NODE_ENV || 'production';

const environment = require('./environment');

// use [contenthash] naming to ensure uniform names across servers
environment.config.output.filename = 'js/[name]-[contenthash].js';
environment.plugins.get('MiniCssExtract').options.filename = 'css/[name]-[contenthash].css';

module.exports = environment.toWebpackConfig();

[geraldchen890806]

check

new webpack.DefinePlugin(****),

[sharang-d]

Can this be closed as #2094 is merged?