Sp-dev-fx-webparts: Change SPFx build/gulpfile help needed

[shaipetel]

Category

• [x] Question
• [ ] Bug
• [x] Enhancement

Expected or Desired Behavior

I would like it when I build using gulp --ship that the JS file name under temp/deploy will not have the hash attached to it, or the ".bundle".
I would like for it to have the "project name"."package version".js for example.
I could not find a way to modify the gulpfile to achieve that.
I tried a lot of different things, and from talking to people it seems I am not the only one concerned about this.
I understand you put the hash for better caching, but that simply doesn't make sense to me since it forces me to produce a new package and force my users to update the package for minor fixes like typo, or other small insignificant issues.

Observed Behavior

The produced JS file name always has the ".bundle_hash" attached to it when building in --ship mode.
I could not find any way in the gulpfile.js build instructions to override that in a way that the file name will be something else and that it will be taken into the JSON package file correctly.
Now, I have to manually edit the file name AND edit it in the JSON file before calling gulp package-solution --ship which makes the build process manual, error prone and very time consuming.

I hope you can either give me a pointer on how / where to intercept this during the gulpfile, or if it is not possible (which is what I think) then consider adding a way to control/modify this file name somehow. Even specifying it manually as a variable in the gulp file would be better.

Steps to Reproduce

simply build using gulp --ship.
make some code changes - build again, you will get another file.

[VelinGeorgiev]

Maybe, you can create post build task to rename the files as you would like to. Have a look at this article: https://dev.office.com/sharepoint/docs/spfx/toolchain/integrate-gulp-tasks-in-build-pipeline

[waldekmastykarz]

@VelinGeorgiev that wouldn't be sufficient. You would also have to change the name of the URL in the manifest included in the generated package. The problem with doing this is, that it would complicate deployment. Because you would use the same file name for files with different contents, you have to shorten the caching period to allow users to see the latest changes on their computers without having to explicitly clear their cache after each deployment. Using a hash in the name allows you to use far future caching which is beneficial for increasing the performance of your solutions.

[VelinGeorgiev]

Got it. That makes perfect sense.

[shaipetel]

@waldekmastykarz
Well, using a hash in the file name might be a good thing for caching (questionable, since most caching servers today can look at the timestamp/modified of the file to bust the cache) but it is pretty bad for version management and tracking, as well as forcing you to install an update of an app before you can receive minor fixes.

Everyone I talk to agree on these two points:

Version tracking

After a short while, you will have no clue looking at your CDN which JS file belong to what version. No clue at all. Especially since using a hash can have 2 or more versions using the same JS file if the JS content did not change. This makes version management and tracking a real issue when you support products that are used by thousands of different tenants, on a product that within a year would have 20+ published updates.

Pushing updates

Forcing users to install a new version before they can receive bug fixes it not a good practice. We all remember the days you spent most of your time on your iPhone going to the app store and installing updates. Today - it is done automatically. Fixes and app updates are pushed to your device. Furthermore - some updates are done on the server and do not require any update on your device.
Imagine you have a release, and there was a typo, a label was wrong, an onclick even was acting up on IE. Minor bugs like that. I would like to be able to push minor fixes to my CDN without bothering the users. I can purge the cache for the file (most servers are smart enough to automatically do that when the file "modified" date has changed) and the user would get the fix.
Also - in O365, we are constantly facing breaking changes pushed in updates. Today, we test these in a first release tenant, and are able to push the fix to production before the breaking change hits our customers. We do that by pushing updates directly to our CDN.

Now, if the package signature has changed (you changed the using/require statements) - of course you will have to update the package.
But for any other fix, I don't see why I need to force my poor customers to update their apps 20+ times a year.

For these 2 reasons we do not include the hash in the file name but a version number of the package, and we track the build number in the code itself to monitor the version of the build that was served.

Makes sense?

You don't have to like it, you don't have to use it, but I do. It would be nice to have the option. After all, isn't that what an open source platform is all about?

[shaipetel]

@VelinGeorgiev Thanks, I am trying to write something to make this automated, but it is not simple.

I have to:

1. Figure out the file name in the temp/deploy folder - which is hard since it is always changing.
2. Get the version number from the package
3. Rename the JS bundle file
4. Edit the JSON definition file and rename the bundle file there

Once this is done, running "gulp package-solution --ship" will use the correct file name.

I am still trying to automate it all, if I ever get it working I will post my solution online for sure.

[waldekmastykarz]

Well, using a hash in the file name might be a good thing for caching (questionable, since most caching servers today can look at the timestamp/modified of the file to bust the cache) but it is pretty bad for version management and tracking, as well as forcing you to install an update of an app before you can receive minor fixes.

@shaipetel sure, you can use timestamp/modified to detect if the file has changed, but that requires the browser to issue a request to the server which is already affecting performance. If the browser can directly load the file from its cache without contacting the server, it's significantly faster.

After a short while, you will have no clue looking at your CDN which JS file belong to what version. No clue at all. Especially since using a hash can have 2 or more versions using the same JS file if the JS content did not change.

You don't need to deploy the files to the same container. If you want to keep track which files belong to which version, you can create a separate container for each version and deploy the files there. When you decommission a version, you can remove the whole container, removing all files with it.

I understand your suggestion with regards to pushing updates, but what if someone used that approach to change your script without you knowing it? What if the script all of a sudden did something else than what it was intended to do? Because it changed without your knowledge, you would have no idea about it. For many organizations it's a no-go, because security and integrity of their data is more important than deployment convenience.

Furthermore - some updates are done on the server and do not require any update on your device.
Imagine you have a release, and there was a typo, a label was wrong, an onclick even was acting up on IE. Minor bugs like that. I would like to be able to push minor fixes to my CDN without bothering the users.

That should be caught by a testing process and if you choose not to have it in place, then the consequence is more deployments to production,

No matter if you include a hash of a version number in the file name, if the file contents change, it's a different file. If you look at how SemVer defines version, the last number is intended for bug fixes, so every single time you push an update, you should update the version number. Even if you're working with prerelease files, there is a separate flag for that. So there is no such thing as updating packages without changing version numbers. That's just a bad practice.

I don't think it would be good to have this option, because it would allow people to circumvent the control that SPFx was designed to give to organizations. I understand that it would be convenient for developers but the data security and integrity is more important.

[shaipetel]

@waldekmastykarz I think you are a bit confused about what it does and how it works.

What do you mean by "someone changed my script without me knowing about it", you mean if someone hack my server and replaced files there?

How does having the hash in the file name helps protect against that?

You might be confusing this with a checksum/hash verification done in other places to verify digital signatures or file contents.
This is NOT the case in SPFx, as the hash is not getting verified anywhere.

So I don't see how data security or integrity has anything to do with this topic.

As for your other suggestion, creating a container: you mean you recommend changing the CDN drop location for my project, right?
So, to you, it makes more sense to ask the developer to modify the CDN setting for the project every time they build a new version?
That seems a bit too much to me. Especially if I have other resources I load from that same CDN location as the relative paths would always change.

Regarding minor issues that "should be caught by testing", you are not serious, are you?
How can you catch in testing a breaking change that was introduced as an update to o365?
The fix may be 1 or 2 lines of script, very minor in the SPFx. Without them all your customers would have a broken product.
By following the current guidelines, they will all have to manually install an update before they would see the fix.

Also, typo and spelling mistakes, also UI glitches are too common to pass QA in my experience. It happens, more that we would like to admit. If you say it never happens to you - you are either the best developer in the world or you don't publish products to production often :)

And again, each and every one of my customers currently love getting updates pushed to their servers without the need for them to manually install updates. I'm basing this on 5000+ real life customer feedback.

Like I said before, you don't like it, I get it. You don't have to use this methodology.
I respectfully disagree with every point you've made, and from my humble experience I think this change would lead to much better product release lifecycle.

What I don't get is, why would you be against having the option to customize it?

[waldekmastykarz]

@shaipetel I know the difference between the hash in the file name and the subresource integrity hash you can put on a script and I know that SPFx currently doesn't use it. What I mean is: some organizations want to ensure that once a particular file is deployed it's never change. They might have for example performed a code review of it and if you have a policy that allows changing existing files, then it's all for nothing because they will not be able to tell if the file is the same without manually verifying its contents/hash.

So, to you, it makes more sense to ask the developer to modify the CDN setting for the project every time they build a new version?

I think it's not up to developers to deploy solutions to production. Even though the target URL and the key to Azure storage are a part of the project, I don't think organizations will allow developers to deploy to production. Instead, both the URL and the access key should be passed as parameters when executing the task as a part of the release process which is definitely not done by developers.

That seems a bit too much to me. Especially if I have other resources I load from that same CDN location as the relative paths would always change.

Shouldn't these resources be a part of your solution anyway?

Like I said before, you don't like it, I get it.

It's not about liking it or not. I think it's simply a bad practice.

What I don't get is, why would you be against having the option to customize it?

As I mentioned previously: _I don't think it would be good to have this option, because it would allow people to circumvent the control that SPFx was designed to give to organizations. I understand that it would be convenient for developers but the data security and integrity is more important._

[shaipetel]

@waldekmastykarz Thanks for taking the time to comment and explain your position.

I'm sorry again, I'm really trying to understand your position but none of this makes any sense to me.

Verifying the file hash

Anyone using O365 already getting file content changes without their knowledge or consent.

The review process you are talking about will not work in O365 regardless of our discussion here. There is no way to stop it or to verify it. Revisions for files are being published constantly under the same name.

The protection for customers here is not to verify the hash of the file (against what? how would they know what it suppose to be?), but that the source is trusted and have a valid SSL certificate.

This practice you are talking about is in theory, I take it? Yet you insist it is very critical - do you actually know anyone who arbitrarily crawl script files loaded from a CDN (SharePoint, or a vendor) and verify their hash?
And even if they did - how do they know what the hash is suppose to be?
for Sp.js, Sp.core.js etc... all these files don't have the hash in their file name.
How do these customers work today if it is a huge security and data integrity risk for them, as you say?

I am sorry, but I am confused. I can't really understand what you are talking about. Either I am missing the point, or this just doesn't make any sense.

Publishing process CDN location

As for changing the CDN location - of course the developer needs to be aware. Not only aware, he needs to know the exact CDN drop location URL during build time, and if that changes - he will need to rebuild his package. He may not be the one putting the file on the CDN but he must know the location.
And if, as you suggest, that location should change per version - he should know and update his package prior to building it.

Not everyone are using automated build and deployment tasks. We do, partially, but I have to say the vast majority of people I know don't even understand it let alone use it in their release process.

In an ideal world, I would love to be able to redirect an existing web part from loading file A to loading file B without requiring the tenant admin to install an update.

Conclusion

Your opinion is clear, although not understood, and I respect it.
It is OK for you to think it is a bad practice to change the file content while keeping its name/url the same.

But you seem to feel very strongly against my opinion.
To me, this is a very reasonable request. For me to be able to control the file names of the build I make.

Furthermore, I can't recall any other platform that does not allow the developer to control the output file names it produces.

I don't understand your agenda in trying to shut this down. Seems like you are taking it personally for some reason.
Perhaps I didn't my my request clear to begin with?

The ask, in simple words

The make it clear, what I'm asking for here is nothing revolutionary.
I'm asking to be able to step into the build process and control the output file name of the bundled JS file, to change it to something that makes sense to me in my release process.
My given example would be: {webpart name}.{version}.js
But it could be anything the developer wanted.
The 2 problems are:

1. having the hash doesn't make sense to me, and turns my CDN into a dumpyard of files.
2. having the ".bundle" messes up other tools we are using to process JS files into our fastring/production CDNs.

So, I need to get rid of the hash and the bundle in the file name.

Hope this makes more sense, and less controversy.

[octogonz]

I agree that the hash is not a security mechanism. As noted above, it accomplishes two main goals:

1. Ensure that when the content changes, the URL changes: This avoids a problem where, after you deploy new scripts, the browser may accidentally load old scripts from its cache. It may even mix old and new assets together, which is a very confusing and frustrating thing to debug.

2. When redeploying unchanged content, keep the URL the same: You could solve # 1 by using a build number instead of a hash in your URL. However, that means every deployment will cause a cache miss for all assets. If you have lots of small files, or infrequent visitors, this can be a significant performance concern. By using a hash in your URL (instead of the build number), unchanged files will keep their old URLs, and there is no cache miss.

We want SPFx developers to have the freedom to name their bundles however they like. (This includes choosing worse performance, or even choosing unsafe cache behavior.) However, we need to be careful about introducing a setting that would inadvertently cause naive developers to end up with the frustrating cache problems I described above.

Long term, I'm thinking it would be something like either (a) a simple setting that says "I want build numbers instead of hashes in my URL", and/or (b) a gulp event hook that allows you to write code that calculates the filenames yourself (in which case Microsoft isn't responsible for any caching problems it causes).

[iclanton]

having the ".bundle" messes up other tools we are using to process JS files into our fastring/production CDNs

You can change the bundle name by editing your config/config.json file. If you post your file contents, I can point out what you need to change if you can't find it.

If you want to include your package.json version number in your base bundle's name, you can include a snippet of code that looks like this in your gulpfile.js:

'use strict';

const gulp = require('gulp');
const build = require('@microsoft/sp-build-web');

const path = require('path');
const packageJson = require('./package.json');

build.configureWebpack.mergeConfig({
  additionalConfiguration: (generatedConfiguration) => {

    const existingOutputName = generatedConfiguration.output.filename;
    const outputNameExt = path.extname(existingOutputName);
    const baseOutputName = path.basename(existingOutputName, outputNameExt);
    generatedConfiguration.output.filename = `${baseOutputName}_${packageJson.version}${outputNameExt}`; // This will name your bundle "<bundle-name>_<version>.js"

    return generatedConfiguration;
  }
});

build.initialize(gulp);

[shaipetel]

@pgonzal precisely. We manage our cache ourselves with expirations that corresponds to our builds and publish times so that's not a concern for us. It never was an issue.

That type of flexibility would be perfect, even if not default.

But as @iclanton mentioned (and I originally tried to do but didn't figure it out) this is probably possible by customizing gulpfile.js
I will I've this code snsippet a try and if it works that would be more than perfect.

Thanks everyone who took the time to comment. I will give it a go tomorrow and update here if that did the job or not.

Love SPFx so far, if this is doable in the gulpfile.js - it just goes to prove this framework delivers on its promise.

[octogonz]

Sounds good. Let us know the result.

[waldekmastykarz]

@shaipetel sorry to hear you think I'm taking it personally: I'm not. You think it's a good idea to have such flexibility, I think it's not. That's it, there's nothing more to it.

Like @pgonzal and @iclanton pointed out, it's totally supported by the toolchain. There are some consequences involved with using this approach and as long as you understand and accept them, there's nothing in your way of using it.

[shaipetel]

@waldekmastykarz no worries,

@pgonzal @iclanton editing config/config.json allows me to get rid of the bundle but not the hash.

gulpfile changes

I tried changing these (I used hard coded text for simplicity):
generatedConfiguration.output.filename = "test.js";
generatedConfiguration.output.chunkFilename = "test.js";
(without changing the chunkFilename gulp --ship doesn't produce any files in temp/deploy)

It replaces the file in the dist folder, not the production build output in the temp/deploy folder.

Why is that important? Since the gulp package-solution builds the package based on the files that are in temp/deploy folder.

I have so far tried at least 8 different ways to remove the hash from the file name with no luck.
It seems to be hard coded in the build process as far as I could investigate.

Any help would be greatly appreciated.

[shaipetel]

According to webpack config, all I need to do is remove the [chunkhash] token from the chunkFilename or the filename properties, but it seems while the guild is taking these settings to the dist folder, it is ignoring them and using some other configuration for the temp/deploy folder.

I'll update if I can figure it out. Let me know if you have a lead on where that could be.

[shaipetel]

So far, it seems CopyAssetsTask which is the task responsible to moving files into temp/deploy production build folder is calling _renameWithHash and hard-coded appends the hash to the file name.

I don't see anywhere a config value that can stop that from happening.

Its ignoring the webpack config and [chunckhash] token as far as I could tell.

Here is the hard coded lines that adds the hash:
getFilenameCallback = function (hash) { return entryName.toLowerCase() + "_" + hash + ".js"; };

getFilenameCallback = function (hash, originalFilename) {
  var locale = reverseEntrypointLookup_1[originalFilename];
  return entryName.toLowerCase() + "_" + locale + "_" + hash + ".js";
};

tasks.push(_this._renameWithHash(gulp.src(localizedFiles[locale]), function (hash) { return baseFilename + "_" + locale + "_" + hash + ".js"; }, function (filename) {
  return callbacks.forEach(function (callback) {
    return callback(locale, filename);
  });
}).pipe(gulp.dest(_this.taskConfig.deployCdnPath)));



md5-e3ff7e846d617346aa5f01ecacd1f815



tasks.push(_this._renameWithHash(gulp.src(externalModule.configExternalFullPath), function (hash) { return baseFilename_1.toLowerCase() + "_" + hash + ".js"; }, function (filename) {
  return externalModule.renameCallbacks.forEach(function (renameCallback) {
    return renameCallback(filename);
  });
})

I don't see how we can change it without a change in the task that would expose such a configuration setting.

[shaipetel]

Other way that does not work, FYI:
In gulpfile.js add a post build task

build.rig.addPostBuildTask(renameFilesProductionTask);

This executes before the bundle task started.

[iclanton]

The build tools don't use webpack's built-in hashing mechanism because debug build bundles don't have hashes in their filenames. As you discovered, the copy-assets task does the rename when it copies the files from the dist directory to the CDN deployment directory (in most cases temp/deploy).

The code snippet I posted yesterday should add the version number in the package.json to the produced filename, but it won't remove the hash. As @pgonzal mentioned, that's a new configuration option that we'll have to expose.

I just realized that I tested the code that I gave you for your gulpfile in a debug build but not in a production build, and it breaks copying the base bundle to the deployment directory. I apologize - that's my fault. It looks like it exposed a bug in the copy-assets task. I'll sync with the team and try to get an estimate on when we can get @pgonzal's suggested

I'm thinking it would be something like either (a) a simple setting that says "I want build numbers instead of hashes in my URL", and/or (b) a gulp event hook that allows you to write code that calculates the filenames yourself (in which case Microsoft isn't responsible for any caching problems it causes).

approach.

[shaipetel]

Thanks @iclanton
Meanwhile, I managed to hack this into place in a way that I'm sure won't survive updates. For now, anyone who wants to do it today can follow these instructions:
http://kwizcom.blogspot.ca/2017/08/controlling-spfx-bundle-file-name-in.html

Again, I know this is not the best way to do it, but at least this works for our current automation needs.

Please let me know if you think there is a better way...

here is the code again for convenience, for updates follow the blog. Put this before build.initialize(gulp); in your gulp file

const packageJson = require('./config/package-solution.json');
...
var original_renameWithHash = build.copyAssets._renameWithHash.bind(build.copyAssets);
build.copyAssets._renameWithHash = function (gulpStream, getFilename, filenameCallback) {
    var original_GetFileName = getFilename;
    getFilename = function (hash) { return original_GetFileName(hash).replace('_' + hash, '.' + packageJson.solution.version); };
    return original_renameWithHash(gulpStream, getFilename, filenameCallback);
};

[FranckyC]

Hi all,

Just for your information, from drop 1.3 (I didn't check earlier versions), if you want to completely remove hashes from generated files, you can now do something like this in your gulpfile.js:

build.copyAssets.taskConfig = { excludeHashFromFileNames: true, }

However, I didn't find a way to add the package version instead.

[shaipetel]

Nice, I will give it a try!
Add the version number - I guess we can edit the file name in the config for that, but I can keep using my code to automate it.

[spdevdocs]

We closed this issue as it had not activity within last 180 days. This is a generic process we have decided to perform for issues, which have not been explicitly marked still to be "work in progress" based on tags. We are performing this cleaning to make sure that old issues that have already been solved (but not closed) or are no longer relevant are cleaned out and make the issues more manageable. If this issue still valid, we would ask you to open a new issue and follow the guidance in the issue template related on the recommended location. We do apologize any inconveniences this might cause. Please do remember that issues in the issue lists are also messages for others in the community, so you can also check if you can assist on any of them. “Sharing is caring!”