Aspnetcore: Support serving of pre-GZIP encoded files

[aspnet-hello]

_From @DamianEdwards on Friday, July 25, 2014 11:42:51 AM_

Pre-GZIPping files is seemingly becoming more popular. This involves running a tool ahead of deployment that creates GZIPped copies of suitable files in the site, e.g. site.js => site.js.gzip. Then the file serving aspect of the web server will serve the GZIPped file when appropriate.

_Copied from original issue: aspnet/StaticFiles#7_

[aspnet-hello]

_From @tugberkugurlu on Friday, July 25, 2014 1:24:16 PM_

:thumbsup: for this one.

[aspnet-hello]

_From @nikmd23 on Wednesday, September 10, 2014 1:47:47 PM_

I'd also like to point out that this shouldn't be about "pre-gziping", but rather pre-compressing in general, no matter what the encoding format is.

For example, Microsoft and Google have both played with improved encoding schemes (peerdist and sdch). At one point Chrome even included sdch in its Accept-Encoding header. Of course these two examples are a bit crumby since neither caught on, but new compression techniques come up all the time.

What this means is, in reality, I might want to pre-compile a file into multiple formats and have the proper one selected based on the Accept-Encoding. My content directory could have several "gzip compatible" files encoded with zopfli (*.gz), the same files encoded in sdch (*.sdch) as well as the original uncompressed files for _really_ old browsers.

I think of this as being very analogous to keeping images in both *.jpg and *.webp formats, and serving the appropriate ones at the appropriate times.

[aspnet-hello]

_From @resnyanskiy on Monday, January 18, 2016 6:37:35 PM_

I found simple workaround based on URL redirect supported in HTTP:

class Startup
{
    private StaticFileOptions StaticFileOptions
    {
        get
        {
            return new StaticFileOptions
            {
                OnPrepareResponse = OnPrepareResponse
            };                
        }
    }

    private void OnPrepareResponse(StaticFileResponseContext context)
    {
        var file = context.File;
        var request = context.Context.Request;
        var response = context.Context.Response;

        if (file.Name.EndsWith(".gz"))
        {
            response.Headers[HeaderNames.ContentEncoding] = "gzip";
            return;
        }

        if (file.Name.IndexOf(".min.", StringComparison.OrdinalIgnoreCase) != -1)
        {
            var requestPath = request.Path.Value;
            var filePath = file.PhysicalPath;

            if (IsDevelopment)
            {
                if (File.Exists(filePath.Replace(".min.", ".")))
                {
                    response.StatusCode = (int)HttpStatusCode.TemporaryRedirect;
                    response.Headers[HeaderNames.Location] = requestPath.Replace(".min.", ".");
                }
            }
            else
            {
                var acceptEncoding = (string)request.Headers[HeaderNames.AcceptEncoding];
                if (acceptEncoding.IndexOf("gzip", StringComparison.OrdinalIgnoreCase) != -1)
                {
                    if (File.Exists(filePath + ".gz"))
                    {
                        response.StatusCode = (int)HttpStatusCode.MovedPermanently;
                        response.Headers[HeaderNames.Location] = requestPath + ".gz";
                    }
                }
            }
        }
    }

   public void Configure(IApplicationBuilder application)
   {
        application
            .UseDefaultFiles()
            .UseStaticFiles(StaticFileOptions)
    }
}

I used Wikipedia as reference. This approach also allows to use not minified files in development environment with no need to change links on client side.

I found that currect implementation of 'aspnet/StaticFiles' lacks of two feature:

1. Declarative setup of 'Content-Encoding' header. I suppose it can be implemented very easy - by adding second dictionary to FileExtensionContentTypeProvider.
2. Redirect support. Current design doesn't provide the ability to cancel invoke 'IHttpSendFileFeature.SendFileAsync/StreamCopyOperation.CopyToAsync'.

@davidfowl fyi.

[aspnet-hello]

_From @nikmd23 on Monday, January 18, 2016 7:30:14 PM_

In support of my comment above, both FireFox and Chrome will be shipping support for Brotli compression soon.

[aspnet-hello]

_From @jods4 on Monday, February 22, 2016 8:44:29 AM_

:+1:

Actually, even a good story for serving gzipped static files (with a cache, of course, since it's static compression) is currently lacking. I'm saying this in the "conventional", "IIS-kind" of way, as opposed to the build-time gzip generation.
Or did I miss something?

For those coming here from Google with dynamic content compression needs, you might try this gist:
https://gist.github.com/PinpointTownes/ac7059733afcf91ec319

[aspnet-hello]

_From @JohannesRudolph on Tuesday, July 5, 2016 1:33:54 AM_

Would it be beyond the scope of this to suggest that StaticFiles should optionally allow cached "on-demand" compression of files? (I.e. gzip on first request and put result to file system cache, serve gzip requests from there)

[aspnet-hello]

_From @twilliamsgsnetx on Friday, July 22, 2016 4:39:03 AM_

Really looking forward to being able to do this. I can go through all the trouble of bundling, minifying, tree shaking and gzipping to get things down to an absolute minimum size... but I can't serve those .js.gz files and it's problematic.

I'm also strugglin to get IIS to gzip itself as well, but that's for another area entirely.

[aspnet-hello]

_From @neyromant on Friday, July 29, 2016 10:26:24 PM_

https://github.com/aspnet/Home/issues/1584

https://github.com/AnderssonPeter/CompressedStaticFiles

[aspnet-hello]

_From @dfaivre on Wednesday, February 15, 2017 4:34:00 AM_

With Angular (2) creating *.gz files when doing --prod builds, it would be great to have this out of the box.

[aspnet-hello]

_From @RehanSaeed on Wednesday, February 15, 2017 4:36:34 AM_

Other related use cases are serving pre-brotli compressed (.br) files and also serving WebP (.webp) files instead of PNG/JPG/etc.

[aspnet-hello]

_From @Tratcher on Friday, February 17, 2017 5:54:44 AM_

@JohannesRudolph you should now be able to combine ResponseCaching, ResponseCompression, and StaticFiles to achieve dynamic compression and caching of static files. @JunTaoLuo this would be a good combo to test.

[aspnet-hello]

_From @Tratcher on Friday, February 17, 2017 5:59:33 AM_

Can folks clarify if they expect the original url to contain the compression extension (e.g. ".gz")? Some of the above samples do and some don't. I assume content negotiation based on accept headers would be the more general case.

[aspnet-hello]

_From @JohannesRudolph on Friday, February 17, 2017 6:03:22 AM_

Great! Content-negotiation is (from my experience) the far more often-used
and preferred method.

On Fri, Feb 17, 2017 at 2:59 PM, Chris R [e-mail] wrote:

Can folks clarify if they expect the original url to contain the
compression extension (e.g. ".gz")? Some of the above samples do and some
don't. I assume content negotiation based on accept headers would be the
more general case.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/aspnet/StaticFiles/issues/7#issuecomment-280656573,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAH8NyZXuqJehp3rBf7VkFTq_XulZ8gYks5rdafHgaJpZM4CQ_LY
.

[aspnet-hello]

_From @dfaivre on Monday, February 20, 2017 5:31:20 AM_

@Tratcher -- I would not expect the original URL to contain the extension.

Also, looks like the Angular CLI will no longer output the "*.gz" files, so @Tratcher's ResponseCaching and ResponseCompression solution may be sufficient in that case?

[aspnet-hello]

_From @firecube on Thursday, March 23, 2017 1:27:59 PM_

@Tratcher do you have an example of combining those three bits of middleware to suit this scenario?

Thank you.

[aspnet-hello]

_From @Tratcher on Thursday, March 23, 2017 1:29:33 PM_

@JunTaoLuo can you do a sample?

[aspnet-hello]

_From @firecube on Sunday, April 2, 2017 5:22:57 AM_

A small example would be really appreciated here if possible. Thanks in advance.

[aspnet-hello]

_From @JunTaoLuo on Sunday, April 2, 2017 3:59:35 PM_

You can take a look at the sample I have created at https://github.com/JunTaoLuo/MiddlewaresSample which uses ResponseCaching, ResponseCompression and StaticFiles to create, cache and serve different representations of the same resource.

Here's the sample output where I made 6 requests to http://localhost:5000/Foo.txt. The first was sent with no Accept-Encoding header, the second contained Accept-Encoding: gzip and the third had Accept-Encoding: custom. Then I made the first three requests again and checked that they were served from cache.

Hosting environment: Production
Content root path: C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 206.5195ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 23.3728ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[24]
      No cached response available for this request.
info: Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware[2]
      Sending file. Request path: '/Foo.txt'. Physical path: 'C:\gh\tp\MiddlewaresSample\MiddlewaresSample\MiddlewaresSample\wwwroot\Foo.txt'
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[26]
      The response has been cached.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 3.5304ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 7.703ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 1.4551ms 200 text/plain
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[1]
      Request starting HTTP/1.1 GET http://localhost:5000/Foo.txt
info: Microsoft.AspNetCore.ResponseCaching.ResponseCachingMiddleware[22]
      Serving response from cache.
info: Microsoft.AspNetCore.Hosting.Internal.WebHost[2]
      Request finished in 1.3005ms 200 text/plain

[aspnet-hello]

_From @Tratcher on Sunday, April 2, 2017 8:24:55 PM_

Hmm, no logs for the compression middleware...

[aspnet-hello]

_From @RudeySH on Thursday, October 12, 2017 3:58:05 AM_

@muratg why was this closed?

[aspnet-hello]

_From @joeaudette on Sunday, October 22, 2017 1:55:08 PM_

I came across this because I've started using webpack to pre-gzip my js and css.
@resnyanskiy your code set me on the right path, but I wanted to avoid the redirect.

The solution I came up with was implementing a custom IFileProvider based on the code from CompositeFileProvider.

I set it up like this:

app.UseStaticFiles(new StaticFileOptions()
{
    OnPrepareResponse = GzipMappingFileProvider.OnPrepareResponse,
    FileProvider = new GzipMappingFileProvider(
    loggerFactory,
    true,
    Environment.WebRootFileProvider
    )
});

it uses a convention, if the requested file ends with .min.js or .min.css it will look for the same file name with .gz on the end, and if found it returns that. Then later I got the idea to add logic to try to create the .gz file if it does not exist, and then return that on success else return the original file.

It seems to be working well, would appreciate any feedback on the implementation, found here:
https://github.com/joeaudette/cloudscribe/blob/master/src/cloudscribe.Web.Common/StaticFiles/GzipMappingFileProvider.cs

My solution is using standard .min.js and .min.css urls, but the .gz file is served, I'm not using .gz in my urls.

One known issue is that this solution is bypassing content negotiation, and just giving you gzip whether you like it or not, but really not a big issue in my view for real browsers.

[aspnet-hello]

_From @Tratcher on Sunday, October 22, 2017 6:59:05 PM_

That approach sounds like it would mess up the content-length and etag headers.

[aspnet-hello]

_From @joeaudette on Monday, October 23, 2017 4:35:28 AM_

@Tratcher could you elaborate on how that would get messed up? The IFileProvider is passing up the IFileInfo about the gz file so it would have the correct content length of the gz file, isn't that what it should have?

We still have an opportunity to tweak the headers in OnPrepareResponse if there is something messed up, but I'm trying to understand what would be messed up and why.

[aspnet-hello]

_From @Tratcher on Monday, October 23, 2017 6:48:18 AM_

Content-length and gzip are rarely used together because the implications are really confusing. I'll need to verify, but I think the content-length is supposed to be the uncompressed length rather than the compressed length.

StaticFiles also uses the length to calculate the etag, so your pre compressed file will have a different etag than the uncompressed version, even if the contents are the same.

StaticFiles also has built in support for range headers, which refer to offsets in the uncompressed file. This won't work with compressed files.

Managing pre compression in the file provider is inadequate, it needs to be built into StaticFiles to make the above scenarios work correctly (or at least be bypassed correctly).

[aspnet-hello]

_From @joeaudette on Monday, October 23, 2017 6:58:57 AM_

@Tratcher that makes sense for dynamic compression, but the issue here is pre-compressed static files and I think we would want content-length for any static file wouldn't we? and it should be used to calculate the etag I would think.

In my scenario the .gz file is created by the webpack build process, while my FileProvider can be configured to generate the .gz file that is a secondary concern, mainly I am serving static files that are already gzipped.

In my scenario without content negotiation the browser is only going to get the already compressed static file, the uncompressed file would not be returned unless the compressed file does not exist and could not be created. If the source file is modified newer than the compressed file I am regenerating the compressed file.

[aspnet-hello]

_From @joeaudette on Monday, October 23, 2017 7:12:19 AM_

@Tratcher so are you saying that for my pre-gzipped files I should remove the Content-Length and Accept-Range headers?

[aspnet-hello]

_From @Tratcher on Tuesday, October 24, 2017 10:16:24 AM_

Never mind on content-length, I re-checked the spec and verified that content-length and transfer-encoding don't mix, but content-encoding is OK.

I still think the Range requests will be broken though, those offsets should be for the un-compressed representation. So yes, at least remove the Accept-Ranges header. I haven't found any spec references to confirm this yet.

[aspnet-hello]

_From @joeaudette on Tuesday, October 24, 2017 1:24:20 PM_

@Tratcher thanks! I am by no means an expert, but I'm still a little doubtful that I should remove the Accept-Ranges:bytes header. Other samples I have found for pre-gzipped files seem to include that header.
I would think the offsets sent by the browser would mainly be to recover from partial downloads and get the rest of the file and that would be true even if the file is pre-gzipped.

I found this which seems to indicate chrome and firefox for example would store the file in its content-encoding format ie .gz and will send range requests for those files to complete a failed partial download.

https://lists.w3.org/Archives/Public/ietf-http-wg/2014AprJun/0112.html

[aspnet-hello]

_From @Tratcher on Tuesday, October 24, 2017 1:38:21 PM_

I expect a problem due to the differences between static and dynamic compression. With dynamic compression a range request would be processed against the un-compressed content and then the result would be compressed. With static compression the range request is processed against the compressed content. How is the client supposed to tell the difference? There's no indication in the response that you used static or dynamic compression, except that dynamic compression often uses chunked rather than content-length.

Maybe it doesn't matter so long as the behavior is consistent on a per resource basis.

[aspnet-hello]

_From @joeaudette on Tuesday, October 24, 2017 1:45:36 PM_

@Tratcher I found this on SO, which seems to indicate content-encoding is a property of the entity whereas transfer-encoding is a property of the message, so it would seem logical if dynamic compression used transfer-encoding and pre compressed static files would use content-encoding
https://stackoverflow.com/questions/11641923/transfer-encoding-gzip-vs-content-encoding-gzip

[aspnet-hello]

_From @Tratcher on Tuesday, October 24, 2017 1:52:30 PM_

The edit on that answer is really telling..
"Edit: That is, unless you want to serve gzipped content to clients that only understand Content-Encoding. Which, unfortunately, seems to be most of them. But be aware that you leave the realms of the spec"

[aspnet-hello]

_From @joeaudette on Tuesday, October 24, 2017 2:08:35 PM_

yes I saw that, you notice I said it would be logical, not that is how it is done, it sounds like in practice dynamic compression does it wrong, they should use transfer-encoding but they don't because of browsers :-D

but nevertheless I think it will be ok to keep the Accept-Ranges header
from my previous link I suspect it would work the same way with dynamic compression the offsets would be based on the compressed file not the decompressed because the dynamic compression does use content-encoding header so the client doesn't know it is dynamic. I don't think it will use offsets based on the the decompressed file, the previous link it mentions chrome and ff cache it compressed and make range requests based on the file, IE would cache the file decompressed but would only make range requests for uncompressed files, if I understood it correctly. It would seem weird if a browser used offsets that were not against the actual file it downloaded but a version transformed after download. It has to use the content-length of what it downloaded and calculate offsets based on how much of that it got already I would think.

[aspnet-hello]

_From @herecydev on Monday, December 4, 2017 1:30:06 AM_

Adding my implementation into the mix. Use case, client calls /foo.js and the WebRootPath has foo.js.gz or foo.js.br and prefers the compressed versions.

```c#
public class CompressionFileProvider : IFileProvider
{
private readonly IFileProvider _fileProvider;
private readonly IHttpContextAccessor _httpContextAccessor;
private readonly string _root;

    public CompressionFileProvider(IHostingEnvironment hostingEnvironment, IHttpContextAccessor httpContextAccessor)
    {
        _fileProvider = hostingEnvironment.WebRootFileProvider;
        _httpContextAccessor = httpContextAccessor;
        _root = hostingEnvironment.WebRootPath;
    }

    public IDirectoryContents GetDirectoryContents(string subpath)
        => _fileProvider.GetDirectoryContents(subpath);

    public IFileInfo GetFileInfo(string subpath)
    {
        if (_httpContextAccessor.HttpContext.Request.Headers.TryGetValue("Accept-Encoding", out var encodings))
        {

            if (encodings.Any(encoding => encoding.Contains("br")))
            {
                var compressedEncoding = _fileProvider.GetFileInfo(subpath + ".br");
                if (compressedEncoding.Exists)
                    return compressedEncoding;
            }

            if (encodings.Any(encoding => encoding.Contains("gzip")))
            {
                var compressedEncoding = _fileProvider.GetFileInfo(subpath + ".gz");
                if (compressedEncoding.Exists)
                    return compressedEncoding;
            }
        }

        return _fileProvider.GetFileInfo(subpath);
    }

    public IChangeToken Watch(string filter)
        => _fileProvider.Watch(filter);
}

public static class ApplicationBuilderExtensions
{
public static IApplicationBuilder UseCompressedStaticFiles(this IApplicationBuilder applicationBuilder, IHostingEnvironment hostingEnvironment, IHttpContextAccessor httpContextAccessor)
{
return applicationBuilder.UseStaticFiles(new StaticFileOptions
{
FileProvider = new CompressionFileProvider(hostingEnvironment, httpContextAccessor),
OnPrepareResponse = ctx =>
{
var headers = ctx.Context.Response.Headers;

                if (ctx.File.Name.EndsWith(".br"))
                    headers.Add("Content-Encoding", "br");
                else if (ctx.File.Name.EndsWith(".gz"))
                    headers.Add("Content-Encoding", "gzip");
            }
        });
    }
}

```

[RehanSaeed]

I'm trying to use the ResponseCompression middleware after the StaticFiles middleware to server Brotli (.br) or GZIP (.gz) encoded files if available.

However, the ResponseCompression keeps GZIP encoding files that are already Brotli encoded. It also ends up double encoding when I serve a static GZIP file too. It seems it's not checking for already encoded files and the two pieces of middleware are clashing.

[Tratcher]

StaticFiles doesn't set the content-encoding headers, you have to do it yourself in the OnPrepareResponse event. ResponseCompression can't tell it's compressed unless you set that header.

[deedubb]

As requested by @DamianEdwards I am adding a note to remind him/others of the chain resulting from https://twitter.com/DamianEdwards/status/980814294756425728 which brings up the CRIME and BREACH exploits in HTTPS when using gzip compression on dynamic pages (any page containing user content).
As per the last comment in the chain this issue should be reviewed for 2.2.

In the meantime: because of these security issues the official stance is disable compression for HTTPS by default. In my opinion, if you know you only put user-modified variables in text/html pages you could use dynamic compression and exclude HTML pages based on mime type. "As-is" rules apply...

services.AddResponseCompression(options => {
    options.EnableForHttps = true;
    /* Includued defaults: "text/plain", "text/css", "ap­pli­ca­tion/javascript", 
            "text/html", "ap­pli­ca­tion/xml", "text/xml", "ap­pli­ca­tion/json", "text/json" */
    options.MimeTypes = ResponseCompressionDefaults.MimeTypes
        .Where(mime => !mime.Contains("text/html"))
        .Concat(new[] {
            "application/font-woff",
            "font/woff2"
        });
});

/cc @blowdart

[blowdart]

This could be enhanced by supporting compression in the static files middleware.

[phatcher]

@deedubb Are we not protected as the CRSF token is randomized? (see https://blog.qualys.com/ssllabs/2013/08/07/defending-against-the-breach-attack)

[DamianEdwards]

I think the eventual fix here would be to introduce a new request feature that the static file middleware can set when serving files that indicates the response is from static content, thus allowing the compression middleware to safely compress responses only when the feature indicates it was static in nature. Of course it's still possible for another middleware in-between to change the content and introduce issues again. If we care enough about that we could have the feature actually poison the response headers and body (e.g. via wrapping) such that they can't be modified. Also, technically it isn't about what's static but what contains user manipulable content, so the feature should likely be built around that premise instead, allowing any response (including dynamic responses from things like MVC actions) to declare that they're free from user manipulation and thus safe to compress over HTTPS.