Imageprocessor: Potential memory issue

[mortenbock]

Using:

<package id="ImageProcessor" version="2.5.4" targetFramework="net451" />
<package id="ImageProcessor.Web" version="4.8.5" targetFramework="net451" />

This is a longshot, but I have an Azure VM hosting my umbraco site, and the VM has started BSOD'ing spontaneously. Looking at the event log, it started after i deployed an update that basically just updated Image Processor from

<package id="ImageProcessor" version="1.9.5.0" targetFramework="net451" />
<package id="ImageProcessor.Web" version="3.3.0.0" targetFramework="net451" />

The eventlog shows this message after the BSOD:

The computer has rebooted from a bugcheck. The bugcheck was: 0x00000050 (0xffffbc0b351661c0, 0x0000000000000000, 0xfffff800790b3f7f, 0x0000000000000000). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 463bfd02-eef3-450e-a5e8-02e5d8506bb5.

That refers to this: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0x50--page-fault-in-nonpaged-area

I have no other evidence that this i caused by ImageProcessor, but just wanted to log the issue here in case it rings a bell for anyone else.

I will update the issue if I confirm/exclude IP as the cause.

[JimBobSquarePants]

Long shot indeed...

I can't see how that would be ImageProcessor at all. If it was somehow doing this to you it would surely be doing it to everyone?

There could be a myriad of driver, software updates happening behind the scenes. Have machine names changed?

[mortenbock]

Them only other thing that changed was the fcnmode. I think I will try and do a code rollback, and see if it stops, and then do tiny changes to see when it reappears.

[dlemstra]

Are you also using the ImageProcessor.Web.PostProcessor?

[mortenbock]

Not using the postprocessor. But I am placing the cache folder outside the web root. I think that is the only non standard setting I use.

[mortenbock]

There is definitely something strange going on. I reverted to my previous deployment, and cpu usage went back down to 10% from 50%.

Will get back when Azure support has completed their analysis.

[JimBobSquarePants]

Previous versions didn't clear the cache properly. The increased cpu usage will be from new files being cached and the cache cleaner running on a background thread, that will drop down once the cache starts filling up. Admittedly, I wish I'd never built that feature.

You can turn it off though

http://imageprocessor.org/imageprocessor-web/configuration/#cacheconfig

[mortenbock]

Just an update here. Since reverting to the old code version, I have not had the "bugcheck" BSOD.

I'm awaiting a response from azure support, who are analyzing the memory dump files. So far they determined that HTTP.sys caused the crash, so it feels related to the website somehow.

Will update here as I learn more.

[JimBobSquarePants]

Thanks, let me know how you get on.

[mortenbock]

Ok, so here's the daily update.

This morning I tried redploying the updated code, and instantly I crashed the server (twice).

I reverted, and no crashes since this morning.

So I've set out the following plan to narrow down the issue, while Microsoft are looking at the memory dumps.

TrimCache=false in all the below scenarios.

I will deploy one at a time, and let them run for a while to determine the effect.

1: Do a straight upgrade of IP. Just new dlls, but still cache inside the webroot
2: Move the cache out of the web root, and clear the old cache folder
3: Set fcnMode=single

I just deployed step 1, and so far it is not crashing. I will update here tomorrow when it's been running for a while.

[JimBobSquarePants]

@mortenbock This is indeed interesting.

Almost everything in ImageSharp uses fully managed memory. There are a few places though, FastBitmap and in the quantization code so if there were to be an issue it would be there.

Since you have already been running the updated code 1. without crashing (And thousands have been using it also) I'm confident that the unmanaged code in those locations is not the culprit.

You should be running 3. already, the memory differences are dramatic.

[mortenbock]

I think you are right in suspecting that step 3. will be the culprit. I should add that this is an Umbraco 6.2.6 site, and I have my media items in a virtual directory, which cancels out a lot of the benefits of the fcnmode as far as I can read, since that virtual directory falls back to the default mode for some reason. I have a LOT of images.

[mortenbock]

Ok, so I just went to step 2, and instant BSOD when the browser was trying to load all the images on the home page.

I have sent the new memory dumps to Microsoft to see if they can tell what causes this.

[JimBobSquarePants]

That... is very interesting!

All caching outside of the root does from my perspective is checks for and saves the file in a different location. I wonder whether we've discovered an IIS bug?

[mortenbock]

Maybe it could also be the combination of the original file being in a virtual directory, and then caching outside the web root. I will do some testing on my dev server tonight to see if I can reproduce on a separate environment.

[JimBobSquarePants]

That would be really useful. I'm certain it's not an ImageProcessor specific issue now, rather something underlying, but I'm really curious to see what it is.

[mortenbock]

Time for the daily update :)

I tried to reproduce the issue on my dev machine, but I could not. But my dev machine is also much more powerful than the server.

I tried to narrow down the exact scenario that causes the crash, and it seems to be either load or concurrency related.

I was able to fetch source html on the site, and static images. I was also able to fetch all the images from the home page individually. But when I load the whole page in the browser, then the server crashes.

My theory so far is that it is either some issue with concurrent requests, or it could be related to the fact that the site runs https and HTTP/2, so maybe there is something going on when fetching multiple images over the same socket connection?

Next step will be to try and clone the prod azure server, and see if I can make that crash, giving me some more room to investigate on a separate environment.

[JimBobSquarePants]

Thanks for the updates; sorry I can't be of more use. Any questions just ask.

[mortenbock]

News from MS. They seem to have identified the issue as a “race-condition in http driver”. They are working on a fix.

[JimBobSquarePants]

Righty-ho then! I'm glad you're getting somewhere.

[JimBobSquarePants]

I'm gonna close this following our offline conversations since we now know it to be a server issue.

[mortenbock]

Zombie update here :)

I received a private fix from MS, and have tested it, and it resolves the issue. It should be released in one of the cumulative updates. Will note it here when I know the KBxxx number.

So just if anyone else happens to get here from searching, there is a fix on the way for this.

Temporary mitigation is to disable http2 on the server.