Umbraco-cms: Upgrade to 8.4.0 causes 503 errors on Authorize Upgrade page

[cpevans]

We just upgraded a project to v8.4.0 and ran the install locally.

This worked fine on our development instance so we deployed it to our pre-production instance on Azure. Once this was deployed, all requests to the site are being redirected as usual to /umbraco/AuthorizeUpgrade?redir=xxx but suddenly the site is returning a 503 error. We are unable to log in to authorize the upgrade as any attempt to get to the login screen is redirected to the authorize upgrade path, and returns a 503.

We tried several different iterations by rolling the deployment back to our previous working version, logging into umbraco first before performing the 8.4 deployment, setting compilation debug to be true, but nothing has resolved the issue and we constantly get 503 on our pre-production instance.

We are not sure why the 503 errors were not present on our development instance when running the upgrade there.

Note this may be related to this issue : https://github.com/umbraco/Umbraco-CMS/issues/6334
and this update: https://github.com/umbraco/Umbraco-CMS/pull/6509

Umbraco version

I am seeing this issue on Umbraco version: 8.4.0

---

_This item has been added to our backlog AB#4398_

[cpevans]

This is what we see when running the upgraded version. Any attempt to navigate to any URL including the /umbraco login url gets redirected to this page, with the same 503 result.

[clausjensen]

I've added a comment on the other issue where you commented, but actually lets try to keep it in this one since the other one is closed.

It would be great to have some more info on this when you figure out more about what is causing it to happen.

[ronaldbarendse]

@cpevans Is the trySkipIisCustomErrors Umbraco setting set to the default true? If not, IIS shows it own error message, which seems to be the case (and custom 404 errors will probably also not work):

https://github.com/umbraco/Umbraco-CMS/blob/853087a75044b814df458457dc9a1f778cc89749/src/Umbraco.Web.UI/config/umbracoSettings.config#L77-L82

[cpevans]

We figured out the issue - it was because we had a custom httpErrors tag inside system.webServer on our production environment web.config file, which must have caused IIS to take over when the 503 code was returned (as we weren't explicitly overriding a 503 result)

<httpErrors existingResponse="Replace" errorMode="Custom">

Removing this httpErrors tag temporarily was enough to allow the site to load the Authorize Upgrade page and perform the migration.

[ronaldbarendse]

Using existingResponse="Replace" does indeed make IIS ignore the custom content for error codes, even when trySkipIisCustomErrors is set...

[ivo-centric]

We ran into this issue too after upgrading from 8.2.0 to 8.4.0 and having a custom httpErrors section in place. Removing this httpErrors section temporarily during a possible upgrade is not very convenient. Is it possible to make this new behaviour optional?

[ronaldbarendse]

@ivo-centric There's no built-in switch to turn this off/change the returned status code. As you're explicitly configuring IIS to ignore custom content on error HTTP status codes (>=400) with existingResponse="Replace", this will cause problems with _any_ error page (returned by Umbraco/ASP.NET).

So, using that configuration, the Umbraco 404s found with the IContentLastChanceFinder probably also don't work...

Maybe you can try to remove the custom 503 error by adding this to your configuration:

<httpErrors errorMode="Custom" existingResponse="Replace">
  <remove statusCode="503" />
  ...
</httpErrors>

[ivo-centric]

@ronaldbarendse Thanks for the answering and the suggestion to remove the statusCode="503", but unfortunately this doesn't work for me. I added the <httpErrors errorMode="Custom" existingResponse="Replace"> section to implement custom 404 (and Error/Exception) handling, because the standard Umbraco 404 handling doesn't support multidomain/multilangage very conveniently. To do this several articles suggest to use the httpErrors configuration and adding pages to specific http status codes.
For people that are in the same situation I solved it by using <httpErrors existingResponse="PassThrough" /> and added aIHttpModule that catches the Application_Error (inpired by ErrorHandlingModule.cs on https://www.marathonus.com/about/blog/error-handling-in-umbraco-7/) and the notfoundhandler on https://our.umbraco.com/documentation/reference/routing/request-pipeline/icontentfinder#notfoundhandlers to handle the multidomain stuff.
With this implementation I do not have the 503 Service Unavailable error page during an upgrade.

[nul800sebastiaan]

Ouch.. It's always difficult to predict all the things people do with their configurations, but we can definitely help here by always expecting the "correct" handling in the /umbraco folder. For backoffice routes, we rely on default behaviors to make the backoffice run smoothly.

An easier fix (without needing an IHttpModule) is to update this part of your web.config:

    <location path="umbraco">
        <system.webServer>
            <urlCompression doStaticCompression="false" doDynamicCompression="false" dynamicCompressionBeforeCache="false" />
        </system.webServer>
    </location>

To reset the <httpErrrors> element:

    <location path="umbraco">
        <system.webServer>
            <urlCompression doStaticCompression="false" doDynamicCompression="false" dynamicCompressionBeforeCache="false" />
            <httpErrors errorMode="Custom" existingResponse="PassThrough" />
        </system.webServer>
    </location>

A PR to do this automatically on upgrade is here: https://github.com/umbraco/Umbraco-CMS/pull/7439

[jaandrews]

Sebastiaan's suggestion helped me got me past the authorization step, but I ran into the error at the install step after that, as the path starts with "install" and not "umbraco." I got around it by commenting out the original httpErrors section, but I think doing the same thing he suggested for the "install" path would also work. Something like

<location path="install">
    <system.webServer>
        <httpErrors errorMode="Custom" existingResponse="PassThrough" />
    </system.webServer>
</location>

[nul800sebastiaan]

Ah yeah of course.. I added the install path to the <location> collections now as well, though I personally didn't have problems with it, I can see why it might become a problem with certain configurations. It can't hurt to also reset to the default httpErrors and disable urlCompression for that path as well.

That said, I would love to know what kind of errors you got on the install path?

All of these paths expect a certain configuration to be true and we never anticipated people overriding it. We always test those paths with the default config (it's also impossible to test with a million other config combos) so we don't see errors like this.
These are not paths part of your frontend so it doesn't affect the rest of your site, just the Umbraco-specific locations.

[ronaldbarendse]

Hmm, so we are now adding additional configuration to Umbraco to undo unsupported (or just plainly wrong) configurations?

If users set existingResponse to Replace, they're doing that for a reason and yes, that will break a lot of things: the upgrader returning a 503 response, but also the IContentLastChanceFinder returning a 404 and of course all other pages that return an error response! But they explicitly configured to replace the response with the custom, generic error page, instead of using the default Auto value and set TrySkipIisCustomErrors on the response.

Having Umbraco automatically update this setting back to the default Auto might be a nice thing to do, but I'm not sure this is the correct way to 'help' users. It's probably better to write good documentation and add this to the common pitfalls.

[nul800sebastiaan]

It's a friendly and helpful thing to do, absolutely. Nobody (except for a few people) reads the documentation and it is completely undiscoverable why people would get these errors. I actually wish we hadn't changed the responses headers since there is no obvious benefit and we are making it harder for people to upgrade.

unsupported (or just plainly wrong) configurations

People have reasons, some of them really good. If we can help them avoid getting into trouble with the Umbraco side of things then that is something we take on.

I think this has been said before: we cater Umbraco to people from all walks of life and they are not necessarily seasoned developers. Many people

[nul800sebastiaan]

Whoops, hit the wrong button. I was going to say: many people might read something about httpErrors on Stack Overflow and apply it without knowing the full implications. And that's fine, we like to help them get it right for the things that we control.

So let's keep an open mind, learn from what a diverse crowd of people are actually trying to achieve and see how we can make it as easy as possible for them if we can. Sometimes we can't or the effort is too large to help. But simple things like this are helpful, non-intrusive and actually set people up for success. 👍

[ronaldbarendse]

I totally agree on everything you're saying, but just keep in mind that setting existingResponse="Replace" will always replace the 404 page returned from the IContentLastChanceFinder, as that can be on any URL (not only /umbraco or /App_Plugins that have this reset to Auto after PR #7439 is merged).

Umbraco already correctly sets the TrySkipIisCustomErrors flag to indicate to IIS to use the existing response and not replace it with a custom/generic one configured in the httpErrors section. Adding this section to the default web.config with existingResponse="Auto" and a message explaining the things that will break when this is changed still seems like the most helpful, non-intrusive thing to do.

[ronaldbarendse]

BTW, the only reason for touching the httpErrors in the first place is because Umbraco does not process all requests and thus does not return a nice 404 error page for URLs containing a dot/file extension.

[jaandrews]

I personally use it to create custom 500 error pages.

[ronaldbarendse]

We also use the httpErrors section with a custom Umbraco route to display 404 and 500 pages, besides the IContentLastChanceFinder (for requests coming through the normal Umbraco request pipeline). This might be quite advanced for some, but it works great; even to the point I might consider creating a PR to add this to Umbraco core (any thoughts?):

<httpErrors errorMode="Custom">
    <remove statusCode="404" />
    <error statusCode="404" path="/umbraco/error" responseMode="ExecuteURL" />
    <remove statusCode="500" />
    <error statusCode="500" path="/umbraco/error" responseMode="ExecuteURL" />
</httpErrors>

And in a component we add (just after the default Umbraco route, so it's matched before the generic umbraco/{action}/{id} route - although you can also just change the route path 😅):

```c#
// Map error route (after default Umbraco route)
var errorRoute = RouteTable.Routes.MapUmbracoRoute("Umbraco_error", "umbraco/error", new
{
controller = "RenderMvc"
}, new UmbracoErrorNodeRouteHandler(/* This is a custom UmbracoVirtualNodeRouteHandler that finds the correct error page */));

var errorRouteIndex = 0;
if (RouteTable.Routes["Umbraco_default"] is RouteBase defaultRoute &&
RouteTable.Routes.IndexOf(defaultRoute) is var defaultRouteIndex)
{
errorRouteIndex = defaultRouteIndex + 1;
}
RouteTable.Routes.Remove(errorRoute);
RouteTable.Routes.Insert(errorRouteIndex, errorRoute);
```

IIS will append a query string to the executed error path/URL in the form of ?404;https://example.com/bad-url-not-processed-by-umbraco.txt, so you can use that to find the correct page based on the error code and domain/URL path (that's basically the only thing UmbracoErrorNodeRouteHandler does). As this URL is added by IIS, it might be an internal/proxy URL (as is the case on Umbraco Cloud), so you might need to use the current request URL to find a matching domain...

Using this approach, you can have nice 404 pages for URLs containing a dot (not processed by Umbraco) and 500 pages for errors on a specific page (Umbraco still has to be able to render this page though). Oh, and no need to set existingResponse to anything other than the default Auto! 🎉

[shearer3000]

just hit this in 8.6. thanks for workaround suggestion until 8.7 👍

[ronaldbarendse]

@shearer3000 There is no workaround for this issue, only advice not to change the default configuration value for system.webServer\httpErrors@existingResponse (which should be left to Auto, as a lot more won't work correctly when set to Replace). PR #7439 just makes sure this is enforced for Umbraco URLs (\Umbraco and \App_Plugins) by automatically updating the configuration when updating (using NuGet).

[AlexanderFalk]

_This comment intends to help those, who are trying to install Umbraco from scratch, and no other solutions have worked (v8.5.4)._

Based on @ronaldbarendse answer by deleting the , I can also confirm this solves an issue with setting up a new Umbraco installation.
I have been fighting for two days, struggling to figure out, why I kept getting the result page: "Service Unavailable" on the installation redirect. No errors from either IIS, Umbraco itself, or the Event Logs in Windows spotted this.

I was trying to setup a new installation in a new environment, using our repository from Azure DevOps, where the Web.config is included. The config was stripped from connectionStrings and ConfigurationStatus. The URLRedirect module was installed in IIS (and everything else). .NET v4.5 and all that was installed. File permissions were in place and there were no other indications of misconfigurations. But in the Web.config, the was added (since we have made our own custom page, which is imported with uSync). After deleting the property and its configurations, I was able to setup a fresh installation of Umbraco.

I would suggest a check which ignores when a fresh installation is made.

I hope someone with the same issues as I have had, stumbles on this comment, and it saves a ton of wasted time.

[ronaldbarendse]

As per MSDN documentation, if you get the IIS error page, either existingResponse is set to Replace or it's still on the default Auto value, but Umbraco doesn't set TrySkipIisCustomErrors = true (see umbracoSettings.config).

Just a reminder: @nul800sebastiaan PR https://github.com/umbraco/Umbraco-CMS/pull/7439 is still not merged 😉