Umbraco-cms: Return 503 status code when in upgrade mode

[ronaldbarendse]

When a new Umbraco version is deployed (to a live website) and the database needs to be upgraded, all requests are redirected to:

• 302 Found, redirect to: /install/?redir=true&url=
• 302 Found, redirect to: /umbraco/AuthorizeUpgrade?redir=
• 200 OK, login screen to authorize/accept the upgrade

This is bad in multiple ways:

• Search engines might index this page, removing the actual content from the index and causing duplicate content (as this page is returned for all requests, albeit via a temporary redirect);
• The first responses are redirects, which might confuse search engines or at least might have a negative impact;
• Status monitoring software (doing HTTP checks) will think everything is OK (which it clearly isn't).

Reproduction

Bug summary

The upgrade screen returns a 200 OK response, instead of a 503 Service Temporarily Unavailable.

Specifics

Umbraco 7:
https://github.com/umbraco/Umbraco-CMS/blob/54a2aa00a78caa4e6fe7b3b3cb9ff418fd1f408d/src/Umbraco.Web/Install/Controllers/InstallController.cs#L37-L62
https://github.com/umbraco/Umbraco-CMS/blob/54a2aa00a78caa4e6fe7b3b3cb9ff418fd1f408d/src/Umbraco.Web/Editors/BackOfficeController.cs#L145-L158

Umbraco 8:
https://github.com/umbraco/Umbraco-CMS/blob/cea8753ff28a3b6c695ffd10e183c9f4e6fb5e45/src/Umbraco.Web/Install/Controllers/InstallController.cs#L37-L60
https://github.com/umbraco/Umbraco-CMS/blob/12dff9bca31aa1486aa418a0d8b66d42154896c9/src/Umbraco.Web/Editors/BackOfficeController.cs#L142-L155

Steps to reproduce

Make sure the installation is in a upgrade state (e.g. assemblies/Umbraco configuration status does not match migrations in database).

Expected result

Return a 503 Service Temporarily Unavailable status code, as is advised by Google: https://webmasters.googleblog.com/2011/01/how-to-deal-with-planned-site-downtime.html.

Actual result

Return a 200 OK status code.

It would be even better if the authorization/acceptence of the upgrade could be done some other way, so this page isn't shown at all. This was already discussed on https://issues.umbraco.org/issue/U4-11520 and https://github.com/umbraco/Umbraco-CMS/pull/2789. Upgrading the site locally and then just deploying a working copy to production should then just automatically upgrade the database (as all other changes are already applied, e.g. updating the Umbraco configuration status, ClientDependency version, etc.).

[ronaldbarendse]

BTW: The InstallController seems to be updating the ClientDependency version and clearing temporary files _on every request_, which is probably quite overkill...

[nul800sebastiaan]

Just as a note that this is a very theoretical "problem" you're pointing out here and quite the edge case that a crawler would index your site in the exact minute that an upgrade is being performed. But we're happy to look at a PR that updates the status codes, without confusing browsers.

As for the upgrade process (U4-11520) - that is still on our list to overhaul in the future but requires a lot more work.

CDF cache should probably only need clearing in the first step.

[umbrabot]

Hi @ronaldbarendse,

We're writing to let you know that we've added the Up For Grabs label to your issue. We feel that this issue is ideal to flag for a community member to work on it. Once flagged here, folk looking for issues to work on will know to look at yours. Of course, please feel free work on this yourself ;-). If there are any changes to this status, we'll be sure to let you know.

For more information about issues and states, have a look at this blog post

Thanks muchly, from your friendly PR team bot :-)

[Shazwazza]

Gonna work on this as part of the hackathon I'm at now

[nul800sebastiaan]

Fixed in https://github.com/umbraco/Umbraco-CMS/issues/6334

[ronaldbarendse]

@Shazwazza Great work!

We could also return a 503 status code when the noNodes.aspx splash is returned (not sure if this page already includes a robots noindex meta tag). Umbraco Deploy overwrites this page with custom restore logic, so that should also include the custom status code :+1:

[clausjensen]

Please see: https://github.com/umbraco/Umbraco-CMS/issues/6754

[cpevans]

Not sure if this is the cause of the issue we're facing, but it seems related.

We just upgraded a project to v8.4.0 and ran the install locally.

This worked fine on our development instance so we deployed it to our pre-production instance on Azure. Once this was deployed, all requests to the site are being redirected as usual to /umbraco/AuthorizeUpgrade?redir=xxx but suddenly the site is returning a 503 error. We are unable to log in to authorize the upgrade as any attempt to get to the login screen is redirected to the authorize upgrade path, and returns a 503.

We tried several different iterations by rolling the deployment back to our previous working version, logging into umbraco first before performing the 8.4 deployment, setting compilation debug to be true, but nothing has resolved the issue and we constantly get 503 on our pre-production instance.

I'm very concerned that this may cause catastrophic errors for people if they deploy to their live websites and get caught in this 503 error issue. We are lucky that we are doing it on a site that has not yet been launched, and that we have azure devops to allow us to quickly roll back to previous working versions.

I'm not sure why it works on our development instances without the 503 issue (one of which is also on Azure and matches the setup of pre-production), we're trying to figure that out now but I thought I'd reach out to see if anyone here had knowledge of what exactly this change has done.

Perhaps the community needs to be notified of this potential breaking change, as every single upgrade experience ever done in Umbraco before now has returned the login screen for authorizing the upgrade; suddenly now people might get 503 errors and think their hosting is broken, and have significant downtime for their sites.

[clausjensen]

Hi @cpevans

I can't really replicate this upgrading from both 8.3.0 and 8.2.0 to 8.4.0. Tried a few times and it worked flawlessly. I also find it a bit strange that you had no problems upgrading your development environments, and this only shows up on your live site - which you claim is set up exactly as your development sites?

We will of course mention this as a breaking change if it turns out it is, but as of yet - we haven't heard anyone else having this problem.

It would be great to have some more info on this when you figure out more about what is causing it to happen.

[DaveYognaught]

So hey. This issue seems very related to mine as well (if not exact)

This Authorize Upgrade page shows up out of the blue at 2am, cuts off ALL TRAFFIC for major clients who rely on their site being live for income for upwards of 10+ hours and to top it all off we don't even want the update. Mayhaps this discussion is in regards to manual upgrades but i've only ever had it happen out of the blue for seemingly no reason and it's a HUGE problem for me.

Just as a note that this is a very theoretical "problem" you're pointing out here and quite the edge case that a crawler would index your site in the exact minute that an upgrade is being performed.

This isn't true in the slightest. The upgrade isn't being performed yet, that's why we're getting redirected to a log-in page to authorise it. Developers, site owners, aren't going to be there to approve the upgrade the exact moment it's requested either. So it's not a matter of a "minute". It might be hours. Worst case scenario, it's days. Yes, it is a "problem". You cannot in good conscience tell me that it'll be authorised instantly and the whole process from the moment of redirect to live is finished within the minute. As far as i'm aware, we're not even notified of it needing authorised in the first place.

Status monitoring software (doing HTTP checks) will think everything is OK (which it clearly isn't).

Hence why this is also a problem as originally stated. The entire site is down (not technically, but from their standpoint it may as well be) make no mistake, and Umbraco is the reason why. Our clients stop making money, it's literally worst case scenario, all hands on deck for them and for us when we finally get the Ticket in.

When a new Umbraco version is (sic: automatically) deployed (to a live website) and the database needs to be upgraded, all requests are redirected.

This is a pretty huge red flag for me. I'm surprised there's not more alarm at ALL TRAFFIC being redirected to your sites admin login page on a LIVE website until an admin notices and then authorises an upgrade. (Assuming it's actually wanted in the first place)

If that isn't a "problem" to you I'm concerned. That's not how updates should be getting delivered. Ever.

We don't want to upgrade because we don't know what'll break in custom sites. It's not essential as relaunching the site will prevent the redirects for that update from happening again. So what is it that's forcing the redirects to all traffic to upgrade in the first place?

We have a lot of Umbraco Clients, it'd be nice to know for the future how to prevent this happening. Can we prevent the redirects?

This would perhaps be the 3rd or 4th time in the last 2 months.

[clausjensen]

I'm not really sure I understand the issue here @RyanWilson-byte

• Are you seeing this problem on Umbraco Cloud because of automatic upgrades?
  This should be handled so I would be very interested to know if it isn't!

• If this is not Umbraco Cloud hosted - are you running automatic upgrades on a live site at 2AM when there's no developers doing it? .. doing an unattended upgrade of a live site is something we would NEVER recommend doing.

Also .. when you actually reach the _authorize upgrade page_ this indicates that the files for Umbraco have already been upgraded and it needs to complete the last steps of the upgrade process (such as upgrading the database schema) for the site to actually be in a complete working state. Umbraco is not necessarily at this point in a state where it can actually render any content.

Please let me know if I'm misunderstanding something here, but so far I fail to see exactly how this issue occurs unless you are doing something that you really should not be doing.

edit: so just to be clear. "Upgrading" is a two step process. You upgrade files in your site - and you authorize the upgrade (in those files) to run. You cannot just do one of the steps and expect your site to still work.

[DaveYognaught]

Hey,

We don't use Umbraco Cloud.

We absolutely should not have automatic upgrades enabled. That was my first
thought as well but all sites built by us have it disabled by default. Our
clients do have limited access to their own sites but they shouldn't be
able to update it, it was our initial thought that perhaps they triggered
it somehow but seeing as the logs started at 2am we ruled that out. Today,
we kinda just fixed it and then moved; Tomorrow I'll need to dig through
those logs again in more detail and share them.

The site in question from today was Umbraco 7.3.8. Though this has happened
a few times on various client sites on, likely, different versions.

I'm also curious, we have never accepted / authorised the upgrade page when
we get it. We just hop on the server and restart the site and it stops
redirecting. It makes me curious what happens to those files that have
already been updated, since as you said, should be in a state it can't
actually render content. I was under the assumption it wouldn't start
upgrade unless authorised, but seeing as we haven't seen conflicts, I guess
upgraded files are reverted?

On Tue, 28 Jan 2020 at 15:35, Claus [e-mail] wrote:

I'm not really sure I understand the issue here @RyanWilson-byte
https://github.com/RyanWilson-byte

-

Are you seeing this problem on Umbraco Cloud because of automatic
upgrades?
This should be handled so I would be very interested to know if it
isn't!
-

If this is not Umbraco Cloud hosted - are you running automatic
upgrades on a live site at 2AM when there's no developers doing it? ..
doing an unattended upgrade of a live site is something we would NEVER
recommend doing.

Also .. when you actually reach the authorize upgrade page this
indicates that the files for Umbraco have already been upgraded and it
needs to complete the last steps of the upgrade process (such as upgrading
the database schema) for the site to actually be in a complete working
state. Umbraco is not necessarily at this point in a state where it can
actually render any content.

Please let me know if I'm misunderstanding something here, but so far I
fail to see exactly how this issue occurs unless you are doing something
that you really should not be doing.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/umbraco/Umbraco-CMS/issues/6334?email_source=notifications&email_token=AN7E3SR37LJ7ESVB4NUQGCLRABGCNA5CNFSM4IWDMKRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKDXTMQ#issuecomment-579303858,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AN7E3STV574GCIJ5UJLILW3RABGCNANCNFSM4IWDMKRA
.

[ronaldbarendse]

@RyanWilson-byte As you can see from the labels on this issue, the indexing problem has been fixed in Umbraco 8.4.0 by returning a 503 status code.

And as @clausjensen mentioned, upgrading has always (at least as long as I've been working with Umbraco) been a two step process: update the files and authorize it after logging in (so the migrations run only once and errors while updating are only shown to logged in users)...

The issues you're facing are not related to this issue and probably because of a bad setup (file permissions) or external process (that reverts files/configuration settings). You should post further questions on the forum instead.

[clausjensen]

@RyanWilson-byte If what you say is true, I think there's something happening in your setup/process for upgrades that you are not completely aware of. No one is pushing any automatic upgrades to your site - there is no such functionality to do so in Umbraco.

If your site is indeed being upgraded, it is because something in your infrastructure setup (build pipeline?) has pushed an upgraded version of Umbraco to your live site. It is not something that Umbraco can do by itself. There's not even an upgrade button or anything that can do this - it is a very manual process.

As @ronaldbarendse mentions however, I think you could trigger the upgrade screen to show even though an upgrade is actually not available, if you by accident put wrong/old version numbers in the configuration files. So maybe this is what is causing it to show up. On the other hand - I don't really see how this would be resolved by just restarting your site, so it might be something else 🤷‍♂

I however concur with @ronaldbarendse that this is not related to this issue and would also encourage you to post on the forums for further assistance on your issue.

[DaveYognaught]

Alright guys, thanks for the heads up.

I think I misread the intentions of the original author. I thought they
were complaining about the upgrade mode redirecting all traffic. That was
my initial grievance. Their highlighted problems hit the nail on the head
why that's an issue for me right now as well. So I was confused when the
solution was to just redirect them somewhere else and that was accepted,
haha.

But from my understanding of what you guys are saying, Umbraco has to do
that since it's in a half-complete state where it would otherwise be
incapable of rendering the site. Ideally, I wouldn't want the 503 error
either, it'd still be downtime on a live site which clients will fume at
the mouth over. So my issue then i'll need to look into is why it's
triggering Upgrade Mode in the first place and prevent it unless we're
actually manually upgrading the site, for realsies.

wrong/old version numbers in the configuration files

I don't see how this would be resolved by restarting my site either, but
there's lot of weird interactions between FileZilla and Umbraco from my
experience, so this actually struck me as being very plausible. It'll be
something to investigate. You've definitely given me a few ideas to look
into now. Once I get some logs together, i'll share them on the forums if I
need additional help deciphering.

Many thanks,

On Tue, 28 Jan 2020 at 19:49, Claus [e-mail] wrote:

@RyanWilson-byte https://github.com/RyanWilson-byte If what you say is
true, I think there's something happening in your setup/process for
upgrades that you are not completely aware of. No one is pushing any
automatic upgrades to your site - there is no such functionality to do so
in Umbraco.

If your site is indeed being upgraded, it is because something in your
infrastructure setup (build pipeline?) has pushed an upgraded version of
Umbraco to your live site. It is not something that Umbraco can do by
itself. There's not even an upgrade button or anything that can do this -
it is a very manual process.

As @ronaldbarendse https://github.com/ronaldbarendse mentions however,
I think you could trigger the upgrade screen to show even though an upgrade
is actually not available, if you by accident put wrong/old version numbers
in the configuration files. So maybe this is what is causing it to show up.
On the other hand - I don't really see how this would be resolved by just
restarting your site, so it might be something else 🤷‍♂

I however concur with @ronaldbarendse https://github.com/ronaldbarendse
that this is not related to this issue and would also encourage you to post
on the forums for further assistance on your issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/umbraco/Umbraco-CMS/issues/6334?email_source=notifications&email_token=AN7E3SWMOXDCFK6URDBSQ4TRACD43A5CNFSM4IWDMKRKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKEU4IY#issuecomment-579423779,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AN7E3STQU7DM4IAK5RP6E3TRACD43ANCNFSM4IWDMKRA
.

[DaveYognaught]

Been a while guys.

https://our.umbraco.com/forum/using-umbraco-and-getting-started/101170-live-site-redirects-to-install-on-server-restart-if-accessed-too-quickly

We're still having this issue, not actually why i came to the github today but stumbled upon this and thought i'd let you know.

It's a bit more clear I think. Would appreciate any help. :)

[ronaldbarendse]

@RyanWilson-byte As already commented earlier, use the forum for these kind of questions and not this unrelated issue.

[DaveYognaught]

@ronaldbarendse I did. I attached it. Posted two whole months ago. With no replies. :)
Feel free to direct me to a better forum to post it on if you wish.

I'm still not convinced it's entirely unrelated either, now that you mentioned it.

As you have already mentioned, it was fixed only in Umbraco 8, with the initial specifics you provided for Umbraco 7 still up for grabs in terms of fixing.

I think if our Umbraco 7 installs got a 503 error as well, it'd be better for our sites.
Since we could actually monitor that and see that, in fact, the site is not loading ok.
In addition to your other points of why it's a problem, those all apply for me obviously.

The end result of our two issues are exactly the same.
The problems and issues we have with being redirected are the same.

I can say that with more confidence now that i've been tracking it a bit more.
Your fix would work for me, if applied to Umbraco 7. So it's still up for grabs.

In addition to the initial problem which is how I got to that upgrade page in the first place, that might be a more niche or specific bug. Which is what the forum post is about and asking for reproduction tests, etc.