Gitpod: all workspace not starting

[HSSANT]

Describe the bug

all of my workspaces are in "starting" status forever, follow bellow the screen that im looking for all eternity:

Steps to reproduce

execute start commad for any of my workspaces like bellow:

https://gitpod.io/start/#ca0909ae-954f-483d-8498-c52d516b5467

Expected behavior

the workspace would be started after some seconds as usual

Additional information

my user name is hssant

Example repository

example workspace

ca0909ae-954f-483d-8498-c52d516b5467

[skinny85]

Can confirm my workspaces also don't start. I also get blank pages when refreshing the main page often, and sometimes I get prompted to log-in again, even if I did it a few seconds before.

[jankeromnes]

Thanks for reporting, confirming partial outage. Seems to be US cluster only this time.

[jankeromnes]

We're redirecting US traffic to EU while investigating US.

[anikethsaha]

I am facing the same issue, I am from India.

[alesanchezr]

This is what is happening to some of my students:

The open button is disabled but no other error.

[jankeromnes]

Thanks @alesanchezr. The outage caused an accumulation of workspaces in "pending" state, but almost all of them have recovered now. The US region looks stabilized, so we're about to re-enable traffic to it.

(Meanwhile, it should have been possible for you to start new Gitpod workspaces, and you would have landed in the EU cluster -- sorry for not stating that more clearly earlier.)

Anyway, the root problem seems resolved now, and Gitpod should be operational in all regions. We also have good data to further investigate & fix this new problem, and prevent it from reoccuring in the future.

Sorry about the bad experience today, and many thanks for your patience as we work hard on making Gitpod scale to record traffic every week while keeping a smooth dev experience.

[anikethsaha]

Thanks for such a quick response.

Always grateful for gitpod and the team.

Thanks 🥇

[jankeromnes]

Thanks. US region is back online, but recovery seems a bit slow. We're watching it to see if it recovers (i.e. is able to actually start workspaces) but if not we'll have to redirect traffic to EU again and investigate more.

EDIT: We're now redirecting US traffic to EU again, because the US cluster still has issues. (If your workspace remains "pending" for a long time, just create a new workspace and it should work -- meanwhile, the "pending" workspace should get back to normal within max 1 hour).

[th3c0d3br34ker]

I guess the issues still exists... maybe because of too much congestion?
I had a few unpushed commits... I guess they are lost?

[jankeromnes]

@th3c0d3br34ker As soon as we redirect the traffic, the problem should no longer happen when you create new workspaces. (So the issue has been quickly mitigated, allowing everyone to use Gitpod normally while we fix the underlying problem in the US cluster.)

But I guess you're talking about a workspace that you've started in US before the traffic got re-routed? In that case, the "status: pending" should automatically go back to "status: stopped" after maximum 1 hour, and your unpushed commits should still be there. Check https://gitpod.io/workspaces/ and look for the workspace with the unsaved work (Gitpod explicitly highlights unpushed commits, uncommitted changes, and untracked files).

[RDIL]

I think Gitpod needs another US cluster.

[jankeromnes]

Problem solved! 🎉

I think Gitpod needs another US cluster.

Gitpod already spans 3 clusters in the US (us-west1-a, us-west1-b, us-west1-c) but they do indeed seem a bit less reliable than the EU clusters. But I think another US region that we tried was even worse. Still, I think the level of service quality is getting pretty good, and we have a number of improvements in the pipeline that should improve it even more without requiring to switch clusters.

Personally I'm also pretty happy that we got better at re-routing traffic -- now we can just flip a switch at the first sign of trouble, and Gitpod can continue to be used even if an entire region is on fire (which, thankfully, doesn't happen that often).