Gitpod: Can't start workspaces in Europe

[kescardoso]

Describe the bug

Worked fine until last night, but today I am trying to open my workspace again and I get this error: Error: "2 UNKNOWN: Channel Destroyed
Please file an issue if you think this is a bug.""

I don't remember doing anything to erase my workspace last night. This is very scary. I would be very thankful to have some help. Thanks

[isontheline]

Same here :(

And now : Acquiring node... (1/3)

[kescardoso]

@isontheline I downloaded the files and will start a new workspace with what I have. I wonder if this is an issue with the system....

[isontheline]

Related to #1377

[jankeromnes]

Hey, sorry about this @kescardoso. We're currently investigating an outage that started about 30 minutes ago that prevents workspaces from starting in the Europe cluster.

Specifically Error: "2 UNKNOWN: Channel Destroyed" means that our workspace manager had to be restarted, and it should be temporary.

But let's track this new outage here, since https://github.com/gitpod-io/gitpod/issues/1377 was about yesterday's Google Cloud global outage.

[jankeromnes]

We're actively investigating this issue, and will resolve or mitigate it as soon as possible. Sorry about the disruption to your work -- we're doing our best to restore service ASAP.

I'll now update https://status.gitpod.io.

[kescardoso]

@jankeromnes thank you for the update, I saw the other issue about the outage, I will keep watching it here. Thanks!

[isontheline]

@jankeromnes May you switch to another continent cluster ?
I can start new workspaces but I have one in a bad state :
Pending - Started 2 hours ago.

[jankeromnes]

@isontheline That's indeed what we did: we re-routed Europe traffic to North America while working on the Europe cluster.

We've now mitigated this issue, and will deploy a fix shortly. The service seems back to operational, so I've resolved the incident on https://status.gitpod.io

Please try again, and if you're still seeing issues, please let me know.

[isontheline]

@jankeromnes Thank you !
New pulled workspace works as expected.
This old one stucks in "pending" state : c9dd8ef3 (but I can now work, thanks !)

[jankeromnes]

FYI, Gitpod also recently became much more popular, and we're seeing record numbers everywhere. While this makes us super happy, it also adds a bit of stress on our system, and challenges some assumptions we've made in the past. We're doing our best to cope with the influx while also guaranteeing a flawless user experience, and we really appreciate your goodwill and patience when some of these small outages happen. I think Gitpod already scales really well (especially given the numbers we're seeing), and we've also made a number of changes that should make it more resilient in the near term. Thanks again!

[jankeromnes]

This old one stucks in "pending" state : c9dd8ef3 (but I can now work, thanks !)

Aha, thanks for mentioning, I'll check back on "stuck" workspaces.

EDIT: Forced all workspaces stuck in "pending" back to "stopped".

[kescardoso]

@jankeromnes thanks, workspaces working again!

[jankeromnes]

Awesome! Many thanks for the updates 👍