Gitpod: Evicted: Pod The node had condition: [DiskPressure]. But need to get code inside workspace

[HSSANT]

Today my workspace an error like:

Started 15 minutes ago: Evicted: Pod The node had condition: [DiskPressure].

But i need get my code inside this workspace before create a fresh new one. There is a hundred hours of development that i cant lost.

[JesterOrNot]

Sounds like a Kubernetes issue @jankeromnes can you take a look at this?

[jankeromnes]

Hi @HSSANT, so sorry about this error. We're looking into our cluster monitoring data to figure out why there was a DiskPressure event.

Meanwhile, in cases like these where you absolutely need the content of a workspace but it can't be started for some reason, please send an email to [email protected], with the workspace ID (found on https://gitpod.io/workspaces/) and request that we email you a backup.

Even when the workspace won't start, as long as you don't manually delete the workspace, we can retrieve a backup for you.

[jankeromnes]

Update: Email received and workspace backup sent.

[jankeromnes]

I confirm that a single node in our US West 1 cluster had a ~2.5-minute-long DiskPressure event between 2020-01-09 14:58:36.571 CET and 15:01:16.180 CET.

This caused at least one pod eviction (I saw many log messages in that timespan, but they all seemed to be about the same pod).

[jankeromnes]

This was a brief problem, which could be due to the image-builder downloading or building a very large or complicated Docker image, but I'm not sure.

I hear that we've seen a similar problem in December. I'm not sure if there is much we can do here, except investigate what creates pressure on our disks, and try to optimize it.

[HSSANT]

@jankeromnes and @JesterOrNot i received the backup file and my workspace are up and running again.
you rocks guys! You both saved my life!

Thank you so much.

[jankeromnes]

Thanks for the update! I'm so happy that you didn't lose any work. 🙂 (Sorry for the bad scare!)

FYI we're continuing to investigate this DiskPressure event, in order to find the root cause and if possible prevent it from happening again.

[HSSANT]

Hi @jankeromnes Today the bug happen again just in the same time/hour than yesterday. Is it possible to recover just like yersterday?(Just make the workspace up and running). Thanks in advance

[HSSANT]

workspace id

a91e6ab3-54a1-474d-bffc-de726469e27c

[jankeromnes]

@HSSANT oh, sorry that I missed this. Hopefully your workspace became available again soon after?

I think that in the worst case, you can simply wait one hour and the workspace will be available again. We're looking into resolving the situation.

If you do need a workspace backup though, please send an email to [email protected] -- that's about the fastest way you can get it (until we implement the "download workspace" feature in Gitpod).

[jankeromnes]

Closing this issue again in favor of #1080.