Gitpod: Latest changes unsaved following Gitpod timeout (resulting in hours of work being lost)

[nakitamccool]

Latest changes unsaved following Gitpod timeout, lost work as a result. Impacting students in the EU region.

Raising this bug on behalf of @PeterLenting (one of multiple Code Institute students affected by this issue)

Original comment from @PeterLenting:
I was working in Gitpod, had just finished adding a checkout-app to my project when Gitpod told me there was a timeout. I went to my dashboard, where Gitpod was ' stopping' my project. After about 20 minutes I was finally able to open my project again. But now my Checkout-app is nowhere to be found

To date, we're aware of ~10 students affected by this issue, all of whom are based in Europe.

[armedcor]

Have 13 students in a classroom affected by this now.

[fudge88]

I was working on my project- sharing my workspace with my tutor- when we it crashed. I reloaded it to find my work from the previous 2 hours had disappeared- despite me pushing my commits. When I logged on again to share my workspace with my tutor again- I received a pop up telling me I cant make any active changes to my workspace.

[kevin-ci]

Have had a further four students affected.
Descriptions:

Originally my code wasnt running properly, Gitpod was still running and i could make changes but they werent changing on my browser. Then the bottom tab went yellow to say it was offline and after a minute it popped up saying something like gitpod stopped and the status section said "Stopping" with a spinning wheel and then gave me an option to go to workspaces. Went to workspaces and the workspace i was just in was greyed out and i could reopen it.

was was working on my project in gitpod, setting up files, folders, writing todo, readme etc. at this point, gitpod would not respond to deleting files, i would click delete, ok , but the file would still stay, okay, strange i thought, but i can deal with that later. during this process i had comitted and pushed twice and saved all files locally (CTRL-s) but when i refreshed the page, all my progress was deleted, even the things i had pushed and committed.

Gitpod crashed while trying to save when i went back to work spaces the space i was using was greyed out and was unable to access the stuff i was working on.

[Taitdanielle]

I've lost a good few hours work. Student care helped but started again. Currently on my 2and project which is making me fall behind.

[MelBiggs]

I returned to my project after lunch to find the new files I worked on today, and committed and pushed to Github, have disappeared from my Gitpod files. My Github files are still there however

[tgarratt]

Took a 40 minute break from my code to go get something to eat. Came back as normal and my workspace had gone to sleep as expected, rebooted it to see old code which was strange because I could see how it should have been on my live site but after a refresh on the site I could see that the code had all gone...

[mark-obeirne]

Received this error message:

Workspace Not Running
Status: stopped
Cannot initialize workspace: cannot create sandbox.

Pressed Start to reopen Gitpod but it was several commits behind what I had pushed to Github.

[jankeromnes]

Hi all, we are super sorry about this problem. Losing hours of work is incredibly frustrating, and should never happen.

The data loss that occurred today was due to a fault in a new workspace disk quota mechanism that was deployed early this morning. We had extensively tested this code in our staging environment, and this bug did not occur then, so this took us by surprise.

We have already disabled the new disk quota mechanism, so this problem should be resolved for now. We will continue to monitor production to ensure everything looks good, and we will both overhaul the faulty disk quota mechanism and improve our testing & release processes in order to prevent such a bad situation from happening again.

[svenefftinge]

Here's what happened:

At 13:00 yesterday we rolled out a feature that broke the workspaces of some users, including yours. A feature we had brought into production that morning falsely flagged some workspaces as “not running” - despite them being actively in use. When we rolled another update at 13:00 the system identified those “not running” workspaces and shut them down which caused the timeout message you saw. Because the system thought the workspaces had never been running in the first place, it did not create a backup prior to removing the container.

This is probably the worst thing that could have happened in our mission to improve developer productivity. We are truly sorry, especially because we are unable to bring back the hours you have spent during the morning writing code.

Now that we have identified and fixed the underlying problem we are hard at work improving our development process, monitoring and resilience measures to ensure this never happens again.

We've reached out to all affected users via e-mail earlier today and are super grateful for all the warm responses. Thank you for your understanding, we won't let you down like this again. ❤️