Gitpod: Unable to create workspaces based on repositories

[mparkcode]

Describe the bug

Unable to create workspaces based on repositories

Steps to reproduce

Click on gitpod button on a repository

Expected behavior

A gitpod workspace to be created

Additional information

error message:
cannot initialize workspace: cannot initialize workspace: cannot create workspace: mkdir /mnt/workingarea/0a371eb9-1600-4629-bd91-c9f0ec04ba66: no space left on device
Please file an issue if you think this is a bug.

Example repository

[JesterOrNot]

This is a known issue with Kubernetes please try refreshing and see if that helps

[mparkcode]

Hmm, the problem seems to be persisting

[505799690]

same problem

[JesterOrNot]

I'll let my colleagues know

[boraoren]

same problem

[devlights]

same problem

[zhangsean]

same problem

[kugared94]

cannot initialize workspace: cannot initialize workspace: cannot create workspace: mkdir /mnt/workingarea/83966c08-1407-4783-9625-d97e12458627: no space left on device
Please file an issue if you think this is a bug.

[jankeromnes]

Thank you for reporting this, and sorry for the disruption. We're investigating.

[jankeromnes]

Update: We found the root cause of this problem, and are starting to free up space in all nodes. Service should start recovering from here on out.

[devlights]

I have confirmed that the problem has been resolved. Thanks :)

[mparkcode]

Yup looks like it's working again, great stuff, thanks.

[JesterOrNot]

Closing this as resolved then!

Please file another issue or reopen this one if this occurs again

[xys20071111]

It works.I can countine my things.

[jankeromnes]

I confirm the issue is resolved. Thanks a lot for your patience and help.

As next steps, we will improve our disk usage monitoring; alerting; and disk quota enforcement.

In the future, when reporting operations issues and/or downtime, please try to mention which global geographical region you're seeing this problem in (e.g. Asia, Europe, North America -- or ideally the cluster name if you know it, e.g. ap01, eu01 or us02). This can help us resolve it faster.