Gitpod: 500 Internal Server Error nginx/1.14.1

[ColbyWTaylor]

Describe the bug

Receiving this error (along with students) in the middle of session, unable to connect to github to push as well.

500 Internal Server Error
nginx/1.14.1

To Reproduce

Open any workspace...

Expected behavior

That it stays open!

Additional Information

Have been having more issues lately- any idea what it may stem from ?

[jankeromnes]

Hi @ColbyWTaylor, thank you for reporting this unfortunate situation. It's an intermittent problem that appears sporadically only in the US region. We're actively investigating it, quickly fixing it whenever it appears, and we believe we're getting closer to finding the root cause of this particular problem.

We'll let you know when we've been able to resolve this for good. See also #1075.

[tylerdotjs]

This issue occurs for me at 1:30 PM CST every day.

[jankeromnes]

Update: We've implemented a fix that should prevent these Error 50x events from happening again.

We'll continue to monitor the situation for a few days to validate this fix. If you see this error again, please kindly report it here, cc @RDIL @JesterOrNot @HSSANT.

Once the fix is confirmed we'll close this issue, and we're also planning to publish a blog post with more details on what happened and how we've fixed it.

[RDIL]

Thanks Jan, you’re the best 😃

[jankeromnes]

(Thanks @RDIL! But all the praise goes to @geropl @32leaves @svenefftinge and @meysholdt 😊)

[davelavoie]

Unfortunately I'm still getting a 500 error right now...

[geropl]

Hi @davelavoie ,

thanks for reporting!

Could you elaborate where exactly you noticed the 50x or what you where doing? This would be a tremendous help for pinning this one down.

[davelavoie]

Currently it's not happening, but while it was happening, I was simply browsing the front-end of a WordPress install.

[RDIL]

It just started happening aah

[HSSANT]

its impossible to access again. Seriously thinking about hire a better
service. I cant work properly day after day because of gitpod.

On Mon, Jan 27, 2020 at 11:43 AM Reece Dunham [e-mail]
wrote:

It just started happening aah

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/gitpod-io/gitpod/issues/1128?email_source=notifications&email_token=ACLMZRRGMWYTTZRI74SRGUDQ73XIPA5CNFSM4KJYMNO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ7X5KA#issuecomment-578780840,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACLMZRQ66QTDXK44Z5DURJDQ73XIPANCNFSM4KJYMNOQ
.

[svenefftinge]

We are all super sorry and work with highest priority to get this reliably working.

[jankeromnes]

I concur with Sven, we're really sorry about these repeated disruptions @RDIL and @HSSANT, and we're doing our best to make Gitpod durably more stable.

The 500 internal server error described in this issue should no longer happen. However, we're currently seeing the Error 504 again: #1151

We're still unclear why specifically the US region started having all these different bugs recently, while the EU and Asia Pacific regions continue to operate smoothly (like the US before). We're able to mitigate each specific bug as it appears relatively quickly, but it's currently our top priority to understand the deeper issues making the US region so unstable. We're currently looking at whether this is due to the particular datacenter we're in; or if it's due to some US-specific workloads we run; or if Gitpod's scary-fast growth could be the reason its biggest region is now suddenly showing signs of stress. We want Gitpod to be always fast & reliable, and we'll make any efforts necessary to achieve this.

[jankeromnes]

[Update] We've found and fixed what we believe was the root cause of all these US issues. More details here: https://github.com/gitpod-io/gitpod/issues/1151#issuecomment-578933856

An immense thank you for all your good will, understanding, and help in getting this resolved.