Scratch-www: Cloudmonitor Shows Spinning Circle

[griffpatch]

The cloudmonitor is showing no data and not loading - stalled? Also cloud projects don't appear to be working. This could just be a service outage?

(index):6 GET https://js-agent.newrelic.com/nr-1099.min.js net::ERR_BLOCKED_BY_CLIENT
r @ (index):6
load (async)
__nr_require.loader @ (index):6
r @ (index):6
__nr_require.1 @ (index):6
(anonymous) @ (index):6
2jquery.min.js:4 GET https://clouddata.scratch.mit.edu/logs?projectid=108566337&limit=40&offset=0 504
send @ jquery.min.js:4
ajax @ jquery.min.js:4
(anonymous) @ (index):864
j @ jquery.min.js:2
fireWith @ jquery.min.js:2
ready @ jquery.min.js:2
J @ jquery.min.js:2
(index):1 Access to XMLHttpRequest at 'https://clouddata.scratch.mit.edu/logs?projectid=108566337&limit=40&offset=0' from origin 'https://scratch.mit.edu' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
base.js:8 Uncaught Uncaught ajax error. Attempted URL: https://clouddata.scratch.mit.edu/logs?projectid=108566337&limit=40&offset=0   Status: 0

Steps to Reproduce

https://scratch.mit.edu/cloudmonitor/108566337/

Operating System and Browser

Windows 10, Chrome

[colbygk]

Thank you for reporting. Since the release of 3.0 we've been experiencing some errors in the clouddata system that may be related to changes in how clouddata works with the frontend. These problems are leading to clouddata being unavailable at times. We're working on it. :)

[griffpatch]

What I am noticing is that the cloud server at times works brilliantly.
When it doesn't, this is what I see:

• Sometimes you can see cloud changes, but your changes do not appear in the log
• Sometimes you can't see other's updates, but your changes do appear in the log
• Often it looks as if writing becomes locked to a single user across all cloud vars? in that rather than all connected projects writing interleaved (as coded), they will suddenly become grouped to single user bursts of updates (player 1 updates cloud var 1 x10 times in a row, then player 3 might update cloud var 3 x7 times in a row, etc...)
• Sometimes when scratch does finally manage to get a connection to the cloud server, the data changes can stream through all of a sudden as a huge backlog (delivering player position changes from the last goodness knows how long condensed into a really short period) the effect being that I could see all the players dashing around the entire level really fast for a few seconds before settling down. This was quite surprising. I don't batch up the messages internally, so this has to have been what occurred. I see this happening mainly when the cloud servers are struggling I believe.

[griffpatch]

Cloud has been down for some time now.
You can update the cloud vars from within your project.
The updates appear in the log.
But other players updates do not get seen my the local project.

[colbygk]

Thanks for the heads up. Normally we are quickly alerted to these issues, but for some reason they were not triggered this morning....

[griffpatch]

The cloud variables have been updating in the cloud log ok for the last day, but they are not being read into the scratch project... Therefore no one can see anyone else. This has been an issue for over 24 hours now.

[griffpatch]

The cloud variables have again been failing for a couple of days... Is there an official page to post this issue to because each time I post this it gets fixed for half a day or so and then appears to stop working again... Does the cloud service just need restarting every day? Would you like me to continue to report cloud data issues, or leave it to you? This doesn't feel like the right place, but it doesn't appear that there are any automatic alert mechanisms in place?

[colbygk]

I'm personally okay with seeing reports here. We're still responding to the range of issues presented once we released 3.0 and the downtime associated with clouddata has been a surprise. We'll be working on it. Thank you for your reports!

[griffpatch]

If the cloud was up again just now, then it's down already :( - I can write
to the cloud, but not read the values back

On Mon, 14 Jan 2019 at 13:49, Colby Gutierrez-Kraybill <
[email protected]> wrote:

I'm personally okay with seeing reports here. We're still responding to
the range of issues presented once we released 3.0 and the downtime
associated with clouddata has been a surprise. We'll be working on it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2610#issuecomment-454010403,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvjdWj5lYmUmAmJ7lVFGayjBp_7Qpks5vDIrdgaJpZM4ZzfWX
.

[griffpatch]

Still down... Can write, but not read

[griffpatch]

It's working - yay :)

[griffpatch]

They're down again...

[griffpatch]

Also, when opening a cloud test project, the project starts up with the cloud vars showing with the 1st cloud var set to the correct value, but the second one is not accurate with the cloud log.

Changing cloud var 1 takes roughly 50 seconds (consistently right now) to change on another pc running the same project. The change is detected and the 2nd cloud var is changed... however, this change 'never' makes it back to the first pc... and even roloading the project still only shows the 1st cloud var is correct... the 2nd on IS correct in the cloud log, but none of the running projects can see that value.

https://scratch.mit.edu/projects/26861174/
https://scratch.mit.edu/cloudmonitor/26861174/

Following on from this I note that if I change cloud var 2, then shortly after cloud var 1, then 50 seconds later cloud var 1 changes on the other pc, but cloud var 2 stays unchanged. Yet both changes appear in the cloud log.

So - to conclude, when the cloud server goes wrong in this case, only one cloud var transmits data, possibly the first one? And it takes 50 seconds to transmit that change. The other cloud vars do not get read from the cloud server at all, not even on start up when the console says it has connected. The 1st cloud var does get read on first connection.

[griffpatch]

Still down (see previous message)

[griffpatch]

Yeah... I haven't seen the cloud working now for a whole week... Shame.

[griffpatch]

Wow - Unexpectedly, the cloud servers were back up and working great for 15 mins there... and pow. They are down again. Plus cloud server log is back to showing a spinning circle. (It was a very good 15 mins though)

[griffpatch]

And it's up again!

[griffpatch]

And down right now

[griffpatch]

And it's up again :)

[griffpatch]

And it's down again

[thisandagain]

Thanks @griffpatch. We are still working on the stability issues with the Cloud Variables server (obviously). Sorry about the issues. @rschamp and @colbygk are on the case.

[colbygk]

@griffpatch Thanks for updating this regularly. I've just deployed a fix that I am pretty confident should lead to long term stability for the backend part of cloud data. Please let me know if you notice it misbehaving.

[griffpatch]

Now that would be amazing...!
Will let you know,

Griffpatch

On Thu, 7 Feb 2019 at 17:45, Colby Gutierrez-Kraybill <
[email protected]> wrote:

@griffpatch https://github.com/griffpatch Thanks for updating this
regularly. I've just deployed a fix that I am pretty confident should lead
to long term stability for the backend part of cloud data. Please let me
know if you notice it misbehaving.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2610#issuecomment-461527572,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvgL2aKzGWjNiFQN509VOVmUXX98qks5vLGYsgaJpZM4ZzfWX
.

[colbygk]

Looking good!