Scratch-www: "My Stuff" Very Slow for large profiles

[griffpatch]

Opening My Stuff on griffpatch account fails / times out. Generally, the main 'All Projects' view is ok, but as soon as you try to show 'shared' projects only, or sort by loves, etc the page never returns and a 503 timeout error is given in the browser console. This has been the case ever since Scratch 3 went live, but was not a problem in Scratch 2.

Request URL: https://scratch.mit.edu/site-api/projects/shared/
Request Method: GET
Status Code: 503 first byte timeout
Remote Address: 151.101.130.133:443
Referrer Policy: no-referrer-when-downgrade

Steps to Reproduce

Are you able to view by profile?

Operating System and Browser

Chrome on Windows 10 PC

[griffpatch]

Note, subsequent visits (by refreshing the page) then become possible, so I assume the database will cache some of the queries... I suspect after a little time the timeouts will return?

[thisandagain]

/cc @rschamp @colbygk

[rschamp]

Interesting that this endpoint didn't have any issues before Scratch 3. I don't think we touched it.

[griffpatch]

Did the underlying database change? index structure, etc? It's like having
to count up the view, comment & love totals might be causing the database
query to take a long time?

On Mon, 4 Mar 2019 at 14:48, Ray Schamp [e-mail] wrote:

Interesting that this endpoint didn't have any issues before Scratch 3. I
don't think we touched it.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2815#issuecomment-469278686,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvi5Hbj4yBK_L05yodlItMkUC6N9-ks5vTTJBgaJpZM4bb6FF
.

[BonfireScratch]

I've got this issue too

[colbygk]

Did the underlying database change? index structure, etc? It's like having to count up the view, comment & love totals might be causing the database query to take a long time?
…
On Mon, 4 Mar 2019 at 14:48, Ray Schamp @.*> wrote: Interesting that this endpoint didn't have any issues before Scratch 3. I don't think we touched it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#2815 (comment)>, or mute the thread https://github.com/notifications/unsubscribe-auth/AGbNvi5Hbj4yBK_L05yodlItMkUC6N9-ks5vTTJBgaJpZM4bb6FF .

Nope, that view is the exact same one and we did not alter the db in areas associated with mystuff as far as I know.

[griffpatch]

Well, whether it was updated or not, it went from being very quick before
Scratch 3 to timing out at each visit after Scratch 3 went live.
Whether it's to do with increased demand on the database server meaning
more cache misses or a change in indexes, etc, there is a definite and very
unpleasant delay going on.
Thanks,
Griffpatch

On Tue, 12 Mar 2019 at 16:10, Colby Gutierrez-Kraybill <
[email protected]> wrote:

Did the underlying database change? index structure, etc? It's like having
to count up the view, comment & love totals might be causing the database
query to take a long time?
… <#m_9209838632357124750_>
On Mon, 4 Mar 2019 at 14:48, Ray Schamp @.*> wrote: Interesting
that this endpoint didn't have any issues before Scratch 3. I don't think
we touched it. — You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#2815 (comment)
https://github.com/LLK/scratch-www/issues/2815#issuecomment-469278686>,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvi5Hbj4yBK_L05yodlItMkUC6N9-ks5vTTJBgaJpZM4bb6FF
.

Nope, that view is the exact same one and we did not alter the db in areas
associated with mystuff as far as I know.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2815#issuecomment-472063586,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvhO51BIjz1qqpTpo6zmvWgJv8Nm6ks5vV9FfgaJpZM4bb6FF
.

[thisandagain]

@griffpatch Agreed! We have seen some slow downs on "My Stuff" and "User Profile" Comments in particular. Neither of these systems were directly changed in the 3.0 launch, but shifts in DB load have clearly caused some issues. We are more broadly investigating DB performance issues and hope to start making some changes soon. Thanks for being patient with us.

[griffpatch]

Man I appreciate the work you guys do - Don't get me wrong and think I'm
impatient in the slightest :) - I'm just letting you know if I see anything
strange going on grin. You're making huge progress with Scratch 3 under a
lot of pressure. The amount of things that are working amazingly is a
testament to all your hard work and far exceeds any of the issues in my
opinion! Scratch on as they say - he he.

On Wed, 13 Mar 2019 at 17:49, Andrew Sliwinski [e-mail]
wrote:

@griffpatch https://github.com/griffpatch Agreed! We have seen some
slow downs on "My Stuff" and "User Profile" Comments in particular. Neither
of these systems were directly changed in the 3.0 launch, but shifts in DB
load have clearly caused some issues. We are more broadly investigating DB
performance issues and hope to start making some changes soon. Thanks for
being patient with us.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2815#issuecomment-472533790,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AGbNvrzu2rR6HzhjHIq5Ip5I0vLz2Bt5ks5vWToegaJpZM4bb6FF
.

[kyleplo]

The sort function, especially for sort by remix never really worked, even in Scratch 2. Also, we should probably just migrate My Stuff to scratch-www.

[BonfireScratch]

@kyleplo we certainly should

[MasterOfTheTiger]

@kyleplo I am sure that is planned for the future, but there are reasons why it has not been done now.

[BonfireScratch]

@MasterOfTheTiger probably, as anyone could create a proper sort function... I guess they have something else planned to do

[benjiwheeler]

It might make sense to add a db index for the query that My Stuff uses.

Yeah, we do have plans to move My Stuff and other pages to scratch-www, and are hoping to get to it soon!

@kyleplo the Sort functions in My Stuff seem to work for me -- but I have way fewer projects than @griffpatch ! Are you just seeing it be slow? Or does it seem to sort, but incorrectly?

[MapMaths]

Really? Quite scary! 😐
I can still open it

[colbygk]

Really? Quite scary! 😐
I can still open it

What is?

[kyleplo]

It loads, but doesn't sort very well. It's usually close, but sometimes projects quite in the right order. For example, if it is supposed to be 1, 2, 3, 4, 5, 6, it might end up being 1, 3, 2, 6, 4, 5

[benjiwheeler]

I'm curious if the performance is any better now, @griffpatch.

[griffpatch]

Yes, I find no issue with my stuff screen these days. I just confirmed it
now and it loaded up in just a few seconds, and I could switch order and
filter and it all stayed super quick thanks.

Griffpatch

On Sun, 16 Aug 2020, 19:01 Benjamin Wheeler, [e-mail]
wrote:

I'm curious if the performance is any better now, @griffpatch
https://github.com/griffpatch.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LLK/scratch-www/issues/2815#issuecomment-674557588,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABTM3PQQKR36TR4ER5YVPHDSBANH7ANCNFSM4G3PUFCQ
.

[ljy-002]

是的，这些天我的东西屏幕没有问题。我现在确认了它，并在短短几秒钟内完成加载，我可以切换顺序和过滤器了，一切一切都非常快捷，谢谢。格里帕奇
…
On Sun, 16 Aug 2020, 19:01 Benjamin Wheeler, @.*> wrote: I'm curious if the performance is any better now, @griffpatch https://github.com/griffpatch. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2815 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTM3PQQKR36TR4ER5YVPHDSBANH7ANCNFSM4G3PUFCQ .

I know, but I will surpass you one day