Scratch-www: Suggestion: Improve Comment API Interface Instead of Reply Limit

[AmazingMech2418]

Just recently, #5495 was merged, implementing a 25-reply limit on Scratch studios.

This is potentially the most controversial update made to Scratch, if it is pushed to the main website, as is expected in mid-June.

However as many members of the Scratch community have expressed through forum posts (note that the link is not completely about this reply limit) and through comments in that PR, there are better solutions to these problems.

While we do not know the official reasoning behind this change, it is likely that it is one of two things:

• Moderation being difficult having to click "see more comments" many times
• Server performance

However, as many have pointed out, the first reason is quite invalid due to the fact that it will just make it harder to find the context of comments, which may make moderation harder, and also that client-side updates or moderation tools might be able to fix this problem without limiting replies in comment threads, by loading all replies at once.

The second reason, however, is quite valid. Every time you click "see more comments," a new HTTP request is being sent, and this likely creates a high load on the Scratch servers. However, limiting replies is not the best solution to this.

I explain the inefficiencies of this more in-depth in the quoted comment I sent on the pull request:

I'm still unsure if this change is actually helpful to solve one of the main issues (as mentioned above, finding the replies and knowing the context becomes harder.) However, it does sound like there are other concerns - perhaps server specs? - that are involved. I'm not going to bother with those things that I don't know, but I hope this change is actually helpful (and not uncomfortable for moderators who this PR intends to benefit mainly.)

Yeah, I think server specs might be an issue. One thing with the 3.0 API is that there are more individual requests when retrieving comments, so it would make sense if the server cannot handle that constant load, that it could potentially crash the server. I honestly think that might even be happening already with projects, given the "loading bug," which essentially means that comment APIs are overly cached. It might be that the separate DB access points per request are putting greater load on the server, or even just the request handling. Maybe, though, a possible solution could be increasing the default maximum per request, or using almost a JSON port of the old scratchr2 API rather than the current 3.0 API? Honestly, that would probably be the most efficient since it would have fewer requests for the same number of comments, fewer separate DB access points, and also the faster speed of JSON-based requests, and the reduced size per request from the old XML-based versions. And maybe still keep the normal replies endpoint with the 3.0 API for messages, so it doesn't have to load the entire page, but use a single endpoint for the rest of the comments on updated pages.

_Originally posted by @AmazingMech2418 in https://github.com/LLK/scratch-www/issues/5495#issuecomment-853239033_

While there are inefficiencies with this, the best solution is not to limit replying, but to improve the API. There was no reason for a limit under scratchr2, simply since all replies are sent at once, along with the base comments. Of course, the scratchr2 APIs for comments are slower since they use XML, which is much more verbose than JSON, but simply porting these APIs to JSON rather than requiring a request for the replies in each comment thread might ease the loads on the servers, without limiting Scratchers' abilities to reply to comments.

As I mentioned in the below quoted comment from the pull request, I also created a mockup of this JSON API:

Here is the API mockup. This may be a better solution to this problem. https://gist.github.com/AmazingMech2418/f85160217c233ebaade8802a2aafcdd2

Of course it would be a bit of work to redesign the API, but at least with studios and profiles, since the 3.0 API hasn't been completely implemented yet, it might be better to switch to this instead.

_Originally posted by @AmazingMech2418 in https://github.com/LLK/scratch-www/pull/5495#issuecomment-853253943_

With the best interest of the Scratch community in mind, I hope you review this suggestion as a possibility, rather than a nuisance in having to rewrite the APIs. In the long run, this will likely increase server efficiency, and improve the relationships between the Scratch Team and Scratch community, as have already been made quite tense over the past few years, which will overall create a more sustainable and healthy Scratch, both as a community and as an application.

To satisfy this issue, two things must be done:

• Revert #5495
• Implement a new API system that sends comments along with their replies, while still using a JSON format, similar to the Scratch 2.0 comment APIs.

Again, I hope you consider this a possibility, and hopefully follow through with it to make Scratch healthier and more sustainable in the long run.

[CluckCluckChicken]

This looks good, sending replies separately is wasteful, adds loading time, and means "request spam".

[AmazingMech2418]

It appears that #5539 is also becoming a part of this, so it too would likely have to be reverted for this issue, if it is merged.

[mxmou]

Would it send all replies at once, even if there are more than 20?

[AmazingMech2418]

Would it send all replies at once, even if there are more than 20?

Yes, it would. It would work just like the 2.0 APIs did, just with JSON instead of XML.

[mxmou]

Would it send all replies at once, even if there are more than 20?

Yes, it would. It would work just like the 2.0 APIs did, just with JSON instead of XML.

That would make loading pages that contain long comment threads slower.

[jeffalo]

Would it send all replies at once, even if there are more than 20?

Yes, it would. It would work just like the 2.0 APIs did, just with JSON instead of XML.

all replies at once is too much. whats currently in place works pretty well, since top level comments are readible while replies load. maybe loading a few replies (5 at most) could make that even better, as some replies could be readable too.

but anything more is unnecessary for one request (reading comments might be slower in some cases since it's a lot of data in one request vs data loaded when neccesary (loading more comment replies) in multiple requests).

since you're also suggesting the reply limit be removed, it would be possible to have an api response with over 2000 replies at once in extreme cases. this could be extremely slow on the backend, as not only does it need to find all of the replies for each of up to 40 top level comments, it would need to also return user data for each of them too. all of the comments on that page would not be visible until this huge request finishes (if it even ever does. - it might time out) this is in contrast to the current system, which fetches replies separately and the rest of the comments will load fast.

there's not much wrong with the current system, and i think it actually is better in some cases.

[AmazingMech2418]

Would it send all replies at once, even if there are more than 20?

Yes, it would. It would work just like the 2.0 APIs did, just with JSON instead of XML.

That would make loading pages that contain long comment threads slower.

It would still be faster than sending multiple requests to get the same comment threads.

[AmazingMech2418]

Would it send all replies at once, even if there are more than 20?

Yes, it would. It would work just like the 2.0 APIs did, just with JSON instead of XML.

all replies at once is too much. whats currently in place works pretty well, since top level comments are readible while replies load. maybe loading a few replies (5 at most) could make that even better, as some replies could be readable too.

but anything more is unnecessary for one request (reading comments might be slower in some cases since it's a lot of data in one request vs data loaded when neccesary (loading more comment replies) in multiple requests).

since you're also suggesting the reply limit be removed, it would be possible to have an api response with over 2000 replies at once in extreme cases. this could be extremely slow on the backend, as not only does it need to find all of the replies for each of up to 40 top level comments, it would need to also return user data for each of them too. all of the comments on that page would not be visible until this huge request finishes (if it even ever does. - it might time out) this is in contrast to the current system, which fetches replies separately and the rest of the comments will load fast.

there's not much wrong with the current system, and i think it actually is better in some cases.

What's currently in place also increases stress on the servers by them having to handle multiple requests to get the same amount of information, which is likely the main reason for the reply limit. If there wasn't this increased stress, there would be no problem.

And with the 2000 replies thing, I feel like this is exaggerated a bit. 2000 replies would slow down the page a bit, but it is not that great of a difference, and likely would not time out. I actually am probably going to create a test application to prove this even. I don't know how the servers themselves work, but I assume based on certain glitches that I have seen, that each comment is saved as an individual entry into the database and that the database is then filtered through to find replies, so I plan to use this model in my test.

[mxmou]

I think the best option would be to include the first few replies of each comment, but fetch the rest after clicking "Load more replies". The current system sends more requests than necessary, but yours could occasionally send too much data per request.

[AmazingMech2418]

I think the best option would be to include the first few replies of each comment, but fetch the rest after clicking "Load more replies". The current system sends more requests than necessary, but yours could occasionally send too much data per request.

That might work. Like send the first 20 with the main request, but for all other replies, use a separate request. I am currently working on a testing system to test the viability of all of these options, and will include this in the tests. Unfortunately though, I'm not sure how accurate the tests would be, considering that we don't have the same size database to pull from as the Scratch servers do. It would need quite rigorous testing for this to work... At least, though, the tests could provide a base point for the Scratch Team if they do decide to implement one of these ideas.