Nodebb: Infinite scroll jumps

[boomzillawtf]

Please include the following information when submitting a bug report/issue:

• NodeBB version and git hash: Whatever community.nodebb.org has running on May 10 2017
• Database Ditto
• Exact steps to cause this issue
  
  
  
  1. Find a long topic and start in the middle: https://community.nodebb.org/topic/180/who-is-using-nodebb/298
  
  
  2. Scroll slowly (up or down) until new posts are loaded
  
  
• What you expected
  
  
  
  • The new posts would be loaded into my browser but my view would be right where it was
  
  
• What happened instead
  
  
  
  • The current post often jumps somewhere else
  
  

We (https://what.thedailywtf.com) noticed this when we upgraded to 1.5.0.
Here, middle of a giant thread:
https://what.thedailywtf.com/topic/12236/the-official-status-thread/30024

What's worse, is that when it jumps and you try to scroll back the position jumps even more.

[AzariahAeroborn]

This issue is also much more reproducible for me on mobile - entering several different threads of a few hundred to about 2000 posts in the middle resulted in instant infinite scrolling problems, until I eventually gave up on trying to wait it out and pressed "go to the end of the thread" or reloaded the topic list instead.

[barisusakli]

Is Delay Image Loading on or off?

[boomzillawtf]

I originally had it off. I turned it back on and have the same problem.

[barisusakli]

Can you guys provide browser and version as well? Couldn't reproduce on chrome Version 58.0.3029.110 (64-bit)

[NedFodder]

FF 53.0.2 Win7. It doesn't happen to me on community.nodebb.org anymore, but it did when this issue was first reported. Still happens on what.thedailywtf.com. I have Delay Image Loading on.

[boomzillawtf]

Chrome Linux Version 58.0.3029.110 (64-bit)
Chrome Android 58.0.3029.83

[julianlam]

Hm... if it doesn't happen on our forum but only on TDWTF, then it might be an issue with something else there changing the DOM on page load?

The image loading code that we have fixes the viewport position only for images... all other DOM manipulations can still make you move about.

[AzariahAeroborn]

I haven't put a ton of effort into rigorous reproduction, but anecdotally it seems to be much more consistent to reproduce if I send a scroll event while the page is still loading. Given that I tend to be a bit impatient when I'm diligently browsing the forum during my bathroom breaks at work, that might explain why I'm having very frequent problems on mobile but nothing really noticeable on desktop.

My mobile: iOS 10.3 (latest) with Safari, iPhone 6S
My desktop: Chrome 58 latest 64-bit on Windows 10

My delay image loading: Enabled

[AzariahAeroborn]

Additional replication information: I'm getting pretty consistent replications in threads that are very heavy in One-Boxed links and pictures. The replication is occurring both with "Delay Image Loading" enabled and disabled, though it seems to replicate more easily with "Delay Image Loading" enabled.

Entering one of our big link threads here: https://what.thedailywtf.com/post/1137471 and scrolling down 20-30 posts generally generates the jumping around, usually jumping down a large chunk of posts and then jumping back up to the original entry post or even earlier.

Edit: An additional replication that's a bit more consistent (with apologies for the tenor of the thread, we have some... interesting... discussions): Enter here - https://what.thedailywtf.com/post/1152945 - Scroll to the 11th post (first "off page" post, which also has a pretty large image), https://what.thedailywtf.com/post/1152990 and the thread will have a seizure. After you hit that second post, any attempt to scroll up will put you above the first post, and any attempt to scroll down past the first post will put you after the second post. That's been extremely consistent for me with Delay Image Loading enabled... with DIL disabled, it has still happened, and still generates the "permanent" breakage when it does happen, but I've had some cases where I can get past the second post normally with DIL disabled.

Platform: Windows 10 desktop
Browser: Chrome Version 58.0.3029.96 (64-bit) AND Version 58.0.3029.110 (64-bit)

[julianlam]

Ah, then it is related to the one-boxer. My jellypotato code doesn't work for plugins that manipulate the DOM, only images. I don't think there's anything I can do about that, it's up to the oneboxing plugin to fix...

[boomzillawtf]

It's not iframely. I just turned it off and the problem is still happening.

[boomzillawtf]

• I started at post 60,000 and hit page down 3 times.
• The post index after the third page down was 60009.
• After posts streamed in, the current post displayed was 60022. Also, 60022 is now the first post displayed at all!
• I hit page down, then page up.
• Previous posts streamed in. But I was taken to post 60011!

https://what.thedailywtf.com/topic/12236/the-official-status-thread/60000

[boomzillawtf]

I disabled our iframely, youtube and tdwtf-customizations plugins and the problem was still there.

Here is the list of our active plugins:

nodebb-plugin-youtube-embed
nodebb-plugin-iframely
nodebb-plugin-composer-default
nodebb-plugin-dbsearch
nodebb-plugin-deny-moderator-access
nodebb-plugin-emailer-amazon
nodebb-plugin-emoji-extended
nodebb-plugin-emoji-one
nodebb-plugin-emoji-static
nodebb-plugin-google-analytics
nodebb-plugin-gravatar
nodebb-plugin-htmlcleaner
nodebb-plugin-imagemagick
nodebb-plugin-import-tdwtf
nodebb-plugin-markdown
nodebb-plugin-mentions
nodebb-plugin-mentions-emoji
nodebb-plugin-ns-login
nodebb-plugin-shortcuts
nodebb-plugin-soundpack-default
nodebb-plugin-spam-be-gone
nodebb-plugin-sso-facebook
nodebb-plugin-sso-github
nodebb-plugin-sso-google
nodebb-plugin-sso-twitter
nodebb-plugin-tdwtf-buttons
nodebb-plugin-tdwtf-customizations
nodebb-plugin-unresponsive
nodebb-plugin-upvote-notifications
nodebb-rewards-essentials
nodebb-theme-persona
nodebb-widget-essentials

[julianlam]

What about youtube-embed?

[boomzillawtf]

I had that disabled, too. No difference. That just has fixed size divs, anyways, so post size is static. The post loading / unloading stuff is really weird. That's far beyond anything that previous jellypotato did.

[barisusakli]

Could it be happening because of multiple infinite loads happening simultaneously? Does it happen when you slowly scroll down(or page down one at a time) and wait for it to load and then trigger another infinite scroll?

[boomzillawtf]

Yes, it definitely happens then. I was going very slowly with my page downs so that I could mark the post index at each step.

[AzariahAeroborn]

I've been doing some more research into problematic threads on our forums and doing a test reproduction, and I believe the issue lies with loading some critical mass number of "blockquote" elements into the DOM.

My test thread right now is in a login-required section of our forum (we keep that section hidden from Google), if you enter here https://what.thedailywtf.com/topic/22907/testing-jellypotato-theory-number-2 and scroll down until the first batch load for more posts, you'll jump from roughly post 16 to post 28.

Entering the same thread at the bottom and scrolling back up behaves similarly - as of the time of this posting, I had 63 posts in that thread, the initial entry at 63 loaded as far back as post 54, and once I scrolled up far enough that post 53 needed to be added to the DOM, I got jumped back to having post 42 as my active post.

Edit: To expand on the replication case - once I've scrolled far enough to hit the page jump, I can keep scrolling in either direction in the thread successfully after that. Looking at the DOM while I'm fiddling, I've observed the following with 100% replication in that thread on Windows 10/Chrome 58-current:

• My "posts per page" setting is 20, though I have the "Paginate topics and posts instead of using infinite scroll" setting disabled. Setting the "posts per page" to a lower number changes the results of what posts initially load when I enter the thread at any given point, so I believe the infinite scroll process is still honoring that setting to a degree.
• If I enter at the top, posts 1-20 are pre-loaded in the DOM, and once I get to where it needs to load the next batch of posts to prepare to show post 21, I get jumped to post 28. After the jump, posts 24-63 are in my DOM.
• If I enter at the bottom, posts 54-63 are pre-loaded in the DOM, and once I get to where it needs to load the next batch of posts to prepare to show post 53, I get jumped to somewhere between post 42 and 44, depending on how fast I'm scrolling. After the jump, the 40 posts ending with the post I got jumped to are in my DOM.
• If I enter in the middle, or at any time after I've triggered one of the previous two replication cases, I can scroll around without further jumping. In those cases, the DOM will flip back and forth between posts 1-40 and posts 24-63, as necessary when I scroll back and forth to break across the limits of what's loaded in the DOM.

Edit 2: We do have two pieces of custom CSS related to the blockquote element in our site customizations:

    blockquote > blockquote, blockquote > p, .iframely-link {
        margin-bottom: 11px;
    }

    blockquote > blockquote:last-child, blockquote > p:last-child, .iframely-link:last-child {
        margin-bottom: 0;
    }

[RaceProUK]

I've done a bit of experimentation myself, and the Post Per Page setting appears to have an influence on how severe the issue is. Originally, I had it set to 20, and was seeing the issue fairly regularly. But when I upped it to 50, it happened less often. It still happens, but the larger 'page' size reduces the likelihood of hitting it.

Not suggesting this as a solution of course, but it may help the investigation.

[boomzillawtf]

https://what.thedailywtf.com/post/1155477
Where DCoder said:

When I had Posts per Page at 50, the "scroll down" case didn't cause problems, "scroll up" did.
After reading the other replies, I changed that setting to 20, and both cases caused problems.
I'm using Chrome 58.0.3029.110 (64-bit) on Windows 10.

After some tinkering, I see two different problems here:

1. The client sends a request "give me 20 posts (the page size) in this topic after post #19", but the server sends back a lot more than 20 (I got 45 = all the remaining posts in that topic). The client adds them all to the DOM and then removes the old posts keeping only the latest 40 posts, which removes the posts you were reading and a few more. :facepalm:
   It seems to me that this pull request is the culprit – it adds a count parameter to the request but looks for postsPerPage when handling it, and it doesn't de-hardcode infiniscroll's fixed page size of 40 posts. Whenever the list of visible posts changes, it only keeps that many posts in the DOM. The request fetches >40 posts, adds them to the page, then this trimmer removes all but the first/last 40 posts, and it all jellypotatoes apart.
2. When scrolling up, the client tries to keep your scroll position unchanged by measuring it in the old DOM, prepending the new posts, calculating the new target position, scrolling to it, and then removing the old posts. While it's scrolling, the browser raises the "hey, the window was scrolled" event, which the infinite scroll uses to decide if it needs to load more posts. Chrome (and probably other browsers these days) does its own bookkeeping for scroll position, which sometimes tricks this behaviour into thinking it needs to load more posts again, doubling the jellypotato.

I'm playing around with this trying to follow what's going on, and this all seems like where I'm headed.

[AzariahAeroborn]

A comment from DCoder on the thread on our forums where we've asked people to specifically replicate my test case (of which we have 100% reproduction rate across 10 people including Chrome and Firefox on a variety of Windows and Linux platforms):

When I had Posts per Page at 50, the "scroll down" case didn't cause problems, "scroll up" did.
After reading the other replies, I changed that setting to 20, and both cases caused problems.
I'm using Chrome 58.0.3029.110 (64-bit) on Windows 10.

After some tinkering, I see two different problems here:

The client sends a request "give me 20 posts (the page size) in this topic after post 19", but the server sends back a lot more than 20 (I got 45 = all the remaining posts in that topic). The client adds them all to the DOM and then removes the old posts keeping only the latest 40 posts, which removes the posts you were reading and a few more
It seems to me that this pull request is the culprit – it adds a count parameter to the request but looks for postsPerPage when handling it, and it doesn't de-hardcode infiniscroll's fixed page size of 40 posts. Whenever the list of visible posts changes, it only keeps that many posts in the DOM. The request fetches >40 posts, adds them to the page, then this trimmer removes all but the first/last 40 posts, and it all jellypotatoes apart.

cc: @julianlam @barisusakli

[boomzillawtf]

I can confirm that commenting out this line stops the crazy navigation:
https://github.com/NodeBB/NodeBB/blob/master/public/src/client/topic/posts.js#L195

Of course, then it doesn't unload posts, either.

[AzariahAeroborn]

Issue is here: https://github.com/NodeBB/NodeBB/pull/5520/files#diff-fc4518b45c06d46136dae8d0e5a68cae

When the socket emits a request for loading the next chunk of posts, it emits:
42167["topics.loadMore", {tid: 10675, after: 71, count: 5, direction: 1, topicPostSort: "oldest_to_newest"}]

But the server is looking for a variable of postsPerPage, not count, and it winds up sending back the server configuration's maximum Posts Per Page setting, because it "didn't get a valid request" from the client. Which is why we're replicating so much more often on our home forum - our max setting there is 50, which overwhelms the hardcoded 40 in the infinitescroll.removeExtra function that is referenced in the preceding post.

[boomzillawtf]

I can reliably reproduce in the "Who is using NodeBB?" thread.

https://community.nodebb.org/topic/180/who-is-using-nodebb/1

• Make sure pagination is turned off.
• Posts per page = 20
• Use the "page down" arrow in the nav tools multiple times.
• Eventually, the navigation jumps. When I did it, I ended up at 50 out of 413. It looks like I'm at the bottom of the thread and nothing below that post loads.
• Click page up. Looks normal.
• Click page down. Now I'm at post 79 or 413, again apparently at the bottom.

[julianlam]

Looks like you guys hit on something 😄

[barisusakli]

OK seems like the fix here is to respect the count setting sent from the client, and change the hard coded infinitescroll limit of 40 to be itemsPerPage x 2 I will make those changes and then we can test again.

[AzariahAeroborn]

Should there also be a change to the copy of the Settings > Pagination options, to provide an indication that the setting is now effecting the way that your infinite scrolling is working?

[barisusakli]

Can you guys try with the above commit.

[barisusakli]

@boomzillawtf the reason why page down isn't triggering a infinitscroll quickly is properly this code https://github.com/NodeBB/NodeBB/commit/06cd4cd4067226991b44ae60282f34c94376ced1, it blocks infinite loads to 1 per second.

[AzariahAeroborn]

@barisusakli We deployed the fix a few hours ago, and I'm no longer having the described issue in any of the replication cases I was using before.

[barisusakli]

@AzariahAeroborn 👍

[sorvani]

On my phone at the moment and not going to poke through the reports, but I thought this had been posted before.

We see this all the time over at mangolassi.it in long threads like this one: https://mangolassi.it/topic/1022/what-are-you-doing-right-now

[julianlam]

Can anybody from tdwtf confirm this is resolved?

[RaceProUK]

As far as I can tell, it's resolved. @boomzillawtf, @AzariahAeroborn, do you want to confirm?

[boomzillawtf]

Yes, I'm back to infiniscroll and haven't had any problems.

[julianlam]

Thank you 😄