Mastodon: Replies sometimes not inserted into home timeline

[ThibG]

Replies between two remote users are sometimes not inserted into home timeline.
This is because toots can arrive (or be processed) out-of-order, and a reply is only inserted into the Home timeline if the toot being replied to is already in the database.
Unfortunately, I'm not sure how to fix that, as we still want the messages to appear in logical order, and the ThreadResolver will work in the opposite order…

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [ ] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[ThibG]

Since it's a bit of a mess, I'll describe the current flow regarding receiving, fetching, and insertion into the Home timeline.

Toots are inserted into timelines by FanOutOnWriteService, either directly, or either through FeedInsertWorker. FanOutOnWriteService is only called within DistributionWorker which is itself spawned when boosting or posting from the instance itself. It is also spawned when processing remote toots, either incoming or by fetching them, if they are recent enough (Status#within_realtime_window?).

So basically, replies are pushed to the timeline if they are recent enough (≤ 6h) and their parent has already been processed when they are first discovered.

[Gargron]

Theoretically you could take a minor performance hit and fetch inReplyTo straight away (non recursively) if it's missing, do NOT save it in DB but only save in_reply_to_account_id data. That way, thread resolving can run its own course but FeedManager can make correct decisions about showing/not showing the toot in home feed.

[ThibG]

I'm not too sure about doing an additional HTTP request, Mastodon already does far too many (another thing I have to look into)… furthermore, this won't prevent toots from appearing out-of-order in the TL (although, I wonder whether having them out-of-order is worse than missing them entirely)

[Gargron]

They won't be out of order though. Perhaps only momentarily by arriving via the streaming API, since the web client just prepends them at the top. But home feed is a zset and we have snowflake IDs.

[ThibG]

Oh, crap. That's both a good thing and a bad thing :<
If I understand correctly, the snowflake ID is created based on the created_at value of the toot, which used to be the time at which it was received, and is now the time at which it is was originally published (which is better for ultimately having the toots in-order, although it's not a guarantee, as different instances' time can be skewed).
But now I'm worried (again) about my previous PR: if the API paginates based on snowflake ID and they don't necessarily follow the order in which they were processed… isn't that an issue?

[ThibG]

So, due to how statuses are expected to be ordered by snowflake ID, it seems like ensuring in-order appearance of toots will not be possible. But I think we may still want to insert out-of-order toots in the timeline if they would be visible had them arrived in-order.

As an aside, consistent ordering across instances is not possible in general, what I'm calling “in-order” and “out-of-order” in this message and above ones is relative to the preorder induced by the in-reply-to relation: while we do not know in which order toots are in general, we can know that a reply is ulterior to the status it is a reply to, even if those statuses were retrieved/received in the opposite order.

[GambaJo]

I noticed this too. Very confusing.