Mastodon: can't see toots on people's profiles or replies to many toots from self-hosted instance

[chaosmonk1]

Expected behaviour

When I view someone's profile locally, I should be able to see their public toots.

If a toot appears in my federated feed, clicking on the toot should display public replies to it.

Actual behaviour

For many users, I can only see one or two toots when I view their profile on my local instance, but can see many toots when I view their profile on the remote instance.

Some toots don't appear to have any replies when viewed from my instance, but when I click on the date to view the toot on the remote instance, I see that there are replies.

Steps to reproduce the problem

If necessary I can link to my instance and to examples of profiles missing toots and toots missing replies.

Specifications

I am running Mastodon 3.1.4.

[SSPJ]

Same issue (?), also 3.1.4. The mastodon-web.service logs don't appear to have anything useful. This is what I get when I visit a profile page of a remote user I am following (line breaks for ease of reading):

... method=GET path=/api/v1/accounts/9/identity_proofs format=html \
controller=Api::V1::Accounts::IdentityProofsController action=index status=200 \
duration=11.66 view=1.45 db=5.84
... method=GET path=/api/v1/accounts/9/statuses format=html  \
controller=Api::V1::Accounts::StatusesController action=index status=200 \
duration=28.12 view=0.83 db=7.34
... method=GET path=/api/v1/accounts/9/statuses format=html \
controller=Api::V1::Accounts::StatusesController action=index status=200 \
duration=65.41 view=19.38 db=20.46

Some of the users I am following do not show up in the federated timeline at all, while others have just pinned toots, or (seemingly) random toots. None have the full public timeline that I can see when I visit the remote instance.

[ThibG]

This is expected, your instance only shows toots it knows about, and it won't pull toots from remote users. Some people suggest changing that (#1745) but it's not obvious how to do that correctly and efficiently.

We used to have the UI tell, on remote profiles, that they might be incomplete or outdated, but for some reason that part of the UI has been removed.

[chaosmonk1]

This is expected, your instance only shows toots it knows about, and it won't pull toots from remote users.

What causes my instance to "know about" a toot? I can see some toots by users, but when I compare those I can see to those that I can't I don't see any obvious pattern.

Also, are you responding only to the issue about missing toots on profiles, or are the missing replies to toots on the federated timeline also to be expected?

[ThibG]

Basically, your instance knows about toots if any of these is true:

• someone on your instance follows the person who posted, at the time it was posted
• someone you (or anyone else on your instance) follow boosts or replies to that toot
• under some conditions, if that toot is a reply to someone you (or anyone else on your instance) follow
• it mentions anyone on your instance
• it was a pinned toot when the author has been discovered by your instance
• someone on your instance manually fetches a toot (by pasting the toot to the search bar)

[chaosmonk1]

Basically, your instance knows about toots if any of these is true:

Thanks, that seems to largely explain the current behavior for profiles, though not entirely. On profiles, I can indeed see all pinned toots, but I can also see a few seemingly random toots, some new and some very old, that don't seem to meet any of the other conditions you listed.

Regarding replies, when a toot by someone I follow appears on my Home feed, it seems I can generally see all replies, whether regardless of who those replies are from and whether they meet any of the conditions you have listed. If the toot is by someone I do not follow, but appears in my Home feed because someone I did follow posted it, I can't see replies. However, following the author of the toot does not result in replies appearing, so it seems I must already have been following them in order to receive replies.

The issue with toots missing from profiles is a very minor issue, I don't really care, although it would be nice if there were some clarification in the UI. However, the issue with not being able to see replies is terrible. The majority of my Home feed consists of small fragments of conversations that I can't follow or participate in because the rest of the conversation is missing.

I can see how it would be inefficient to fetch all of the toots of every author of every toot or reply that appears in the local instance. However, is there a reason that Mastodon could not at least fetch replies to the toots that it displays? If that's impossible for some reason, is there a way to make it so that if Mastodon cannot access replies to a toot, it just doesn't display that toot at all? Displaying all these toots without also displaying the replies results in a frustratingly low signal-to-noise ratio.

[ThibG]

Yes, replies are forwarded to the followers by the followed person's instance when they arrive, they aren't fetched or forwarded afterwards. Fetching replies is expensive, and you'd need to do that each time the toot is displayed, not when it's discovered, because in most cases you'd discover it quickly after it's created, but replies would come in later.

Regarding replies, there is something we do: we try fetching self-replies when a toot is discovered, in order to fetch whole threads, but it has the limitation described above, that replies get fetched when the toot is discovered, which means the threads are only fully-fetched if they were published in their entirety when part of them was retrieved.

Not having all the replies can be an issue indeed, but in practice I have not found that to be so much of one. But maybe it is because I know how it works and will occasionally check for replies upstream instead.

[chaosmonk1]

Fetching replies is expensive, and you'd need to do that each time the toot is displayed, not when it's discovered, because in most cases you'd discover it quickly after it's created, but replies would come in later.

How does my instance fetch replies to toots by users I follow? What makes that less expensive than fetching replies to toots that are by users I do not follow but which appear in the federated timeline? Thanks for your patient explanations.

Not having all the replies can be an issue indeed, but in practice I have not found that to be so much of one. But maybe it is because I know how it works and will occasionally check for replies upstream instead.

What percentage of toots in your Home feed are affected? For me it is a majority, but perhaps this is a function of being a new instance that does not yet have a large number of local users following a large number of remote users. I am preparing to invite a number of interested people in my local community to join my Mastodon instance, and I am trying to smooth out in advance the UX issues I am running into in order to give them a good first impression. These missing replies are the last issue I am concerned about, but if this is not a common complaint then maybe I am worrying too much.

[ThibG]

How does my instance fetch replies to toots by users I follow? What makes that less expensive than fetching replies to toots that are by users I do not follow but which appear in the federated timeline? Thanks for your patient explanations.

It does not fetch them, the instance receiving them forwards them to your instance. So it gets them when they are posted, it doesn't have to decide when to fetch replies or do it multiple times, it just gets them when needed.

[chaosmonk1]

Okay, thanks. To make sure I understand correctly, if I am on instance A, and I am following a user on instance B, and they boost a toot by a user on instance C, then what happens is instance C forwards to B both the boosted toot and replies to that toot, but C does not forward anything to A, and instance B forwards the only boosted toot to A but does not forward replies as they come in?

If this is correct, would it be feasible for instance B to instead go, "okay, now that I've forwarded this toot to A, whenever C forwards me a reply to the toot I should also forward that reply to A"?

[ThibG]

No, in the scenario you described, C doesn't do anything fancy and A doesn't get replies to C's toot.

The forwarding mechanism forwards replies to a local toot to remote followers of the person who authored that toot. It doesn't concern itself with boosts, and instances do not keep track of who received/fetched their toots.

[chaosmonk1]

Ok, I understand now that the boost is irrelevant. The boost just causes the toot to appear in my Home feed, but even without the boost, A would still be aware of the toot and it would still appear in my federated timeline.

What I'm trying to understand is how the toot by the user on C made its way to A even though no one on A follows anyone on C, and whether replies to the toot can do the same. Is it true that the toot was first forwarded from C to B, and that B in turn forwarded the toot to A? If so, what were the set of conditions that caused B to decide to forward that toot to A? Would it be feasible for the same set of conditions will also cause B to forward to A any replies to that toot?

What I'm trying to get out, is that if instance foo forwards a toot to instance bar, then as foo becomes aware of replies to that toot it should forward those to bar as well. If it is for some reason not possible to for foo to ensure that bar will receive replies to the toot, then I think it should not forward the toot to bar in the first place.

[ThibG]

Is it true that the toot was first forwarded from C to B, and that B in turn forwarded the toot to A? If so, what were the set of conditions that caused B to decide to forward that toot to A?

If the toot from C was a reply to someone on B followed by someone on A, the toot has been sent to B from C and then forwarded through the mechanism I described above. Toots are only forwarded once at most by Mastodon. And the mechanism I described is the only one used for forwarding toots in Mastodon (there are other mechanisms for content deletion but they are more expensive—maybe even too expensive).

Forwarding to more people than your direct followers would be possible but more expensive for each instance involved. Worse, it would mean that for a long thread of toots written by multiple accounts, intermediary instances would likely forward duplicated copies of each reply, which is a waste.

Furthermore, this wouldn't solve the issue when boosts are involved. You could then extend the forwarding rules for boosts as well, but it gets even more expensive. Prohibitively so.

[chaosmonk1]

Aghhh, sorry for dragging this out, and thanks for your patience.

So to clarify:

• Say I am on instance example.com, and the only person that I or anyone else on example.com follows is [email protected]. Alice follows [email protected].
• When Bob publishes a toot, mastodon.online fowards the toot to mastodon.social, where it shows up in Alice's federated timeline.
• She boosts or replies to Bob's toot, at which point mastodon.social fowards Bob's toot and Alice's reply or boost to example.com, where it appears in my federated timeline.

• Then someone else replies to Bob's toot, and mastodon.online forwards that reply to mastodon.social where Alice sees it.

• The current behavior at this point is that mastodon.social does not forward the new reply to example.com, which is why I don't see it. I behavior I was suggesting is that mastodon.social instead would forward the new reply to example.com. In other words, currently mastodon.social says, "Okay, Alice interacted with Bob's toot and someone on example.com follows Alice, so I had better forward Bob's toot and Alice's boost or reply to example.com so that they can see it", and my suggestion was that mastodon.social would also say, "Additionally, from now on when I become aware of a new reply to Bob's toot, I will forward that reply to example.com as well".

• The problem is that forwarding these replies would be too expensive for mastodon.social and example.com?

[ThibG]

* When Bob publishes a toot,  `mastodon.online` fowards the toot to `mastodon.social`, where it shows up in Alice's federated timeline.

I would say “sends” rather than “forwards”, as the toot originates from mastodon.online and not something else, but yes.

* She boosts or replies to Bob's toot, at which point `mastodon.social` fowards Bob's toot and Alice's reply or boost to `example.com`, where it appears in my federated timeline.

Technically, mastodon.social doesn't forward the toot to example.com, just sends the boost/reply, which references the toot on mastodon.online, and upon receiving the boost/reply, example.com resolves that reference by fetching the original toot.

* Then someone else replies to Bob's toot, and `mastodon.online` forwards that reply to `mastodon.social` where Alice sees it.

Yes.

* The current behavior at this point is that `mastodon.social` does **not** forward the new reply to `example.com`, which is why I don't see it.

Yes.

In other words, currently mastodon.social says, "Okay, Alice interacted with Bob's toot and someone on example.com follows Alice, so I had better forward Bob's toot and Alice's boost or reply to example.com so that they can see it",

not really, Mastodon just sends the reply/boost to Alice's followers and does not perform any extra step here, it's the receiving end which discovers a new reference and tries fetching it

and my suggestion was that mastodon.social would also say, "Additionally, from now on when I become aware of a new reply to Bob's toot, I will forward that reply to example.com as well".

* The problem is that forwarding these replies would be too expensive for `mastodon.social` and `example.com`?

Yes, that last suggestion would involve, for each received toot, check all the ancestry and for each of those ancestor toots, check who they have been boosted by, etc., which is expensive. Additionally, it would mean that multiple servers would ultimately forward you the same messages, which is a waste. And with all of that, it wouldn't even guarantee the forwarding would always work: for instance, consider what happens if Alice stops following Bob, or she unboosts his toot, or someone gets blocked along the way, etc. You'd stop getting replies.

[chaosmonk1]

Okay, that makes sense. Thanks for explaining.

I think the issue is that the UI creates an expectation of different behavior from what actually happens. I was hoping that the behavior could be changed to match the UI, but since it sounds like this is not feasible, I think the next best thing would be to change the UI to match the behavior.

Currently, the simplest possible interaction with a toot (clicking on it) displays some replies, and there is nothing to indicate that other replies are missing and can only be viewed on the remote instance. This creates an expection of a very simple behavior (when I click on a toot, I can see the replies). It is not obvious that there are additional replies that can only be seen by clicking on the date of the toot, and when the user eventually discovers this behavior, it is unclear why this is the case for some replies and not others.

In situations where Mastodon presents potentially incomplete information (replies to a toot, toots on a remote user's profile) it would be helpful if the UI indicated ideally

• what subset of the information is being shown
• what subset of the information is not being shown
• how to access the latter subset

or if it is too difficult to concisely summarize what information is/isn't shown, at least

• that some information may be missing
• how to access it

A simple form that this could take is a "view all toots" button on a remote user's profile, which links to the user's profile on the remote instance, and a "view all replies" on a toot by a remote user, which links to the toot on the remote instance.

[kuba-orlik]

@ThibG

Yes, that last suggestion would involve, for each received toot, check all the ancestry and for each of those ancestor toots, check who they have been boosted by, etc., which is expensive

I don't know much about Masto infrastructure, but in this scenario, can't the replies be just "pulled" from the original instance? I mean, with some coding duct tape they could even be scraped from the web interface of the orig instance

[ThibG]

Yes, Mastodon could theoretically pull replies (some of them at least) from the original instance (with some added machinery). But that expensive. And the question is when and how often? If you don't do it every single time someone clicks on the toot, you are likely to miss some replies. And even then, doing it at that moment will often be “too late” unless we add an asynchronous API for fetching replies and accept that the UI would suddenly have a bunch of new posts seconds after displaying the initial post.

[kuba-orlik]

Hm, I see. It's a shame though, because it makes it harder for small instances to view the entire discussion.

Maybe an indicator saying "some replies might be missing" and a "pull more replies" button would be sufficient?

[Krinkle]

I'm encountering the same issue on polyglot.city (on masto.host), and observe it both when viewing a profile after searching for it, and after having followed them. I tried this in a web app (Pinafore), and in two mobile apps (Amaroq, and Tootle).

I was nearly sure this was a bug in one of the apps, until I realized that other apps suffer the same issue. I then thought it was a problem with Masto Host, but someone pointed me here, that it is in fact a more general issue that just happens to affect smaller instances the most.

| Web | Amaroq app | Tootle app
|--|--|--
| | |

I can sort of understand and explain to people around me, that replies and like/boost counts can be incomplete when looking at a specific toot, if that toot is from someone else, outside your local group.

However, I am unable to explain to people why it is that Mastodon lets them paste/search for any account, and click on accounts of boosted toots or comments, but then cannot "just" browse those account's profile and public toots. We let them click on it and show recent meta data, but then the profile contains only a handful of random/old toots. I think this might be negatively affecting adoption and growth among communities as it means accounts look unappealing or abandoned.

I don't know much about the inner workings of Mastodon. I can totally see how it might be infeasible to federate these ahead of time, and undesirable to locally federate these on-demand when the user isn't followed. But, that suggest to me a few things:

• Perhaps we should not consult the local database at all then when browsing a profile unless it is followed. Instead, use an upstream API query, possibly cached locally in some way. (Either naively, by query; or normalized into an efficient internal structure separate from the timeline data, with stronger cache expiry or something).
• Once a new account is followed, we either import "everything" (unlikely), or import just one toot and remember in some way the timestamp of their last public toot and/or when they were followed by non-zero accounts, and for any incomplete query crossing over that date, do the same as for unfollowed accounts.

I also note that we don't really promote web links to the public profile, and on mobile apps these links are often omitted from the interface entirely. I suppose an alternate solution could be to encourage apps to short-circuit links to non-followed accounts and have them point to public account URLs always (thus for apps, switching to a browser and removing the ability to easily follow them without resorting to separately logging in and/or tech-savvy gestures like copying URLs into clipboards), where some apps might choose to instead query upstream APIs directly from the client-side (which might be undesirable in terms of privacy and performance/caching/latency). If this direction is preferred, it might make sense to deprecate or otherwise emit a warning to apps when querying profile data for accounts that are not followed an known to be sparse/outdated.

See also:

• "proxy pagination", ref https://github.com/tootsuite/mastodon/issues/14023#issuecomment-689026176.
• "replies missing", ref https://github.com/tootsuite/mastodon/issues/3794.
• "backfill upon first follow", ref https://github.com/tootsuite/mastodon/issues/34.

[chaosmonk1]

Ok, I've spent a few months keeping an open mind and trying to get used to this behavior, but I still think that this is a serious problem, at least for small instances. The way I've adapted is to just always open toots and profiles via the remote instance in a new tab. I don't feel comfortable replying to a toot until I have done this, because otherwise I don't know what context I might be missing. At that point, Mastodon might as well just proxy all of the replies from the remote instance in the first place. The added latency would surely be less than the time it takes to open up the remote instance in a new tab and load its web interface plus all of the replies.

[Krinkle]

I too open profiles in a new tab via the browser (outside any app or web app I use), and then copy-paste the urls of the latest 10 toots into the local search field on my home instance so that they get imported and show up for the next person.

Whatever reservations we might have about how importing affects or doesn't affect the federated timeline, it seems like this is something Mastodon could faccilitate to do more easily and is effectively already happening. Both by hand as I do, as well as for pinned toots. Perhaps in through the same handling as for pinned toots that are updated/imported, it could e.g. also fetch the last 10 toots on-demand and import them if not already present.