Mastodon: Custom instance not synchronizing

[Korko]

Hi,

I tried to add my own custom instance of mastodon but it seems like it never sync. The global timeline is empty, I follow several people who posted since I followed them and my home is totally empty.

Is it a problem with iptables?

What can I do?

Thanks

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.

[rhaamo]

Hi,

As a first try, could you try if you are able to access to the URLs /.well-known/host-meta and /.well-known/goldfinger with something like Curl ?
It should returns XML. Theses URLs are handled by the mastodon server.

Let's encrypt also use .well-known and some users have alias for that dir, causing issues.

[Korko]

On my own instance, I have https://mastodon.k0rko.fr/.well-known/host-meta -> 406 https://mastodon.k0rko.fr/.well-known/goldfinger -> 404 (from Mastodon)

[rhaamo]

Ok good. I would say to check if workers are ok (sidekiq logs -> systemd journald), and no SSL issues if you are using HTTPS (like an incomplete chain).
I don't have more ideas right now except the basic steps required to make the instance communicate.

[Korko]

Thanks @rhaamo for your help.
After upgrading to the last version I have some of the posts but it's rather strange. For example, I follow a guy on mastodon.social. I have an account on the same instance (mastodon.social) and his posts doesn't appear in my home there. While I have an account on my own instance and it does appear while other posts from other people I follow on both instances aren't visible on mine but are on mastodon.social...

[selectric401]

I'm having a similar issue -- .well-known/goldfinger doesn't return anything, and .well-known/host-meta returns some snippet of XML. I've got an account on mastodon.xyz, and it can't follow any test accounts on my own instance.

[emilweth]

Same issue here, https://pouet.me/.well-known/host-meta returns a 406 and https://pouet.me/.well-known/goldfinger a 404

[yipdw]

A few clarifications:

The /.well-known/host-meta route requires that the client identify itself as accepting XML, or it will respond with 406 Not Acceptable. The /.well-known/webfinger (note: not goldfinger; Webfinger specifies the former route) route requires a resource parameter; if the resource cannot be found, it will respond with 404 Not Found.

As of 2017-04-05 @ ~0500 UTC, these curl invocations worked on the above-mentioned instances:

$ curl -H 'Accept: application/xml' https://pouet.me/.well-known/host-meta
<?xml version="1.0"?>
<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
  <Link rel="lrdd" type="application/xrd+xml" template="https://pouet.me/.well-known/webfinger?resource={uri}"/>
</XRD>

$ curl -H 'Accept: application/xml' https://mastodon.k0rko.fr/.well-known/host-meta
<?xml version="1.0"?>
<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
  <Link rel="lrdd" type="application/xrd+xml" template="https://mastodon.k0rko.fr/.well-known/webfinger?resource={uri}"/>
</XRD>

A Webfinger client can then use this information:

$ curl -H 'Accept: application/xml' 'https://mastodon.k0rko.fr/.well-known/webfinger?resource=acct:[email protected]'
<?xml version="1.0"?>
<XRD xmlns="http://docs.oasis-open.org/ns/xri/xrd-1.0">
  <Subject>acct:[email protected]</Subject>
  <Alias>https://mastodon.k0rko.fr/@Korko</Alias>
  [rest redacted for brevity]

So Webfinger seems ok.

[ludovicchabant]

Same problem for me with the instance I just setup. The host-meta returns what looks like the correct XML code (referencing webfinger), and then I can... ahem... finger myself and get the XML code describing the various feeds URLs for my account.... but my timelines are desperately empty.

The sidekiq job doesn't seem to be doing much, so that could be related, but it does mention webfinger'ing some users I follow on other instances, and it does mention downloading atom feeds for those. I'm not sure what could be wrong there.

For those getting 406/404 statuses, it may be because you're using a browser like Firefox to request those things -- that would fail (it does for me at least). But it works if you use curl or something else that, I assume, declares the proper Accept header.

[KillianKemps]

Same issue here. I have setup Mastodon with Docker and the curl on /.well-known/host-meta and /.well-known/webfinger work as expected.

But, I can only see my own messages in the timelines.

[ludovicchabant]

So according to this comment from @Gargron it sounds like you're only supposed to see the _next_ statuses posted by the people you follow? That could explain why my timelines are empty since I bet nobody I follow posted since yesterday evening...

I thought Mastodon was downloading the Atom feed of each followed user in order to include, right away, the recent messages from those users? But maybe that's not how it works right now?

__Update__: nope. Started following @Gargron (@Eugen) but nothing shows up either even though I can see a few new posts in the last hour.

__Update 2__: I found out that my sidekiq job was stalled for some reason! I'm not sure what happened but I restarted it and all of a sudden it did a whole bunch of requests and lo and behold my timelines filled up (including old messages). So that resolved it for me.

[KillianKemps]

I have restarted sidekiq and I changed the LOCAL_HTTPS env var to true and it seems that the instance begins to federate slowly.

As the change come very slowly, I'm not sure which action had an effect. I just kept restarting the containers, and remote following people.

[duke-]

Hi there,
I'm having nearly the same issue. I get notification from remote users, but I cannot see their toots.
No errors in any logs, webfinger etc. works fine. The remote user gets my toots, no problem there ...
Any suggestions?

Issue resolved!
Unfollowing/Refollowing did the trick.

[alexgleason]

This looks bad. Not sure what to do.

EDIT:

I did just run

docker-compose restart redis
docker-compose restart sidekiq

this seemed to resolve it.

[dunn]

Tried all the above, and no luck. I'm assuming I should see entries in the PuSH view when federation is working?

Possibly related is that Amaroq POST requests are apparently getting translated to GETs: https://mastodon.social/@catte/2120596 I can manually curl -X POST my instance without issue, so I don't know if this is a subtle NGINX configuration issue, or SELinux, or something else.

NGINX configuration: https://github.com/dunn/ansible-mastodon/blob/master/roles/mastodon/templates/app.conf.j2

[dunn]

In my case, the problem was doing a 301 for all requests to the bare domain, sending them to www. Neither Amaroq nor PuSH were following those redirects correctly.

[drequivalent]

Have the same problem. What to do - beats me, as webfinger and host-meta both seem to be available

[yookoala]

Have any users in your instance followed anyone on another instance?

[drequivalent]

I followed @Gargron, and now I do get toots from him, but the previous messages did not download.

[yookoala]

@drequivalent Mastodon doesn't do synchronization. It does subscription.

When any of your user requested, your instance subscribes to user@instance that it knew about. The user@instance you or any user follows will be sent by that instance to yours after the subscriptions.

Federated timeline is the collective result of the subscriptions of your users.

It is the expected behaviour.

[drequivalent]

Okay, may be that. But how do you explain @ludovicchabant 's case, when it downloaded old messages?
Also, I see now newer messages in global feed than I see in local feed. Is that normal?

[yookoala]

@drequivalent Sorry. I can't read the language. But I think older messages will be downloaded if either,

1. There are other users subscribed to that user earlier.
2. There are some user in federation boosted that old message.

[bnjbvr]

Hitting the same issue here. I can understand the logic that only new messages get after subscribing to a particular user; but I don't understand why followings/followers wouldn't show up in that case: this is either strange UX, or there's an actual bug underlying there.

[yookoala]

@bnjbvr I do agree the UX is a bit confusing. From experience,

• The "follower" count only counts the number of followers (to that remote account) in your instance.
• The "following" count only counts the number of account that the remote account is following on your instance.

[Ulrar]

I think I have a similar problem, I installed an instance earlier today and I started following a few users on remote instances (including Gargron), and I can see on mastodon.social that they tooted stuff since but those toots never make it to my instance.
Something must be working though since someone I don't know from mastodon.social just followed me, I must have somehow showed up over there, but I don't see anything from them.

I've looked at the logs, no errors, sidekiq isn't even doing any retries or anything, it's just saying 21 success right now.
The only exotic thing is that I have a different LOCAL and WEB urls, but I've tried webfinger and my redirections seem to be working fine, I can "finger" myself, if I may put it this way. I even saw a few successfully webfinger requests in my front nginx access logs over the hours.

I imagine the PuSh menu is callbacks for people following me, right ? Since someone followed me, I have one entry there (the delivery column has a little X in it, I imagine that's because I didn't toot anything).

[alexgleason]

@ulrar this is actually by design. History does not begin until the first time someone is followed on an instance.

[Ulrar]

Yes, I realise that, what I'm saying is nothing appears, ever. @Gargron did toot stuff since I followed, I can see it on https://mastodon.social/@Gargron
And my federated timeline doesn't fill up either, I can only assume mastodon.social should be pretty active .. I can't even see anything in my access logs, no callbacks at all after the intial webfinger (that does return 200)

[yookoala]

This is probably more a UX issue than a program bug. As a programmer who read a bit of OStatus, PuSH and Salmon, I understand how this subscription work. But there is a gap between how the program is working and how user expects it to work.

1. Profile Page

Right now, profile of remote user (in local instance) only reflect local statistics:

• follower of local instance
• follows to local instance
• posts sent to local instance after first subscription in local instance.

However, as a layman, one would expect to see the same profile of certain "@user@instance" even if it is displaying on a remote instance. So even if I search a remote user on my instance, one would expect to see the full profile and statistic:

• follower of the origin instance.
• follows stored in the origin instance (hence include his remote follows).
• posts in the origin instance (maybe only get older post on pressing "load more").

2. Home and Federated Timeline

Users would expect to see posts, before and after subscription, of a remote user in "Home".

I think the best way to solve the UX is have better description in "Remote Follow" page about the behaviour.

[Ulrar]

@yookoala Yes, all those are valid points, and I agree that's it's a bit counter intuitive, it should be made clearer.

But that's unfortunately not my problem, there is simply an actual bug, see #2672

[Gargron]

I am afraid that this issue at its core is a misunderstanding of what is supposed to happen.

Mastodon doesn't do synchronization. It does subscription.

Mastodon stores only data that is demanded, e.g. you follow someone, you start getting their new content after the follow. If somebody you follow shares an older status, it is pulled down. But not otherwise. This also includes follower/following lists. There are a couple issues if we were to make them work like you expect them to work. One is verification. As long as it's just displaying your instance neighbours' relation to that account, you can be certain the data is correct. If the profile was using some kind of provided number, it could be easily inflated. These lists can be quite huge, too. If you were to look up a profile of someone with a million followers, would you expect the instance you are on to go through and download the profiles of a million users? I think not. Mastodon aims for more organically growing database.

[Korko]

I think you misunderstand the problem and there's two problems here.

The first is still a problem. At least seeing the comments. Have not retested yet. You can see it explicitely in the message of Ulrar:

I started following a few users on remote instances (including Gargron), and I can see on mastodon.social that they tooted stuff since but those toots never make it to my instance.

The second is a misunderstanding and you explained it. So the problem here is NOT about toots posted BEFORE the subscription. It has to do with toots posted AFTER the subscription not reaching some instances.

Once again, I don't use Twitter nor Mastodon so I cannot confirm or infirm if the issue is still there but there was no misunderstanding for my part.

[Ulrar]

Yes, glad to see someone understood :)
As for the problem it was still there last time I checked, but I have to
admit I got tired and deleted my instance since, so I don't know if that
has been fixed since.

To test it just create an instance with LOCAL_DOMAIN != WEB_DOMAIN and
follow anyone on an other instance.

Le 4 juil. 2017 13:45, "Jérémy Lemesle" [e-mail] a écrit :

I think you misunderstand the problem and there's two problems here.

The first is still a problem. At least seeing the comments. Have not
retested yet. You can see it explicitely in the message of Ulrar:

I started following a few users on remote instances (including Gargron),
and I can see on mastodon.social that they tooted stuff since but those
toots never make it to my instance.

The second is a misunderstanding and you explained it. So the problem here
is NOT about toots posted BEFORE the subscription. It has to do with toots
posted AFTER the subscription not reaching some instances.

Once again, I don't use Twitter nor Mastodon so I cannot confirm or infirm
if the issue is still there but there was no misunderstanding for my part.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/864#issuecomment-312868866,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA1TzyBV_H2cGEvLNeCI7FCMimQnlVk2ks5sKjPxgaJpZM4MzJu4
.

[ludovicchabant]

I am afraid that this issue at its core is a misunderstanding of what is supposed to happen.

This is only tangentially related to this issue, but I want to point out that this misunderstanding seems quite common, and might therefore point to a broader UX issue. It may not be a bug, but it looks like a problem still.

For example, RSS clients, at their core, only do subscription too... yet, when you subscribe to a new website, they generally all download the last dozen posts and show them to you. Mastodon could do the same. That would even fix another big UX issue for me, which is this:

You see someone boost a toot from user FooBar. It's funny/interesting/whatever, and you wonder if you should follow FooBar. You click on their profile and, of course, the only toot there is the one that got boosted -- because, like you said, Mastodon doesn't have any other reason to download the other toots. But as a result, you don't know what FooBar's timeline looks like, or even if they posted anything. You have to go check it out on their instance and, if it's interesting, come back to your instance and follow them. This is not only a waste of clicks, but it's actually disruptive on mobile, where you have to exit your client app and go to Safari/Chrome/whatever to check FooBar's instance. This could be greatly improved.