Mastodon: v1.3+ does not send toots to instances using a separate WEB_DOMAIN

[ghost]

EDIT: #2111 could be causing this. See comment https://github.com/tootsuite/mastodon/issues/2672#issuecomment-298348460.

Original issue:

---

Toots from a number of remote instances stopped appearing on my instance. Notifications remain working. Toots originating from my instance reach other instances just fine.

My instance is on v1.3.2, running on a separate WEB_DOMAIN. Logs show nothing particularly suspicious.

The problem started a couple of days ago, around Apr. 29. The problem appears to be instance-wide, not limited to individual users / toots, unlike #2613 . Manually refreshing PuSH, and re-following users did not fix the problem. Instances from which toots are missing (that I know of) range from v1.2.2 to v1.3.2.

Could someone please shed some light on why and how this is happening? Thank you!

---

• [x] I searched or browsed the repo’s other issues to ensure this is not a duplicate.
• [x] This bug happens on a tagged release and not on master (If you're a user, don't worry about this).

[s7b4]

Hi, same here.

I running an instance on a separate WEB_DOMAIN.

• version 1.3.2
• docker installation with docker hub images
• daily run of media:clear and push:refresh

Since 1.3.1 or 1.3.2, new remote subscriptions are not working anymore.

[ghost]

So I'm looking through the release notes, and found this PR in v1.3: #2111 , which might be the culprit:

# in DistributionWorker#perform
# https://github.com/tootsuite/mastodon/blob/501514960a9de238e23cd607d2e8f4c1ff9f16c1/app/workers/pubsubhubbub/distribution_worker.rb#L18
    Subscription.where(account: account).active.select('id, callback_url').find_each do |subscription|
      next unless domains.include?(Addressable::URI.parse(subscription.callback_url).host)
      Pubsubhubbub::DeliveryWorker.perform_async(subscription.id, payload)
    end

Here, subscription.callback_url likely contains WEB_DOMAIN, while account.domain is LOCAL_DOMAIN. That should prevent toots from going out to instances where LOCAL_DOMAIN != WEB_DOMAIN.

I'm not entirely sure, as I'm completely oblivious of how the magic that is api_subscription_url works. Running it from irb certainly didn't do much. Hope someone better versed at Rails could take a look at this.

That one v1.2.2 instance I encountered though, I still have no idea. Maybe it is running code off some master commit that still report itself as v1.2.2 before v1.3 was tagged.

[Ulrar]

I have that exact same problem, I thought I messed up my config somehow, glad to see it wasn't a problem on my end :)
Available to test a patch if needed

[ghost]

The OStatus/PuSH subscription request is in essence anonymous, so checking callback_url is really the only way if we were to preserve the 'only push to follower domains' semantic. However, we can GET /.well-known/host-meta to get a list of domains than can act 'on behalf' of the followers' domains, fixing this issue.

@Ulrar
I might attempt a patch if the maintainers would agree. Not seeing much attention here, though. WEB_DOMAIN is so yesterday now there is #2668 aka. The Future. Might as well move your instance to LOCAL_DOMAIN and refresh PuSH if you want it working fast.

[Ulrar]

Oh I don't really care about working fast, I'm the only one on it and I've never used twitter so I don't have an actual need, just want to see what the fuss is about. If it takes a while it's fine :)
But happy to help if I can

[saper]

@TacyT would hold my breath for #2668 and I also believe that any successful implementation of it will depend on working WEB_DOMAIN.

So yes if you can please try to fix it as it stands now! Thank you for your analysis!

[dariusk]

I'm on a separate WEB_DOMAIN and this makes my whole Mastodon experience unusable.

@TecyT If you're working on a patch I would love to see it and/or help if possible.

[ThibG]

I didn't notice, probably because the largest instances I'm connected to have a lingering remote account from when I was using my WEB_DOMAIN as LOCAL_DOMAIN (which causes all sorts of other issues), but I'm definitely affected.

2668 is still a long way and will have the same kind of issues, if not worse.

I have opened a pull request to:

• Push public toots without trying to check anything, since they are, well, public.
• Also extend the list of domains with follower's remote_url domain, as that's what contains their WEB_DOMAIN. It should fix this for Mastodon instances, but that's still making some assumptions not specified in OStatus.

[ghost]

@ThibG
Wow, thanks. That was so quick! I agree that this is making assumptions not specified in OStatus, but #2111 did that with filtering domains anyway, so it is a kind of necessary evil if we are to keep the semantic. It is going to be something temporary after all, as #2668 would supposedly come up with a side channel for this sort of information.

[ThibG]

@TecyT sure, it doesn't make more assumptions than #2111 added, but I'm a bit uneasy on having those assumptions at all. I can't see how #2668 would help, I don't expect it to add some side-channel for delegation or private messaging information, just make sure Mastodon makes sane assumptions, and is able to serve users from multiple domains.

[dariusk]

Any updates on this issue? As a WEB_DOMAIN user, my instance can't see 99% of Mastodon and it's... really not good. Do I need to migrate away from that feature? @ThibG, would this have drastic federation implications if I did? Basically I'd like to know how screwed I am in case this doesn't get fixed.

Edited to add: I see the PR was merged, which is great! Is there a way to tell which tagged release this will be in and when it's coming out? Sorry, I'm still relatively new to Mastodon development and I haven't seen the release process documented anywhere.

[ThibG]

@dariusk if you only change WEB_DOMAIN to match your LOCAL_DOMAIN, I think you should be mostly fine; as all URIs will change and you will be considered as a new user (hence, you'd have to re-follow everyone, and your followers would need to re-follow you). Incoming salmon requests (direct messages, follow requests and other notifications) from remote instances that knew you will be broken, until they refresh your account details (which is not done consistently, and at most once a day). Outgoing toots should be fine at least until the PuSH subscription expires.

If on the other hand you want to change your LOCAL_DOMAIN to match your WEB_DOMAIN, you will be considered as a new remote user, and the actor URI will clash with a “dangling” remote user account for your old acct: URI… I am not sure how much of an issue that would be… in both cases, you should be able to see other peoples' toots, though.

Anyway, I really hope this issue gets fixed soon, as I'm also badly affected, and I think the WEB_DOMAIN feature should not be dropped.

[ghost]

@ThibG @dariusk You won't be considered a new user as your LOCAL_DOMAIN
didn't change. You only need to forcefully refresh PuSH to see toots get
in. Outgoing toots would just work, as those are PuSHed by you to remote
URLs, which didn't change, not the reverse.
On Sat, 6 May 2017 at 00:21 ThibG [e-mail] wrote:

@dariusk https://github.com/dariusk if you only change WEB_DOMAIN to
match your LOCAL_DOMAIN, I think you should be mostly fine, as all URIs
will change and you will be considered as a new user (hence, you'd have to
re-follow everyone, and your followers would need to re-follow you). If on
the other hand you want to change your LOCAL_DOMAIN to match your
WEB_DOMAIN, the URIs will clash with a “dangling” remote user account for
your old acct: URI… I am not sure how much of an issue that would be… in
both cases, you should be able to see other peoples' toots, though.

Anyway, I really hope this issue gets fixed soon, as I'm also badly
affected, and I think the WEB_DOMAIN feature should not be dropped.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/2672#issuecomment-299509951,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AC_141SNdIrxCG9a9uVsmn52lpe1FCimks5r20yIgaJpZM4NM4u_
.

[ThibG]

@TecyT is right, if you change WEB_DOMAIN to match your LOCAL_DOMAIN, you won't be considered a new user, but all your URLs and URIs except your acct: URI will change. The part about being considered a new user applies to the other case. You will need to renew your PuSH subscriptions with your new callback URL, as @TecyT said. Outgoing salmon requests should be fine, too. But you probably won't get incoming salmon requests before a while as remote instances would have to refresh your info, and they do not do that consistently. Also, PuSH subscriptions from remote instances might not get correctly renewed.

[TheCapsLock]

My instance stopped receiving updates for a lot of instances for about 17 days ; I added LOCAL_DOMAIN to .env.production.yml with same value as WEB_DOMAIN and restarted all of the services but it has not fixed anything. My Instance is homed at social.cloudfrancois.fr ; running on latest upstream commit

[ThibG]

@TheCapsLock changing the LOCAL_DOMAIN is the kind of migrations you shouldn't do, as explained in one of my earlier message. If you really want to stick to this, though, it is not enough: you also need to re-subscribe to the remote users you are following.

[TheCapsLock]

@ThibG ok ; but What should be done to fix this issue though ?

[ThibG]

@TheCapsLock I'm afraid there's not much you can do, as it's not your instance that is buggy (well, it probably is too), but the remote ones.

[TheCapsLock]

@ThibG is this issue has been addressed : How can we tell Instance maintainers to upgrade ?

[dariusk]

It's a bad situation, but there is a fix that @thibg made that should go in
a future release, though not sure which one. Once the remote instances
update to that future version, you'll see their messages again.

I'm in the same situation too, all we can do now is wait...

On May 7, 2017 9:16 AM, "ThibG" [e-mail] wrote:

@TheCapsLock https://github.com/TheCapsLock I'm afraid there's not much
you can do, as it's not your instance that is buggy (well, it probably is
too), but the remote ones.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tootsuite/mastodon/issues/2672#issuecomment-299716645,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAQQ1p8n_zF5cGPeRJpmcXiQjgWKkh_pks5r3e5kgaJpZM4NM4u_
.

[TheCapsLock]

@dariusk ok thanks