Mastodon: Adding a relay stuck on "Waiting for relay's approval"

[cyanreg]

Adding a relay under the admin interface should result in statuses being relayed in between the relay and the instance. However, while the instance does pull the main key from the relay (and the relay's log does show this happens), the relay is never activated, instead its status is forever stuck on "Waiting for relay's approval" on the interface.

The relay itself does however show the instance as being correctly signed up and sending posts, despite the instance logs showing no exchange happens at all.
If the stuck relay request is removed from the instance, the relay does also mark the instance as not sending any statuses, despite this never actually happening.

Steps to reproduce the problem

Add a relay using the admin interface. Doesn't matter if the relay has already accepted the instance, the issue always happens.

Specifications

Was on 3.1.3, upgraded to v3.2, noticed a failure, pre- and post- upgrade was running postgres v12, ruby 2.6.6.
Upgraded to git master and to postgres v13, ruby 2.7, still doesn't work.
Its been over a month and happened during a server migration, so I could be misremembering the sequence.

Logs

Puma (sidekiq is empty, and this message keeps repeating even after the relay has been removed):

Oct 10 00:12:53 instance_hostname bundle[830]: [4c33d3ea-0e28-467a-aef4-0f6004cd799b] method=POST path=/inbox format=*/* controller=ActivityPub::InboxesController action=create status=401 duration=19.08 view=0.32 db=0.00 key=https://relay.relay_domain_name.tld/actor#main-key

Relay (don't mind the timestamp difference, was from an earlier test but this always happens):

Oct 09 23:20:58 relay_hostname python3[937]: [2020-10-09 23:20:58,924] INFO: 199.19.224.159 [09/Oct/2020:23:20:58 +0000] "POST /inbox HTTP/1.1" 200 152 "-" "http.rb/4.4.1 (Mastodon/3.2.0; +https://instance_domain_name.tld/)"

The 401 error code seems odd, and the fact it keeps repeating even after relay removal is weird.

[ThibG]

It seems the relay's signatures on incoming messages can't be verified by your instance. Is your instance running 3.2, or master? What software is the relay running?

If, in app/controllers/concerns/signature_verification.rb, you edit require_signature! so it reads

  def require_signature!
    unless signed_request_account
      Rails.logger.warn "Signature verification failed: #{signature_verification_failure_reason}"
      render plain: signature_verification_failure_reason, status: signature_verification_failure_code
    end
  end

is there more information in puma's logs?

[cyanreg]

Running git master.
With the change made, there's now some more info:

With relay.mastodon.host (which isn't even listed in the menu):

Oct 10 19:07:35 host_name bundle[293462]: [d6331d05-27a5-4961-84fb-bf3f79ba5909] Signature verification failed: Mastodon requires the Date header or (created) pseudo-header to be signed
Oct 10 19:07:35 host_name bundle[293462]: [d6331d05-27a5-4961-84fb-bf3f79ba5909] method=POST path=/inbox format=*/* controller=ActivityPub::InboxesController action=create status=401 duration=32.95 view=0.24 db=0.00 key=https://relay.mastodon.host/actor#main-key

With relay.blob.cat (which runs on https://git.pleroma.social/pleroma/relay):

Oct 10 19:05:39 host_name bundle[293479]: [0dac1c4b-9839-45cc-af29-16490c102787] Signature verification failed: Mastodon requires the Date header or (created) pseudo-header to be signed
Oct 10 19:05:39 host_name bundle[293479]: [0dac1c4b-9839-45cc-af29-16490c102787] method=POST path=/inbox format=*/* controller=ActivityPub::InboxesController action=create status=401 duration=14.39 view=0.38 db=0.00 key=https://relay.blob.cat/actor#main-key

[ThibG]

Yeah basically, this relay software's signatures are insecure because they don't cover anything really important, and a valid signature received by anyone is also valid for another payload at a later date to another recipient. A somewhat recent change in Mastodon master introduces stronger requirements for HTTP signatures in general.

A MR has been opened for that relay software, but it seems unmaintained: https://git.pleroma.social/pleroma/relay/-/merge_requests/23

[cyanreg]

Ah, I see. Thanks for explaining.
I'll get in touch with the relay admins to see if they'd be willing to upgrade/push the PR through reviews. relay.mastodon.host seems overdue for a restart and a cleanup anyway.

[mashirozx]

Hello, may we reopen this issue and continue discuss? My condition is a bit different.

I'm also stuck on "Waiting for relay's approval", but my logs shows as:

web:

35618:Oct 21 02:01:56 ubuntu-skadi bundle[2539]: [24b5a49e-526d-4e78-ae47-6c299a186749] method=POST path=/inbox format=html controller=ActivityPub::InboxesController action=create status=202 duration=10.75 view=0.00 db=1.37 key=https://mastodon-relay.moew.science/actor

sidekiq:

67920:Oct 21 02:03:23 ubuntu bundle[2557]: Rejected Create activity https://SOME_DOMAIN/users/SOME_USER_NAME/statuses/105068429349487190/activity from https://SOME_DOMAIN/users/SOME_USER_NAME https://mastodon-relay.moew.science/actor

The relay runs https://github.com/noellabo/pub-relay modified by @noellabo

[mashirozx]

It seems the relay's signatures on incoming messages can't be verified by your instance. Is your instance running 3.2, or master? What software is the relay running?

If, in app/controllers/concerns/signature_verification.rb, you edit require_signature! so it reads

  def require_signature!
    unless signed_request_account
      Rails.logger.warn "Signature verification failed: #{signature_verification_failure_reason}"
      render plain: signature_verification_failure_reason, status: signature_verification_failure_code
    end
  end

is there more information in puma's logs?

@ThibG There is no output of this message in my log.

[ThibG]

I can't reproduce it, I get the approval soon after adding https://mastodon-relay.moew.science/inbox to the list of enabled realys.

[mashirozx]

Yep, I'm not sure why, but this happen in my production environment, and not only this relay, I tried several other pub-relay powered relay, all stuck here:

The three relays that stuck all powered by pub-relay, the first one powered by Golang, the last one powered by Eugen's Ruby prototype.

[mashirozx]

I have no idea why this message

67920:Oct 21 02:03:23 ubuntu bundle[2557]: Rejected Create activity https://SOME_DOMAIN/users/SOME_USER_NAME/statuses/105068429349487190/activity from https://SOME_DOMAIN/users/SOME_USER_NAME https://mastodon-relay.moew.science/actor

[noellabo]

A Create Activity relayed from pub-relay will be rejected because the relay has not been approved by Mastodon. If the relay has not been joined, this is a normal message.

The pub-relay prototype on relay.taruntarun.net has been modified by @mayaeh to accommodate the master. That change may have alleviated some problems.

The YUKIMOCHI Toot Relay Service is implemented by Golang, but it is not allowed to connect to servers that mainly post in non-Japanese languages. This relay program is written by @yukimochi and implemented by Golang, but I don't know the technical reason for not being able to connect.

Fedibird Relay Service is powered by pub-relay 2.0 (fork by noellabo). The pre-release code is running, try this one.
https://relay.fedibird.com

If that doesn't work, open an issue in https://github.com/noellabo/pub-relay . We should be able to help you solve the problem.

[umonaca]

The three relays that stuck all powered by pub-relay, the first one powered by Golang, the last one powered by Eugen's Ruby prototype.

No. relay.ccp.ovh and relay.mastodon.host are pleroma relays, not pub-relay. They are affected by the change mentioned in the threads above:

Yeah basically, this relay software's signatures are insecure because they don't cover anything really important, and a valid signature received by anyone is also valid for another payload at a later date to another recipient. A somewhat recent change in Mastodon master introduces stronger requirements for HTTP signatures in general.
A MR has been opened for that relay software, but it seems unmaintained: https://git.pleroma.social/pleroma/relay/-/merge_requests/23

[ThibG]

I have no idea why this message

67920:Oct 21 02:03:23 ubuntu bundle[2557]: Rejected Create activity https://SOME_DOMAIN/users/SOME_USER_NAME/statuses/105068429349487190/activity from https://SOME_DOMAIN/users/SOME_USER_NAME https://mastodon-relay.moew.science/actor

This indicates that the relay has accepted your subscription and does send you content, but for some reason Mastodon isn't aware that the relay has approved you. I am not sure how that could happen, and I cannot reproduce it. This may be because the relay does not send you an Accept activity or sends you one referring to an incorrect Follow activity, but I don't know how it could happen.

[noellabo]

Subsequent investigation confirmed that the pub-relay running on mastodon-relay.moew.science was not up to date and was a buggy version. The update seems to have solved the problem.

[mashirozx]

Subsequent investigation confirmed that the pub-relay running on mastodon-relay.moew.science was not up to date and was a buggy version. The update seems to have solved the problem.

Yep, problem solved yesterday after merging that PR, thanks for your kindly reply! 🧡