Lemmy: Compliance with ActivityPub standard

[Nutomic]

These should be the most important items, but I left out some of the more complicated requirements.

• [x] populate content field of posts/comments with html, and put markdown into source field
  -> for now both content and source.content are Markdown, we need to revert b2288fcb9a after a while (and remove this line as well)
• [x] make apub extension fields optional (https://yerbamate.ml/LemmyNet/lemmy/pulls/163)
• [x] store and use URLs discovered from remote actors, instead of assuming everything is Lemmy (inbox, outbox, sharedInbox, followers) (see #808)
• [x] community outbox must contain activities, not objects (see #1216)
• [x] community deletion is not federated correctly under some circumstances (#1256)
• [x] requesting a non-existing object should give status code 404, currently gives 500
  curl -H "Accept: application/activity+json" https://enterprise.lemmy.ml/post/asd -v
• [x] use valid context
  https://git.asonix.dog/Aardwolf/activitystreams/issues/5
• [x] inboxes should only accept incoming requests with Content-Type: application/activity+json (or long variant)
  https://www.w3.org/TR/activitypub/#server-to-server-interactions
• [x] user outbox is missing (mandatory, but we could just have an empty user outbox for now)
  https://www.w3.org/TR/activitypub/#outbox
• [x] accept activities both via /c/{community_name}/inbox and sharedinbox (same for user)
• [x] sharedinbox should ignore incoming activities that are not public
• [x] inbox must ignore activities which were already received before, by comparing their ID (this also includes activities which are wrapped in an Announce)
• [x] implement http endpoints which return activities (/activities/follow/fd05d28e-5d8c-4787-8bcd-68aebda7fe30)
• [ ] correctly parse incoming activities which are differently formatted (eg array value vs string value) (its not clear which fields would benefit from this, lets wait for other projects to actually request this)
• [ ] return correct error codes (#1259)

[Nutomic]

I'm pretty much out of ideas on the status codes (most importantly, returning http 404 for diesel::result::Error::NotFound). Here is the best solution I could come up with:

#[derive(Debug)]
pub struct LemmyError {
  pub inner: anyhow::Error,
  pub status_code: Option<StatusCode>
}

impl<T> From<T> for LemmyError
where
  T: Into<anyhow::Error>,
{
  default fn from(t: T) -> Self {
    LemmyError { inner: t.into(), status_code: None }
  }
}

impl From<diesel::result::Error> for LemmyError {
  fn from(e: diesel::result::Error) -> Self {
    let status_code = match e {
      diesel::result::Error::NotFound => Some(StatusCode::NOT_FOUND),
      _ => None
    };
    LemmyError { inner: e.into(), status_code }
  }
}

impl actix_web::error::ResponseError for LemmyError {
  fn status_code(&self) -> StatusCode {
    self.status_code.unwrap_or(StatusCode::INTERNAL_SERVER_ERROR)
  }
}

But there are two problems with this code:

• It requires #![feature(min_specialization)], and for that reason only works on nightly
• It means that utils depends on diesel, which can slow down the compilation

[dessalines]

I like this solution... why does it say that feature is necessary?

[Nutomic]

Without that feature, the compilation fails with this error:

error[E0119]: conflicting implementations of trait `std::convert::From<diesel::result::Error>` for type `LemmyError`:
--> lemmy_utils/src/lib.rs:72:1
|
57 | / impl<T> From<T> for LemmyError
58 | | where
59 | |   T: Into<anyhow::Error>,
60 | | {
...  |
63 | |   }
64 | | }
| |_- first implementation here
...
72 |   impl From<diesel::result::Error> for LemmyError {
|   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `LemmyError`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0119`.
error: could not compile `lemmy_utils`

[Nutomic]

This issue should be finished, as far as I can tell we are now fully compatible with ActivityPub.

[dessalines]

Nice! We can re-open if other issues of being out of spec arise.

[Nutomic]

@vpzomtrrfrt @lanodan We fixed the ActivityPub problems that you reported, and some others as well. The fixes are already live on our test and production servers. So please give it a try, and let us know if you find any additional federation problems.

[lanodan]

I'm guessing lemmy.ml is up-to-date enough:

• curl -v -H 'Accept: application/activity+json' https://lemmy.ml/post/52475/comment/34246 | jq . → 404 even though the post is visible without AP
• curl -v -H 'Accept: application/activity+json' https://lemmy.ml/u/dessalines/outbox | jq . → An empty outbox
• Accept: application/ld+json,application/activity+json,application/json Doesn't works, not sure if some production software uses multiple accept formats but I have it in my script for fetching AP data, seems like an HTTP server/framework issue that isn't able to fallback or manage multi-formats.
• Page (in example https://lemmy.ml/post/52763) is missing https://www.w3.org/ns/activitystreams#Public in to or cc so it ends up being hidden like a direct message to it's Group (in example https://lemmy.ml/c/lemmy) rather than a public post.
• Page's content field seems to still be in markdown, we already have a conversion because of Peertube but I'd rather keep that as the exception than the rule as markdown format is seriously implementation-defined.

[Nutomic]

curl -v -H 'Accept: application/activity+json' https://lemmy.ml/post/52475/comment/34246 | jq . → 404 even though the post is visible without AP

This is an inconsistency between HTTP and JSON routes, the comment ID is actually https://lemmy.ml/comment/34246. Part of https://github.com/LemmyNet/lemmy/issues/698

curl -v -H 'Accept: application/activity+json' https://lemmy.ml/u/dessalines/outbox | jq . → An empty outbox

We only have community outboxes implemented for now (and they only contain the last 20 posts, nothing else). We would implement it as part of https://github.com/LemmyNet/lemmy/issues/752

Accept: application/ld+json,application/activity+json,application/json Doesn't works, not sure if some production software uses multiple accept formats but I have it in my script for fetching AP data, seems like an HTTP server/framework issue that isn't able to fallback or manage multi-formats.

I opened an issue at https://github.com/LemmyNet/lemmy/issues/1420

Page (in example https://lemmy.ml/post/52763) is missing https://www.w3.org/ns/activitystreams#Public in to or cc so it ends up being hidden like a direct message to it's Group (in example https://lemmy.ml/c/lemmy) rather than a public post.

Thanks, we weren't sure if that should be set on the activities or on the objects, sounds like it should be on both. I added it in !167

Page's content field seems to still be in markdown, we already have a conversion because of Peertube but I'd rather keep that as the exception than the rule as markdown format is seriously implementation-defined.

This is for compatibility with older Lemmy versions, but I already have a commit ready to change it to HTML. We can merge that any time, but its a breaking change and not urgent for now, so we are waiting to release it together with some other breaking changes.

[lanodan]

Page (in example https://lemmy.ml/post/52763) is missing https://www.w3.org/ns/activitystreams#Public in to or cc so it ends up being hidden like a direct message to it's Group (in example https://lemmy.ml/c/lemmy) rather than a public post.

Thanks, we weren't sure if that should be set on the activities or on the objects, sounds like it should be on both. I added it in !167

Yeah, most important in addressing is objects since the Create activity is not always seen. And pleroma is moving to requiring the Activity and Objects to have the same addressing anyway.

Page's content field seems to still be in markdown, we already have a conversion because of Peertube but I'd rather keep that as the exception than the rule as markdown format is seriously implementation-defined.

This is for compatibility with older Lemmy versions, but I already have a commit ready to change it to HTML. We can merge that any time, but its a breaking change and not urgent for now, so we are waiting to release it together with some other breaking changes.

Okay, will add a check that our markdown→HTML conversion works for your objects as well. If you have an example Page showing all your markdown features that would be nice.

Thanks for the quick response

[Nutomic]

@lanodan I think this page has everything: https://join.lemmy.ml/docs/en/about/guide.html#markdown-guide

And FYI, we use different libraries for markdown in the frontend (for html) and in the backend (for activitypub. So its possible that there are also differences between them.

[lanodan]

So I tested federation between lemmy(using lemmy.ml) and pleroma(using queer.hacktivis.me) and discovered:

• Your page objects are using the summary field instead of the name field, meaning that we don't get a proper title in pleroma, maybe could be just a fix on our side (like when name is null/empty summary is moved to it)
• I can't seem to be able to follow a user, maybe this is normal for now but if it's by design maybe there should be a field in your Person actor advertising that?
• I can't seem to be able to comment. Tried with https://queer.hacktivis.me/objects/5f62890d-62d7-4684-b519-6b19d29a3cbc and https://queer.hacktivis.me/objects/5b40c4e4-e77e-44c7-91bc-79266fd50d21

Also after looking at your markdown guide, we're missing:

• horizontal rule (probably HTML filtering)
• spoiler (we don't have support for <details><summary></summary></details> in HTML but that could be added)
• subscript/superscript (supported in HTML)

Btw the way we do previews in pleroma is by sending a false request to the backend and in our experience it's not too slow.

[Nutomic]

Your page objects are using the summary field instead of the name field, meaning that we don't get a proper title in pleroma, maybe could be just a fix on our side (like when name is null/empty summary is moved to it)

Is there any difference between the two? The description in Activitystreams Vocabulary is almost identical.

I can't seem to be able to follow a user, maybe this is normal for now but if it's by design maybe there should be a field in your Person actor advertising that?

We haven't implemented user following yet. Which field can we use for that?

I can't seem to be able to comment. Tried with https://queer.hacktivis.me/objects/5f62890d-62d7-4684-b519-6b19d29a3cbc and https://queer.hacktivis.me/objects/5b40c4e4-e77e-44c7-91bc-79266fd50d21

It prints this error when fetching your user: Failed to resolve search query as activitypub ID: value too long for type character varying(20). Must be because of the name field which has 24 chars. You could shorten that and try again. What is the max name length in Pleroma? (ping @dessalines)

Btw you can also open a new issue for federation with Pleroma ;)

[lanodan]

Your page objects are using the summary field instead of the name field, meaning that we don't get a proper title in pleroma, maybe could be just a fix on our side (like when name is null/empty summary is moved to it)

Is there any difference between the two? The description in Activitystreams Vocabulary is almost identical.

Well de-facto summary is used as a smaller version of "content" that can be used for a spoiler-tag, like the <summary> element in HTML and name is used as a title (articles, peertube video, friendica posts).

I can't seem to be able to follow a user, maybe this is normal for now but if it's by design maybe there should be a field in your Person actor advertising that?

We haven't implemented user following yet. Which field can we use for that?

None in ActivityStreams but you can extend it. That said if it's a "yet" I don't think it would be worth it to add a new one.

I can't seem to be able to comment. Tried with https://queer.hacktivis.me/objects/5f62890d-62d7-4684-b519-6b19d29a3cbc and https://queer.hacktivis.me/objects/5b40c4e4-e77e-44c7-91bc-79266fd50d21

It prints this error when fetching your user: Failed to resolve search query as activitypub ID: value too long for type character varying(20). Must be because of the name field which has 24 chars. You could shorten that and try again. What is the max name length in Pleroma? (ping @dessalines)

The default limits in pleroma are 100 for the name, 5000 for the bio, and 5000 for the post payload ((name+summary+content) < 5000).
When it's longer than that we simply truncate the content.
Usually a rule of thumb that I have for field limits -specially when hardcoded- is to pick something usual and multiply that by about two as a margin.

Btw you can also open a new issue for federation with Pleroma ;)

Ack, will do for next ones

[Nutomic]

Well de-facto summary is used as a smaller version of "content" that can be used for a spoiler-tag, like the

element in HTML and name is used as a title (articles, peertube video, friendica posts).

In fact we used summary to be compatible with Mastodon's spoiler tag, but Peertube videos or articles are probably the more similar one. I implemented a fix in !173