Ejabberd: XEP-0359: stanza-id in each of received message
Hello!
Please advise how I can resolve the following issue:
What I have:
- up-to-date version of eJabberd
- mod_mam is enabled
- mod_carboncopy is enabled
In my case I must be sure that each of received message-stanza (except of those which should not be archived) includes a subtag with xmlns ''urn:xmpp:sid:0' (XEP-0359: Unique and Stable Stanza IDs).
A client always sends messages to a recipient's Bare JID. Let's pretend that we have two users Alice and Bob. Bob has three active resources with the similar priority ({0, rA}, {0, rB}, {0,rC}).
Alice sends a message to Bob:
<message type='chat' to='[email protected]'>
<body>Hey Bob!</body>
</message>
Bob (rA) receives the message:
<message from='[email protected]/resource' to='[email protected]' type='chat'>
<body>Hey Bob!</body>
</message>
Bob (rB) receives the message:
<message from='[email protected]/resource' to='[email protected]' type='chat'>
<body>Hey Bob!</body>
</message>
Bob (rC) receives the message:
<message from='[email protected]/resource' to='[email protected]' type='chat'>
<archived by='domain.com' xmlns='urn:xmpp:mam:tmp' id='1476876111664513'/>
<stanza-id by='domain.com' xmlns='urn:xmpp:sid:0' id='1476876111664513'/>
<body>Hey Bob!</body>
</message>
What eJabberd does in cases like that:
Gets the active resources of the recipient. After that it gets with lists:max/1 the first element of list ([{0, <<"rA">>}, {0, <<"rB">>}, {0, <<"rC">>}] that compares greater than or equal to all other elements of list. And after if the current resource of recipient is matched to a given element from the resources list, it saves the message in the DB and update the stanza by adding 'archived' and 'stanza-id' subtags.
Even if Alice sends a message to the Full JID, I won't be sure that all of Bob's clients will receive the packet with 'stanza-id' subtag, because the specified by Alice resource can be unavailable, and eJabberd processes unavailable resources like bare Jid.
Any hints are welcome.
iAchilles
All 25 comments
Maybe we should store (and carbon-copy) incoming messages from a new hook in ejabberd_sm? This would avoid the is_bare_copy/2 foo in mod_mam and mod_carboncopy. It would assume all incoming messages go through ejabberd_sm; and MAM messages would be stored before they are retrieved from offline storage, which sounds good to me (see #1348).
weiss
on 19 Oct 2016
@weiss , thank you!
Yes, it sounds good and it solves my problem.
iAchilles
on 19 Oct 2016
@weiss, one more question.
How can I get 'state' record in ejabberd_sm? I need to pass it as an argument in my new 'myprefix_sm_route_message' hook.
mod_mam uses it to get and check subscriptions (ejabberd_c2s:get_subscription)
iAchilles
on 20 Oct 2016
Why did you close the issue? I think this needs fixing indeed, so I'll reopen it.
weiss
on 20 Oct 2016
Sorry. I thought that it was just an advice for my case and the current behavior is not needed to be changed. :)
On my server I'm using now modified ejabberd_sm module and custom mam module.
ejabberd_sm (I added fox_sm_route_message hook):
do_route(From, To, #xmlel{} = Packet) ->
?DEBUG("session manager~n\tfrom ~p~n\tto ~p~n\tpacket "
"~P~n",
[From, To, Packet, 8]),
#jid{user = User, server = Server,
luser = LUser, lserver = LServer, lresource = LResource} = To,
#xmlel{name = Name, attrs = Attrs} = Packet,
Lang = fxml:get_attr_s(<<"xml:lang">>, Attrs),
case LResource of
<<"">> ->
case Name of
<<"presence">> ->
{Pass, _Subsc} = case fxml:get_attr_s(<<"type">>, Attrs)
of
<<"subscribe">> ->
Reason = fxml:get_path_s(Packet,
[{elem,
<<"status">>},
cdata]),
{is_privacy_allow(From, To, Packet)
andalso
ejabberd_hooks:run_fold(roster_in_subscription,
LServer,
false,
[User, Server,
From,
subscribe,
Reason]),
true};
<<"subscribed">> ->
{is_privacy_allow(From, To, Packet)
andalso
ejabberd_hooks:run_fold(roster_in_subscription,
LServer,
false,
[User, Server,
From,
subscribed,
<<"">>]),
true};
<<"unsubscribe">> ->
{is_privacy_allow(From, To, Packet)
andalso
ejabberd_hooks:run_fold(roster_in_subscription,
LServer,
false,
[User, Server,
From,
unsubscribe,
<<"">>]),
true};
<<"unsubscribed">> ->
{is_privacy_allow(From, To, Packet)
andalso
ejabberd_hooks:run_fold(roster_in_subscription,
LServer,
false,
[User, Server,
From,
unsubscribed,
<<"">>]),
true};
_ -> {true, false}
end,
if Pass ->
PResources = get_user_present_resources(LUser, LServer),
lists:foreach(fun({_, R}) ->
do_route(From,
jid:replace_resource(To,
R),
Packet)
end,
PResources);
true -> ok
end;
<<"message">> ->
case fxml:get_attr_s(<<"type">>, Attrs) of
<<"chat">> -> route_message(From, To, Packet, chat);
<<"headline">> -> route_message(From, To, Packet, headline);
<<"error">> -> ok;
<<"groupchat">> ->
ErrTxt = <<"User session not found">>,
Err = jlib:make_error_reply(
Packet, ?ERRT_SERVICE_UNAVAILABLE(Lang, ErrTxt)),
ejabberd_router:route(To, From, Err);
_ ->
route_message(From, To, Packet, normal)
end;
<<"iq">> -> process_iq(From, To, Packet);
_ -> ok
end;
_ ->
Mod = get_sm_backend(LServer),
case online(Mod:get_sessions(LUser, LServer, LResource)) of
[] ->
case Name of
<<"message">> ->
case fxml:get_attr_s(<<"type">>, Attrs) of
<<"chat">> -> route_message(From, To, Packet, chat);
<<"headline">> -> ok;
<<"error">> -> ok;
<<"groupchat">> ->
ErrTxt = <<"User session not found">>,
Err = jlib:make_error_reply(
Packet,
?ERRT_SERVICE_UNAVAILABLE(Lang, ErrTxt)),
ejabberd_router:route(To, From, Err);
_ ->
route_message(From, To, Packet, normal)
end;
<<"iq">> ->
case fxml:get_attr_s(<<"type">>, Attrs) of
<<"error">> -> ok;
<<"result">> -> ok;
_ ->
ErrTxt = <<"User session not found">>,
Err = jlib:make_error_reply(
Packet,
?ERRT_SERVICE_UNAVAILABLE(Lang, ErrTxt)),
ejabberd_router:route(To, From, Err)
end;
_ -> ?DEBUG("packet dropped~n", [])
end;
Ss ->
NewPkt = case Packet of
#xmlel{name = <<"message">>} ->
case ejabberd_hooks:run_fold(fox_sm_route_message, LServer, Packet, [From, To]) of
{_, Pkt} -> Pkt;
Pkt -> Pkt
end;
_-> Packet
end,
Session = lists:max(Ss),
Pid = element(2, Session#session.sid),
?DEBUG("sending to process ~p~n", [Pid]),
Pid ! {route, From, To, NewPkt}
end
end.
route_message(From, To, Packet, Type) ->
LUser = To#jid.luser,
LServer = To#jid.lserver,
PrioRes = get_user_present_resources(LUser, LServer),
case catch lists:max(PrioRes) of
{Priority, _R} when is_integer(Priority), Priority >= 0 ->
NewPkt = case ejabberd_hooks:run_fold(fox_sm_route_message, LServer, Packet, [From, To]) of
{_, Pkt} -> Pkt;
Pkt -> Pkt
end,
F =
fun({P, R}) when P == Priority; (P >= 0) and (Type == headline) ->
LResource = jid:resourceprep(R),
Mod = get_sm_backend(LServer),
case online(Mod:get_sessions(LUser, LServer, LResource)) of
[] ->
ok;
Ss ->
Session = lists:max(Ss),
Pid = element(2, Session#session.sid),
?DEBUG("sending to process ~p~n", [Pid]),
Pid ! {route, From, To, NewPkt}
end;
({_Prio, _Res}) -> ok
end,
lists:foreach(F, PrioRes);
_ ->
case Type of
headline -> ok;
_ ->
case ejabberd_auth:is_user_exists(LUser, LServer) andalso
is_privacy_allow(From, To, Packet) of
true ->
ejabberd_hooks:run(offline_message_hook, LServer,
[From, To, Packet]);
false ->
Err = jlib:make_error_reply(Packet,
?ERR_SERVICE_UNAVAILABLE),
ejabberd_router:route(To, From, Err)
end
end
end.
In mod_mam I use fox_sm_route_message hook instead of user_receive_packet, also I use offline_message_hook and it does the same thing like fox_sm_route_message (I store all incoming messages into archive table. I don't use 'spool' table for the offline messages.
iAchilles
on 20 Oct 2016
And I think it will be good to allow save incoming messages right in the archive table if mod_mam is enabled and mod_offline is disabled (or add an option to the 'mod_mam' module to switch this behavior).
iAchilles
on 20 Oct 2016
it will be good to allow save incoming messages right in the archive table if mod_mam is enabled and mod_offline is disabled
It should work this way in 17.03 and newer. But fixing the original issue is still to-do.
weiss
on 12 May 2017
@weiss do you want to fix the original issue ?
cromain
on 3 Aug 2017
I had a few thoughts on this but didn't get to it yet, sorry. So if someone else wants to do this I'm happy :-) Otherwise I'll let you know when I got back to it, to avoid duplicated work.
weiss
on 7 Aug 2017
This is what we need to do to announce urn:xmpp:sid:0 and urn:xmpp:mam:2 support, I think:
- Strip any
<stanza-id/>tags from all incoming message stanzas if thebyattribute matches a user/room JID of our server. - Add
<stanza-id/>tags to all 1:1 chat messages sent to our clients via c2s connections. This includes incoming carbon-copies of outgoing messages sent by other clients of the same account. - Don't add
<stanza-id/>tags to outgoing 1:1 messages. We currently have a hacky hook inmod_mamto strip our tags. Would be nice if that could be avoided. - Add
<stanza-id/>tags to all outgoing MUC messages. - Make sure
<origin-id/>tags are preserved, also for MUC messages (that's probably happening already?).
One idea I had was a mod_stanza_id that takes care of stripping tags and adding an ID to the #message.meta data as soon as a stanza enters ejabberd (from ejabberd_c2s, ejabberd_s2s_in, ejabberd_service, or ejabberd_admin; anywhere else?). mod_mam would hard-depend on mod_stanza_id and use that ID. Not sure about the best place to add the actual <stanza-id/> tags (for 1:1 and MUC messages). One caveat is that the meta data won't survive mod_offline.
Another option might be to just let mod_mam take care of stripping/adding stanza IDs, as we do now. In order to make sure that message copies all get the same <stanza-id/> tag, incoming 1:1 messages could be stored from a new hook in ejabberd_sm (before any copies are created).
weiss
on 7 Aug 2017
Hi, sorry to ask a possibly unrelated question. I have 17.08 with carboncopy, mam, and all messages in this example using omemo. Another user on this server sends me a message. If one of my two devices is offline, when reconnecting, it will only fetch my own messages I sent from my other device to the other user, and not theirs. So on the device that was offline, I only have my sent messages, not theirs. Why is this happening? Is there any config tweak I can implement or is this just how it works? I've checked all linked threads and still can't get an answer. Thanks!
nordurljosahvida
on 24 Oct 2017
all messages in this example using omemo.
Could you please first double-check you can reproduce this with OMEMO disabled?
weiss
on 24 Oct 2017
Just checked, disabled omemo mid conversation and the other device only received unencrypted messages from the other user. It still received my encrypted sent messages, as always. So yes, omemo is making the difference, but this is wrong, as those messages should also be stored. How does ejabberd know when to purge received messages from cache? Does it check, like signal does for example, that all registered devices have received such messages, or does it follow the mod_mam rules of max storage and max age?
nordurljosahvida
on 25 Oct 2017
For reference. This is the device that is always online, where I actually followed the conversation by receiving and writing messages: https://pasteboard.co/GQxIg1r.png
This instead is my other device which was offline during that time, and when it came back it only received messages I had sent, or that I had received unencrypted [omemo received messages were not received] https://pasteboard.co/GQxI2Id.png
nordurljosahvida
on 25 Oct 2017
How does ejabberd know when to purge received messages from cache?
From what cache?
zinid
on 25 Oct 2017
So yes, omemo is making the difference
Are you sure it works with OMEMO enabled when that other device is online?
weiss
on 25 Oct 2017
@zinid: not sure cache is the right term, I mean ejabberd's internal message storage, where it keeps messages until receiving devices are back online.
@weiss: yes, absolutely, if the device is online it receives both received and sent [from the other device] and also sends perfectly all with omemo enabled.
nordurljosahvida
on 25 Oct 2017
@weiss: yes, absolutely, if the device is online it receives both received and sent [from the other device] and also sends perfectly all with omemo enabled.
You could try to track it down a bit by checking whether you can reproduce it with different client and/or with a different account on the local and/or remote server.
To track it down systematically, we'd have to see the relevant traffic.
weiss
on 25 Oct 2017
Same issue with a third conversations client. Same issue with another account and server. Is there any way to send over a traffic dump?
nordurljosahvida
on 25 Oct 2017
Same issue with a third conversations client. Same issue with another account and server.
I'm not sure what exactly you tested now. You received OMEMO messages from another client/account and again had the problem that your other client does see those messages when online but not when offline? Or was it the other way round; i.e., you created a new account on your server, logged in to that new account with multiple clients, and then the same other client didn't receive encrypted MAM messages?
I'd actually test both ways (don't bother with remote servers if no remote server was involved with your original issue though). The problem might be caused by various different things, e.g. PEP or MAM or the client's clock going wrong, so these details are important.
weiss
on 25 Oct 2017
(And yes please open a separate issue next time. It's much easier to reference issues that turn out to be related than it is to follow discussions on unrelated topics within a single GitHub issue.)
weiss
on 25 Oct 2017
Ok yes sorry will do that next time. I'll try both ways tonight. How do I report back with logs or data dumps? Thanks!
nordurljosahvida
on 25 Oct 2017
How do I report back with logs or data dumps?
If you don't want to post them in public, you could send them (by email or XMPP) to [email protected]. If I see something relevant I'd then post it with JIDs and IP addresses anonymized.
weiss
on 27 Oct 2017
Thanks. I've now purchased the play store version both as a small donation to the project and as a way to test things and I can't reproduce the issue any longer. I'll keep trying and see if the F-Droid version I was using originally is the cause of the problem. Thanks again!
nordurljosahvida
on 5 Nov 2017
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
![lock[bot] picture](https://avatars.githubusercontent.com/in/6672?v=4&s=40)
lock[bot]
on 10 Jun 2019
Related issues
sujankumar4593
路
4Comments
SamWhited
路
4Comments
lucastimotiofirmino
路
3Comments
haegar
路
4Comments
pacija
路
4Comments
Most helpful comment
This is what we need to do to announce
urn:xmpp:sid:0andurn:xmpp:mam:2support, I think:<stanza-id/>tags from all incoming message stanzas if thebyattribute matches a user/room JID of our server.<stanza-id/>tags to all 1:1 chat messages sent to our clients via c2s connections. This includes incoming carbon-copies of outgoing messages sent by other clients of the same account.<stanza-id/>tags to outgoing 1:1 messages. We currently have a hacky hook inmod_mamto strip our tags. Would be nice if that could be avoided.<stanza-id/>tags to all outgoing MUC messages.<origin-id/>tags are preserved, also for MUC messages (that's probably happening already?).One idea I had was a
mod_stanza_idthat takes care of stripping tags and adding an ID to the#message.metadata as soon as a stanza enters ejabberd (fromejabberd_c2s,ejabberd_s2s_in,ejabberd_service, orejabberd_admin; anywhere else?).mod_mamwould hard-depend onmod_stanza_idand use that ID. Not sure about the best place to add the actual<stanza-id/>tags (for 1:1 and MUC messages). One caveat is that the meta data won't survivemod_offline.Another option might be to just let
mod_mamtake care of stripping/adding stanza IDs, as we do now. In order to make sure that message copies all get the same<stanza-id/>tag, incoming 1:1 messages could be stored from a new hook inejabberd_sm(before any copies are created).