Emqx: Rewritten Messages Can't be Received on Fully Rewritten Topic without Wildcards

[louisburton]

Environment:

• EMQ X version (e.g. emqx_ctl status): 4.1.0
• Docker: emqx/emqx-edge:v4.1.0

What happened and what you expected to happen:
I used the rewrite module and configured the following rule:
a/# ^a/([^/]+)/(.+)$ a/$2
It successfully rewrote a message published on topic a/b/c/d to a/c/d
I was able to subscribe to a/# and receive the message a/c/d
I was able to subscribe to a/c/# and receive the message saying its topic is a/c/d
I was not able to subscribe to a/c/d and receive any message

How to reproduce it (as minimally and precisely as possible):
Turn on the rewrite module and configure a rewrite rule as described above.
EMQX_MODULE__REWRITE__RULE__1="a/# ^a/([^/]+)/(.+)\$ a/\$2"
Confirm publishing on a/b/c/d rewrites to a/c/d
Confirm subscribing to a/c/d does not receive this message

Anything else we need to know?:
I thought I was going mad when I wasn't received messages on a topic I could see them appearing on.
I worked around this by doing something like :
a/+/+/+ ^a/(.+)/(.+)/(.+)$ a/c/\$3 - but this also only worked if it was rule 1 - which is also limiting
after this, a message published on topic a/b/c/d was rewritten to a/c/d _and_ I could successfully subscribe and receive on a/c/d.

[zhanghongtong]

Hi, @louisburton
You have subscribed to the a/c/dtopic but cannot receive any messages because the topic rewriting module is also effective for the subscription message. According to the rewriting rules, you actually subscribed to the a/d topic

[louisburton]

Hi @zhanghongtong, thanks for the prompt response.
Sorry for my misunderstanding. I had thought the rewrite applied to the single message, but rather you are saying it is more a rewrite of the production and subscription topics separately. In which case, this behaviour is not a bug. It would be great if the documentation made this explicit, as I did not find it obvious that for a single message being sent and received, the rewrite rule translation can be applied twice.

I was able to subscribe to a/c/# and receive the message saying its topic is a/c/d

So this functioned because my subscription was rewritten to a/# effectively. a/c/d did not because as you say it was rewritten to a/d.

My aim was just to drop the second level of any incoming message (once), propagating the remainder of the topic path (however many levels).
Is there a way of only applying a rewrite to just the publishing?
Or is what I want to achieve impossible without making some part of the matching distinct (either by path name or distinct number of levels) to prevent double rewrite from publishing _and_ subscription?

[zhanghongtong]

Hi, @louisburton
Sorry, we can't just rewrite the topic of the publish message,
We will consider adding this feature later, thanks for your suggestions
CC @wivwiv

[zhanghongtong]

Hi, @louisburton
We have fixed this problem on 4.2-rc.1. Now you can set module.rewrite.pub.rule.1 and module.rewrite.sub.rule.1 . If you need to use it in a production environment, it is recommended to wait another one to two weeks before we will release the official version of 4.2.0.