Quarkus: Quarkus 1.3: Smallrye AMQP not working with RabbitMQ any more, address undefined

[andreas-eberle]

Describe the bug
I try sending a simple message with the quarkus-smallrye-reactive-messaging-amqp extension via an Emitter<String>. On the sender side, there is no indication that something went wrong. However, the rabbitmq container displays the following exception

{'v1_0.error',{symbol,<<"amqp:invalid-field">>},{utf8,<<"Attach rejected: {address_not_utf8_string,undefined}">>},undefined}

The same code works with Quarkus 1.2.1.Final.

Expected behavior

• Should work as before.
• When message cannot be sent into queue, I would expect an error. If I add an exceptionally handler to the returned CompletionStage, it is also never called.

Actual behavior
Queue cannot accept message and no error is displayed.

To Reproduce
Steps to reproduce the behavior:

1. Download reproducer
   reproducer-smallrye-amqp-emitter-not-working.zip
   and unzip it.
2. Run docker-compose up -d in the extracted directory.
3. Start quarkus with ./mvnw compile quarkus:dev
4. Call GET localhost:8080/hello
5. No error in quarkus console. But the error shown above is displayed in the rabbitmq container log.

Environment (please complete the following information):

• Quarkus 1.3.0.Final
• Windows 10
• RabbitMQ 3.8.3

[geoand]

cc @cescoffier

[geoand]

@andreas-eberle can you please give the complete command you used to start RabbitMQ and how you are obtaining the username / password that you then use from the Quarkus application?

I naive attempt of mine to run your reproducer failed...

I tried:

docker run --rm -p 5672:5672 -p 15672:15672 rabbitmq:3.8-management-alpine

and then set:

amqp-username=guest
amqp-password=guest

but the application couldn't connect to RabbitMQ

[cescoffier]

The AMQP connector is for AMQP 1.0, not for RabbitMQ. You would need to enable the AMQP 1.0 in RabbitMQ.

[geoand]

The AMQP connector is for AMQP 1.0, not for RabbitMQ. You would need to enable the AMQP 1.0 in RabbitMQ.

Is there a way to enable that support when running the docker container?

[andreas-eberle]

@geoand @cescoffier: Sorry, I forgot to include them RabbitMQ with AMQP 1.0 plugin enabled. I'll update the reproduce soon.

[andreas-eberle]

@geoand @cescoffier: I updated the reproducer zip file. I added a docker-compose file starting the container and adding the AMQP 1.0 plugin for RabbitMQ. The user and password is now also already set in the application.properties. So you should only need to start the compose file and run the code.

[geoand]

After enabling DEBUG logging in Quarkus I see:

2020-03-17 13:32:48,728 FINE  [org.apa.qpi.pro.eng.imp.SaslImpl] (vert.x-eventloop-thread-1) Handled outcome: SaslImpl [_outcome=PN_SASL_OK, state=PN_SASL_PASS, done=true, role=CLIENT]
2020-03-17 13:32:48,732 FINE  [pro.trace] (vert.x-eventloop-thread-1) IN: CH[0] : Open{ containerId='rabbit@0069c93292a4', hostname='null', maxFrameSize=32768, channelMax=65535, idleTimeOut=60000, outgoingLocales=null, incomingLocales=null, offeredCapabilities=null, desiredCapabilities=null, properties={cluster_name=rabbit@0069c93292a4, copyright=Copyright (c) 2007-2020 Pivotal Software, Inc., information=Licensed under the MPL 1.1. Website: https://rabbitmq.com, platform=Erlang/OTP 22.2.8, product=RabbitMQ, version=3.8.3}}
2020-03-17 13:32:48,740 FINE  [pro.trace] (vert.x-eventloop-thread-1) IN: CH[0] : Begin{remoteChannel=0, nextOutgoingId=0, incomingWindow=65535, outgoingWindow=65535, handleMax=65535, offeredCapabilities=null, desiredCapabilities=null, properties=null}
2020-03-17 13:32:48,741 FINE  [pro.trace] (vert.x-eventloop-thread-1) IN: CH[0] : End{error=Error{condition=amqp:invalid-field, description='Attach rejected: {address_not_utf8_string,undefined}', info=null}}

@cescoffier I have no idea where to start looking, but if you give me some pointers I can give it a shot.

[gemmellr]

If you set environment variable PN_TRACE_FRM=1 for applications (or servers) using proton then you will get AMQP protocol trace output emitted to stdout by proton. (In this case since the application is using proton-j, using PN_TRACE_FRM=true also works).

I expect doing so for both versions as a comparison would show that previously with Quarkus 1.2.1 a sender link was being attached using a 'target' with a fixed address, while with 1.3 an 'anonymous sender' is now trying to be attached with a target not specifying any address (i.e null).

Anonymous senders use a layered extension mechanism [1] that many AMQP 1.0 servers have supported for up to 5+ years. I'd guess RabbitMQ is rejecting the link attach because it doesn't support the mechanism, and so is rejecting the sender link with an error thats _trying_ to convey it is happened because it didn't specify a target address string.

If that is the case, it wont work against the broker unless you can make the sender link attach use a fixed target address again.

[1] http://docs.oasis-open.org/amqp/anonterm/v1.0/cs01/anonterm-v1.0-cs01.html

[geoand]

Thank you for the analysis @gemmellr

[andreas-eberle]

@gemmellr: So what can be done to make it work like in 1.2.1.Final again? Can you add an option to not use the anonymous sender?

[andreas-eberle]

@gemmellr @geoand: Any idea what we could do? Can you give me a hint where I would need to change something to make this work again. Maybe I can provide a PR.

[geoand]

I don't know about AMQP I am afraid

[cescoffier]

I believe the only way would be to add a way in the AMQP connector to not use an anonymous sender. This should be an opt-in as it breaks the ability for the application to indicate the address in the message.

[cescoffier]

I have created https://github.com/smallrye/smallrye-reactive-messaging/issues/474.

[andreas-eberle]

@cescoffier: Thanks! In which quarkus version can we expect the fix to be in?

[cescoffier]

Quarkus 1.5 is my target. I've added this option to SmallRye Reactive Messaging yesterday.

[andreas-eberle]

Ok, thanks for the info.

[jbcodeforce]

Hi, I'm testing exactly the same configuration with quarkus 1.5.1 final, enabling amqp1.0 in rabbitmq, still get this trace in rabbitmq container:
rabbitmq_1 | 2020-06-17 01:07:40.800 [info] <0.2517.0> accepting AMQP connection <0.2517.0> (172.19.0.1:58792 -> 172.19.0.3:5672)
rabbitmq_1 | 2020-06-17 01:07:40.857 [warning] <0.2523.0> Closing session for connection <0.2517.0>:
{'v1_0.error',{symbol,<<"amqp:invalid-field">>},{utf8,<<"Attach rejected: {address_not_utf8_string,undefined}">>},undefined}

and an exception with quarkus app:
java.lang.Exception: Connection disconnected
at io.vertx.amqp.impl.AmqpConnectionImpl.onDisconnect(AmqpConnectionImpl.java:135)
at io.vertx.amqp.impl.AmqpConnectionImpl.lambda$null$2(AmqpConnectionImpl.java:98)
at io.vertx.proton.impl.ProtonConnectionImpl.lambda$getDefaultSession$6(ProtonConnectionImpl.java:263)
at io.vertx.proton.impl.ProtonSessionImpl.fireRemoteClose(ProtonSessionImpl.java:276)
at io.vertx.proton.impl.ProtonTransport.handleSocketBuffer(ProtonTransport.java:131)
at io.vertx.core.net.impl.NetSocketImpl$DataMessageHandler.handle(NetSocketImpl.java:379)
at io.vertx.core.net.impl.NetSocketImpl.lambda$new$2(NetSocketImpl.java:101)
at io.vertx.core.streams.impl.InboundBuffer.handleEvent(InboundBuffer.java:237)
at io.vertx.core.streams.impl.InboundBuffer.write(InboundBuffer.java:127)
at io.vertx.core.net.impl.NetSocketImpl.handleMessage(NetSocketImpl.java:357)
at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:369)
at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:43)
at io.vertx.core.impl.ContextImpl.executeFromIO(ContextImpl.java:232)
at io.vertx.core.net.impl.VertxHandler.channelRead(VertxHandler.java:173)
...

properties file:
amqp-host=localhost
amqp-port=5672
amqp-username=rabbit-user
amqp-password=rabbit-pass
amqp-reconnect-interval=5
mp.messaging.outgoing.itemsales.connector=smallrye-amqp
mp.messaging.outgoing.itemsales.address=items
mp.messaging.outgoing.itemsales.durable=false

Docker compose looks very similar that the one in the above zip file with the enabledplugin file mounted to /etc/rabbitmq/
rabbitmq:
hostname: rabbitmq
image: rabbitmq:3-management
ports:
- '15672:15672'
- '5672:5672'
environment:
RABBITMQ_DEFAULT_USER: rabbit-user
RABBITMQ_DEFAULT_PASS: rabbit-pass
volumes:
- ./rabbitmq/enable_plugins:/etc/rabbitmq/enabled_plugins

I added the user with matching password and vhost = / with admin permission via the rabbitmq console. I can see those traces in rabbitmq container
rabbitmq_1 | 2020-06-17 01:05:33.390 [info] <0.2292.0> Creating user 'rabbit-user'
rabbitmq_1 | 2020-06-17 01:05:33.392 [info] <0.2292.0> Setting user tags for user 'rabbit-user' to []
rabbitmq_1 | 2020-06-17 01:06:25.930 [info] <0.2396.0> Changing password for 'rabbit-user'
rabbitmq_1 | 2020-06-17 01:06:25.933 [info] <0.2396.0> Setting user tags for user 'rabbit-user' to [administrator]
rabbitmq_1 | 2020-06-17 01:06:45.310 [info] <0.2436.0> Setting permissions for 'rabbit-user' in '/' to '.', '.', '.*'

[PieterjanDeconinck]

The error messages can be resolved by updating your configuration to disable anonymous sender:
mp.messaging.outgoing.{channel-name}.use-anonymous-sender=false

However, this results in all messages being sent to the default AMQP exchange on RabbitMQ.
The target exchange (address) is completely ignored, which makes it unusable in my case.
Is there any way around this, @cescoffier? Or do you have any indication on where to start if we want to resolve this?

[cescoffier]

Yes, if you disable the anonymous sender you cannot change the target.

[cescoffier]

I would say that for such kind of advanced usage, you would need a connector for RabbitMQ (and not using the AMQP 1.0 connector). Wrapping the Vert.x RabbitMQ client would be a good start.

[PieterjanDeconinck]

Thanks for the clarification, @cescoffier. Creating a dedicated RabbitMQ extension for AMQP 0.9.1 might indeed make more sense.