Kombu: ActiveMQ support

[auvipy]

any plan or anyone working on this? this would be great to be in!

[auvipy]

and appollo too

[bmbouter]

@auvipy I'm interested in an ActiveMQ transport, and I could help mentor such an activity. I had to write the Qpid transport from scratch so we should leverage what I've learned.

[auvipy]

If you mentor me then I hope I would be able to add some inputs. i'm willing to work for it :)

[bmbouter]

How shall we collaborate on this? I'm in #celery a lot as user 'bmbouter' we could talk there.

In terms of getting started, we need to know what client library we would use for ActiveMQ. What are our choices? Do you know how many threads that client library uses? What API does it have?

[ask]

Closing this, as we don't have the resources to complete this task.

If you do decide to start up again, we welcome the contribution of this transport!

[abhaykadam]

https://issues.apache.org/jira/browse/QPID-7432 says, qpidtoollibs works over AMQP 1.0 now. Can we integrate this with Kombu+Celery now. I am ready to help do this. Though' i'll need to pointers/guidance for this. :)

[bmbouter]

@abhaykadam and I have talked some on IRC. I can help guide his contribution of a new transport for Kombu. I'm reopening to allow for discussion.

[bmbouter]

When making a transport, you need to handle two things:

1) Broker configuration. This includes CRUD operations on exchanges, queues, and bindings. For ActiveMQ you need to learn exactly what python-based client can be used to perform those things on an ActiveMQ broker.

2) Message send/receive. It looks like AMQ supports Proton with 1.0 as a client, but we need to possibly form those messages in specific ways so that when the AMQ broker receives them it can provide the routing and delivery that celery expects. That routing and delivery is build on the idea of exchanges, queues, and bindings.

[bmbouter]

After talking with the qpid team some, I've learned that:

• Proton is a fine client for message send/receive using 1.0 between the celery endpoints and the activeMQ broker.
• Having 0-10 style delivery inside the broker should work the same as on my branch
• The broker management between qpidd and activeMQ are very different and so while I use qmf.client here you would have to implement a python client to do "management" commands for ActiveMQ so that part can't be reused.

NOTE: I'm calling "broker" management all of the CRUD of exchanges, queues, and bindings.

[bmbouter]

I'm planning to advise @abhaykadam on the issue, but they will need to do the implementation.

[bmbouter]

I don't know how to add @abhaykadam as an assignee. I think he may not be in a group or something. Anyways if this is assigned then it should be assigned to @abhaykadam. If @abhaykadam can't take it as assigned then it can go back to closed for now.

[abhaykadam]

@bmbouter, i am not in a contributors group, may be that's why i am not able to assign the issue to myself.

[bmbouter]

I'm going to close this issue for now. We can continue to gather information here in case this effort is prioritized. If someone writes and pushes code to a branch, please post a link here.

[auvipy]

I wouldn't mind keeping this as a feature for future

[bmbouter]

I also would like this to be a future feature. If someone wants to implement it or posts a question here, I'll see it and can try to answer. I think this comment contains the most up-to-date info.

[jcampbell05]

So how would a newbie start implementing this?

[bmbouter]

@jcampbell05 You'll need to implement a new transport. Look at the objects defined in this file to see the Transport that I made. https://github.com/celery/kombu/blob/3.0/kombu/transport/qpid.py

You need to implement the various interfaces so that Kombu (and Celery in some places if you plan to use it with Celery) call them.

The docstrings are copies. Go read them, and then I can answer questions if you post them here.

[avermeer]

Hi @bmbouter,

I am interested in ActiveMQ support for kombu.
Is there any update with regard to this initiative ?

Is ActiveMQ's AMQP's support (see https://activemq.apache.org/amqp) enough for Kombu?

Kind regards,
Alexandre

[bmbouter]

@avermeer our project has switched from celery to using RQ so I'm not sure. We posted our reasons about why here: https://pulpproject.org/2018/05/08/pulp3-moving-to-rq/#reasoning

[avermeer]

Okay but kumbo is still supported, isn't it ? so maybe other users/contributors are interested in ActiveMQ support in kumbo ?

[bmbouter]

celery and kombu continue to work together, and I believe with effort ActiveMQ support could be added to kombu and in-turn celery.

[auvipy]

Okay but kumbo is still supported, isn't it ? so maybe other users/contributors are interested in ActiveMQ support in kumbo ?

yes eventually :)