Celery: Celery release cycle

[Jamim]

There is no reasonable _release cycle_ for Celery.
If something is broken, it could be broken in stable version for a very long time, even if it's actually fixed at master.

Steps to reproduce

• Fix some Celery bug
• Merge the fix to master branch
• Close issue on GitHub

Expected behavior

Issue is closed, fixed version is released.

Actual behavior

Issue is closed, fixed version is not released for ages.
People experience that bug again, get confusing and report to closed issue that the bug isn't fixed.

Examples

• #2649
• #4500

[robertknight]

Our project is waiting for the next release for Python 3.7 support. If there is some way we can help with the process, please let me know.

[auvipy]

Help us find some sponsor for maintaining celery project. that's the most needed stuff to dedicate some serious amount of time for this huge project.

[johnarnold]

@auvipy define "sponsor" :) Money, time? Many big players using celery.

One possible approach is time-based releases, where everything in master just gets shipped (once/month?). Major bugfixes or security fixes ship immediately. You can get a tight feedback loop going with the user base with faster releases. People can pin versions in their project as needed (we do) to avoid unexpected churn.

I spend time on fixing bugs, enhancements my company needs, and trying to push milestone releases out the door...

@robertknight Under Issues, click Milestones, pick the next milestone and go close out open issues. Typically there are a dozen or so reported bugs with no fix, documentation issues etc. See if you can resolve some?

[MattBlack85]

Chiming in, I'm not in a huge hurry for moving to python 3.7 although it would be nice. Was looking for some answers around on release 4.3. Nothing. I get and understand lack of money is an issue on such a big project, cause it cannot be otherwise, but I still think things should be done transparently saying everybody which are the short and long term plans for Celery. There is no such a thing as last releases speak for themselves and while I'm fine with it, I'd feel more comfortable knowing where the project is leading. As the main project is really complicated and has a lot of stuff to maintain how do you core people see the future of celery? Personally I don't think throwing money at the project is sufficient, as the python world is changing and evolving rapidly at least some things should be revisited and a strategy should be created.

[auvipy]

thanks all for your inputs. apart from the issues opened here, some more things those are on my plan for the future of celery,

1. Release celery 4.3 ASAP if possible by October.
2. Drop python 2 from master branch and make celery 4.x branch a LTS with bug fix only until the end of 2019.
3. embrace asyncio and it's eco system where possible. major overhaul needed.
4. Find out async based alternative for billiard or rewrite billiard to make it async friendly [inputs wanted]
5. implement something native like redbeat for distributed celery beat.
6. Kafka support and related overhaul.
7. Embrace property-based testing gradually and improve test coverage.
8. Improve docs and fix open bugs.
9. add others......

These are my priorities and I am going to start working for this first 8 and possibly some more team members and community member will also contribute to implementing feature requests and squash open bugs.
We also have more features in the plan but inho these are minimal priorities for now.

feel free to share your opinions.

[xirdneh]

Do we want to list the issues related to these features somewhere in the documentation so people can easily see what's the plan?
I know for No. 5 we have: https://github.com/celery/celery/issues/4815
Not sure about the rest.

[auvipy]

maybe we can create a roadmap section and link related issues with little descriptions? and add roadmap in readme and docs to make it more visible?

[MattBlack85]

sounds like a good idea

[mariokostelac]

I think it still does make sense to talk about the release cycles and how the project ticks. I think that future roadmap and release cycles are only loosely coupled. Stable release cadence will give us a mechanism to ship fixes regularly, while the roadmap will help to map future work on release cycles. More funding (in money or time) will just "compress" the roadmap.

[webjunkie]

I think the wiki could be updated with the proposed roadmap (instead of cluttering the readme) and then milestones from the GitHub issues could be linked, so it's clear what has a ticket already and so on.

[georgepsarakis]

I think that too, the Wiki sounds like a good place to document the roadmap.

[xirdneh]

Can we also add another item?
What about adding support for redis task queues?

[auvipy]

why not? aren't they already supported?

[thedrow]

@xirdneh What do you mean?

[xirdneh]

Sorry for the late response. Maybe I'm a bit confused about this one.
I thought celery was using redis' pub/sub which means that messages get delivered to subscribers as soon as they come.
But we could also use FIFO queues in redis to make it work more like a queue and to power celery beat.
That last part is already mentioned in #4815
Please, correct me if I'm wrong on any of this @auvipy @thedrow Thanks :)

[xirdneh]

Ok, I believe I am wrong and Kombu does uses LPUSH and LPOP to handle messages. I guess I was thinking about something else but I went back to the code to double check it. Sorry about that.

[auvipy]

Haha no worries :dagger:

[antoine-gallix]

Hello there. I've read this thread carefully, but I can't see any conclusion regarding the release cycle. As @mariokostelac remarked, maintenance and adding features requires work, but on the other hand, releasing a new version when changes are already merged on the master branch shouldn't require as much work, yet no version have been released for some months now. That was the original subject of this issue raised by @Jamim . For example, in our company, the only blocker for using python 3.7 is that it's not supported by celery. If I understand correctly, the master branch contains changes that allows using celery with python 3.7 . Is there any scheduled date for the release of those changes?

[xirdneh]

@antoine-gallix Probably @auvipy can correct me if I'm wrong. But I believe we can't do a release supporting python 3.7 until we do some more testing with 3.7 as well as add it to the CI workflow.
Have you been able to test the latest master with your project and make sure it's working correctly?

[thedrow]

Actually the tests fail when running with 3.7 and we need to fix https://github.com/celery/py-amqp/issues/206.
This is an open source project with very little donations. We work on it on our spare time.
Contributions are required to improve and support this project.
We can't really make deadlines. We're hoping to release in the coming months if the contributions to support Python 3.7 arrive.

[antoine-gallix]

@thedrow That's totally understandable. Thanks for the precisions.

[davidbarton]

Can we find somewhere a checklist of what is missing in order to get the next release finished? Would help us to navigate what stuff we can help with and what is the status.

[Jamim]

Hi @davidbarton,
I believe you could look at milestones.

[ewenger]

@auvipy My company has a customer that is requiring Kafka support. We'd be interested in funding a celery dev to help get it there. Happy to talk more next week if you'd like.

[auvipy]

@ewenger ping me at [email protected]

[manojmspython]

Hi Gents, would i be able to expect a Release date for celery 4.3.
I am waiting for the following fixes
https://github.com/celery/celery/issues
https://github.com/celery/celery/issues/4995

[jsh2134]

For those on this thread Celery 4.3 has been officially released

[thedrow]

We're going to document the release cycle and support policy soon.
Stay tuned.

[waleedsamy]

We're going to document the release cycle and support policy soon.
Stay tuned.

where I could find info about the coming release? thanks.

[auvipy]

We're going to document the release cycle and support policy soon.
Stay tuned.

where I could find info about the coming release? thanks.

check the Github milestones

[auvipy]

celery now release more frequent bug fix minor releases.

[auvipy]

IMHO we should stick with SemVer based releases & if possible continuous release or weekly/forthnightly/monthly patch/minor releases with small new feature