kombu 4.4.0 arguably includes a breaking change and can break user's software.

[dbaxa]

User's of kombu who were using it with celery 4.2.x previously had to pin the version of python redis to a version < 3. When kombu version 4.4.0 was released, those who had pinned redis to a version < 3, because of https://github.com/celery/celery/issues/5175 , found their software/deployments broken. I strongly suggest in the interests of not causing unintentional breakage to users of celery & kombu to follow semantic versioning and to avoid making breaking changes like was done (requiring a version of redis >= 3.2.0).

Additionally, the celery 4.2.x branch still pins the version of redis to versions below < 3, see https://github.com/celery/celery/blob/4.2/requirements/extras/redis.txt .

[alexandre-paroissien]

I confirm I encountered the same issue (and then I encountered #1018 and #1019 )

[auvipy]

@thedrow

[thedrow]

This is completely intentional since py-redis<3.20 contains bugs which crash Celery or makes certain features unusable.
This is exactly why I bumped the minor version.
We don't follow SemVer to the letter since we want Celery 5 to contain certain features and signify a new path for the project.

This breakage is declared in both the Celery 4.3.0 RC2 & Kombu 4.4.0 release notes.
The What's New document will contain a fair warning.
If you want to keep using py-redis 2, please downgrade to Kombu 4.3 and keep using Celery 4.2 or Celery 4.3RC1.
Alternatively, try to backport the relevant py-redis 3 patches and contribute them upstream if possible. You can also maintain your own fork.

I apologize for the inconvenience but we had to do this since any other path would involve heavy monkey patching or breakage for users.

Thank you for noticing and opening an issue about this.
I'm going to close this issue as won't fix.

[dbaxa]

@thedrow It is fine to close this issue as won't fix. However, I hope that in the future the celery and kombu maintainers work to ensure that dependency management changes are done in a more considered fashion. I note that there was no 4.2.x release of celery that contained the change that was related to redis version pinning but having one would have meant that users may have needed to pin the version for redis redis themselves. (Then later arguably when the issue was fixed in kombu, you could have released celery 4.2.3 that removed the redis version pinning).

Based on what you have said about not following sem-ver, it would seem that you want people to pin specific versions of kombu when using celery. That's something that I will be doing in the future as kombu changes have included breaking changes outside of major version releases.

[thedrow]

You can open an issue about releasing 4.2.3 or change this one and I will reopen and do so.
That's a very good suggestion.

[ThiefMaster]

This kind of behavior is getting annoying, there was a similarly breaking change not too long ago, and every time this happens you are forcing people who pin a specific celery version (but rely on celery specifying proper requirements e.g. for kombu) to make a new release of their package.

And especially with a library as common as redis-py it's very likely that an application requires a specific version of the library as well as a specific celery version.

[thedrow]

@ThiefMaster I understand the inconvenience.
What would you rather us do? Break other people's code or cut lose the common language over Celery 5?
We can do the latter. Not the former.

Seriously, we're trying our best here.
Your welcome to join and contribute as a core member.

[nyhobbit]

This is completely intentional since py-redis<3.20 contains bugs which crash Celery or makes certain features unusable.
This is exactly why I bumped the minor version.
We don't follow SemVer to the letter since we want Celery 5 to contain certain features and signify a new path for the project.

This breakage is declared in both the Celery 4.3.0 RC2 & Kombu 4.4.0 release notes.
The What's New document will contain a fair warning.
If you want to keep using py-redis 2, please downgrade to Kombu 4.3 and keep using Celery 4.2 or Celery 4.3RC1.
Alternatively, try to backport the relevant py-redis 3 patches and contribute them upstream if possible. You can also maintain your own fork.

I apologize for the inconvenience but we had to do this since any other path would involve heavy monkey patching or breakage for users.

Thank you for noticing and opening an issue about this.
I'm going to close this issue as won't fix.

@thedrow
@dbaxa
I want to report that, even after making sure I am all up-to-date with:

• celery 4.3.0 RC2
• kombu 4.4.0
  AND, last but not least,
• redis 3.2.0

I still have the issues reported above by others (although intermittently, NOT systematically), i.e.:

• Error 110 (connection timed out)
• Error 32 (broken pipe)

Please advice on the right combination of libraries that guarantees a stable behavior for celery used in conjunction with Redis. This is probably critical for many production applications everywhere.

[alexandre-paroissien]

This is completely intentional since py-redis<3.20 contains bugs which crash Celery or makes certain features unusable.
This is exactly why I bumped the minor version.
We don't follow SemVer to the letter since we want Celery 5 to contain certain features and signify a new path for the project.
This breakage is declared in both the Celery 4.3.0 RC2 & Kombu 4.4.0 release notes.
The What's New document will contain a fair warning.
If you want to keep using py-redis 2, please downgrade to Kombu 4.3 and keep using Celery 4.2 or Celery 4.3RC1.
Alternatively, try to backport the relevant py-redis 3 patches and contribute them upstream if possible. You can also maintain your own fork.
I apologize for the inconvenience but we had to do this since any other path would involve heavy monkey patching or breakage for users.
Thank you for noticing and opening an issue about this.
I'm going to close this issue as won't fix.

@thedrow
@dbaxa
I want to report that, even after making sure I am all up-to-date with:

* celery 4.3.0 RC2

* kombu 4.4.0
  AND, last but not least,

* redis 3.2.0

I still have the issues reported above by others (although intermittently, NOT systematically), i.e.:

* Error 110 (connection timed out)

* Error 32 (broken pipe)

Please advice on the right combination of libraries that guarantees a stable behavior for celery used in conjunction with Redis. This is probably critical for many production applications everywhere.

I had these issues when updating kombu (not celery). I have celery 4.2.1 with kombu 4.3.0 and redis 2.10.6 working fine

When updating to kombu 4.4.0 and redis 3.2.0 (keeping celery 4.2.1) I encountered these two issues too, intermittently as well:

Error 110 (connection timed out) #1019

Error 32 (broken pipe) #1018

I included stack traces in the tickets

[brianmay]

It seems unfortunate that "celery 5" was used way before release. Version numbers should only be allocated when the product is about to be released, and this problem would not have occurred.

Once I encountered this same problem with a piece of my own software, and found that I had to skip the major version that was still under development (and indefinitely stalled due to sudden and unexpected lack of resources). If I ever continued development with this old version we would have to renumber it again before releasing it.

Perhaps in future a code name could be used instead?

It also highlights that semver is different from traditional versioning. Traditionally the major version number is incremented to indicate major changes in design or approach. Where as semver uses it to indicate breaking changes. A major redesign may not necessarily have breaking changes if it continues to be compatible with the previous API.

Semver very much helps with maintaining dependencies when you have a large number of dependencies. As is common with typically Django software. You don't really want to have to look up the release notes of every minor or patch update of every piece of software in case it might break something. That is the purpose of the version number, to give some clues when an update needs special care.

(having said that you still need to beware of updates that could contain malicious code: e.g. https://github.com/dominictarr/event-stream/issues/116)

[nyhobbit]

I had these issues when updating kombu (not celery). I have celery 4.2.1 with kombu 4.3.0 and redis 2.10.6 working fine

When updating to kombu 4.4.0 and redis 3.2.0 (keeping celery 4.2.1) I encountered these two issues too, intermittently as well:

Thanks @alexandre-paroissien - I am pinning this configuration and hoping for the best.
Question, though, for maybe @thedrow and @dbaxa - is there a configuration of the three library above that would allow me to use with confidence redis >=3.0 (hopefully 3.2.0) since it looks like that library version does fix a bunch of redis issues and bugs wrt 2.10.6 ?

[thedrow]

I have no idea what's going on in @alexandre-paroissien's environment.
The stack traces are not enough for me to reproduce these problems.

[thedrow]

4.4.0 fixed #1006 for example.
I really don't know what to do here.

[bogdanpetrea]

A solution would be for people to start making their builds reproducible. Pipfile is nice to help with that.

[thedrow]

I'm going to lock this issue. If you have a specific problem I need to resolve such as releasing a new 4.2.x release please open a new one and describe exactly what do you need.