Kombu: Kombu 4.2.2 PyPi package does not match GitHub release/tag changelog

[mshade]

It seems the kombu-4.2.2 package that was released (and found here https://pypi.org/project/kombu/#files) does not actually reflect the v4.2.2 tag/release on GitHub. It appears instead to be built from master.

$ unzip kombu-4.2.2-py2.py3-none-any.whl
Archive:  kombu-4.2.2-py2.py3-none-any.whl
<snip>

$ head kombu/compression.py
"""Compression utilities."""
from __future__ import absolute_import, unicode_literals

from kombu.utils.encoding import ensure_bytes

import bz2
import zlib

_aliases = {}
_encoders = {}

Compare the above with compression.py on v4.2.2.

We found this by chance when running on a Python installation without bz2 available.

[kojiromike]

Confirmed. I ran

git reset --hard v4.2.2 &&
curl https://files.pythonhosted.org/packages/52/29/2c4e26fb944327f7043361243c46a141b62d34d55bdb5bbef5438c7dcff0/kombu-4.2.2.tar.gz |
  tar --strip-components=1 -xvzpf-

to get the full difference. There are a lot of differences between what's tagged as v4.2.2 in this repo and what's released in the tar.gz on pypi.

[007]

@thedrow @auvipy @georgepsarakis please take note

[auvipy]

@thedrow

[atiking]

met the same issue with 4.2.2
ImportError: No module named bz2
I think we still need the bz2 import check removed by #938

[thedrow]

I am very very sorry. I didn't have the time to deal with this.
I'm going to yank the wheel and tarball and release a new one.
Thank you.

@atiking If you think so, please open an issue or a PR and explain why that's the case.

[thedrow]

I released 4.2.2post1.
Again, I apologize for the inconvenience.

[arlaneenalra]

Was it intentional for 4.2.2.post1 to replace 4.2.2? It looks like 4.2.2 has completely disappeared off pypi https://pypi.org/project/kombu/#history

[ntravis]

@arlaneenalra from a few messages up above: "I'm going to yank the wheel and tarball and release a new one."

https://github.com/celery/kombu/issues/966#issuecomment-450725054

and the justification: "It contained code which should not have been released and causes errors in production."

https://github.com/celery/kombu/issues/974#issuecomment-450808002

[butla]

I guess 4.2.2 could have stayed there in PyPI, along with post1. We had it in dependencies (don't know if it was actually being used, though) and stuff was working. Dropping the version made our build fail.

Got a bit stressed when I had to update it with poetry when it's not a first-level dependency :) But poetry lock kinda did it, though I'm not sure how different it's to poetry update.

[thedrow]

I'd rather not have broken dependencies available as it will simply create more bug reports which we don't have time to handle.

[weakcamel]

I appreciate taking the action to fix it, just .. well... deleting an already released version instead of of just bumping the version and releasing a newer one is a bit of a pain on the receiving end.

[...] it will simply create more bug reports which we don't have time to handle.

The alternative is also likely to cause bug reports from people asking "where has the release gone?"

[thedrow]

I guess that there are no good answers here.
I'll do my best to avoid such mistakes in the future.

[ntravis]

@thedrow @weakcamel I think that this was the best possible choice after the bad release. Leaving known breaking code out in the wild is definitely worse than making people pop a second update. However, the only advice I have for the future would be to include something in the changelog or release on github (or both) indicating why there was a second release done, since that would, assuming people read them, give people the answer and avoid more issues being opened.

[thedrow]

Noted, I'll do so tomorrow.

[weakcamel]

Great!

[ntravis] Leaving known breaking code out in the wild is definitely worse than making people pop a second update.

@ntravis True, if something went really, really horribly wrong, the last resort of removing the release might be a way out. I'm just not sure if this case was serious enough (it'd be easy to recover from by upgrade of the dependency, it didn't crash OS, it didn't delete anything - just wasn't exactly what it said on the tin) to warrant that. But that's my personal opinion and I can see why other's view might differ.

[thedrow] I'll do my best to avoid such mistakes in the future.

@thedrow Oh, well - mistakes will always happen, that's only natural. What's appreciated is the effort to remediate that - and kudos for that! :-)

[kojiromike]

For whatever my opinion is worth, I think as long as pypi doesn’t offer a way to mark a release as “BAD”, removing that release is the least of evils.

[andrelaszlo]

I have no issues with how this was handled. Things happen. Thanks for your hard work :heart:

We had no issues with the missing dependency but when our auto scaler decided to add a new instance, the build failed because of the missing 4.2.2 version and took our whole (admittedly tiny and unstable) production environment down with it.

My personal opinion is that once you've released something, it should stay released. People end up relying on it sooner than you know and the bugs may not affect them. This breaks things that were actually working.

I'm only saying this to provide context. In the end it's your decision and we're all benefiting from your work for free, so thanks again.

[kojiromike]

For whatever my opinion is worth, I think as long as pypi doesn’t offer a way to mark a release as “BAD”, removing that release is the least of evils.

Relevant to that comment, I will link this ticket to pypa/warehouse#345