Smart_open: Install failure with 1.8.1 on Centos 7

[caboteria]

smart_open 1.8.1 fails to install on Centos 7:

$ pip install --user smart_open Collecting smart_open Downloading https://files.pythonhosted.org/packages/27/90/39f18b9258cd76e98df581213a2b7a393b426821e920f1e820e525f54ed4/smart_open-1.8.1.tar.gz (48kB) 100% |████████████████████████████████| 51kB 7.4MB/s Complete output from command python setup.py egg_info: error in smart_open setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers
It looks like the addition of 'backports.lzma;python_version<"3.3"' is the only substantive change to setup.py since 1.8.0. According to the setuptools docs, that feature was added in 20.5 but Centos 7 ships with 0.9.8.

Not sure what to do here but I guess that setup.py should have an install_requires on setuptools>=20.5 so at least it's clear what's going on. Might want to also bump the major version to indicate a back-compatibility change.

[piskvorky]

Thanks for investigating @caboteria .

@vmarkovtsev @mpenkov do we have any other way to handle this, in setup.py?
I'm not happy about complicating our core dependencies, introducing backward compatibility issues (apparently), for the benefit of such arcane features as .xz format (I never even heard of it), from PR #262.

[piskvorky]

Ah I see you guys are already on it, on #282. Nice!

[vmarkovtsev]

Not sure what is meant by "arcane", but xz-utls have been around in Linux since 2014, that is 5 years ago. XZ is the GNU equivalent of 7z. The main advantage is higher compression ratio than bz2 at roughly the same compression speed.

Reference: https://unix.stackexchange.com/questions/108100/why-are-tar-archive-formats-switching-to-xz-compression-to-replace-bzip2-and-wha

Debian, Ubuntu and Fedora use XZ to compress their binary packages.

[mpenkov]

I think "uncommon" or "esoteric" is a better word than "arcane". Regardless of the precise meaning of the word, I think we agree in sentiment: we shouldn't handle .xz compression inside the smart_open library. Instead, we should offer support indirectly through the compressor registry.

https://stackoverflow.com/questions/6493270/why-is-tar-gz-still-much-more-common-than-tar-xz

[vmarkovtsev]

Well, I don't, but nvm.

Regarding the uncommonness, the link from the above shows that it is common in Linux - and it is included into the standard Python3 distribution. What's truly uncommon for tar is snappy, brotly, zophly, lz4, and zstd - all of them used at Google and Facebook nevertheless.

For Windows - yes, I have not seen xz archives for sure.

[piskvorky]

I'm not against adding more "fringe" features per se (.xz looks nice) – just not at the cost of compromising core dependencies.

The question is: is there something we can do to get rid of the setuptools >= 20.5 requirement?

Ideally without breaking builds on older systems that don't have new setuptools out of the box. Python 2.7 is still super popular, we're not dropping support for it any time soon. Smart_open is a production library.

[mpenkov]

Fixed by #282

[caboteria]

Thanks everyone for jumping on this so quickly!