Smart_open: Investigate building wheels for smart_open

[mpenkov]

This has certain benefits:

1. The pip client no longer has to build wheels itself when installing
2. The install process is marginally faster
3. Any others?

Also, are there any compelling reasons to avoid building wheels?

@menshikh-iv @piskvorky @gojomo

[piskvorky]

Increased complexity and headache. Smart_open doesn't compile anything, why would we need binary wheels?

[menshikh-iv]

The wheel doesn't contain any "binary" stuff for the pure-python project.
Wheel building has no difference with default tarball if the project contains ONLY python files.

You just do python setup.py bdist_wheel instead of python setup.py sdist, where is complexity? Also, you need an only a single wheel (same as tarball) for all stuff (instead of compiling separate wheel for each python X each OS version as in gensim). See https://packaging.python.org/guides/distributing-packages-using-setuptools/#pure-python-wheels for more information.

[menshikh-iv]

So, here is all needed changes https://github.com/RaRe-Technologies/smart_open/pull/492 (1 line)

[piskvorky]

Wheel building has no difference with default tarball if the project contains ONLY python files.

Exactly – I see no point to complicating things with wheels.

[menshikh-iv]

I don't see any "complication" here, can you point to exact "complicated things", please?

[piskvorky]

Extra features / files / installation paths = extra complications, extra maintenance. It's common sense.

To summarize:

Pros:

1. The pip client no longer has to build wheels itself when installing. (that's an observation; what's the actual benefit?)
2. The install process is marginally faster

Cons:

1. Extra maintenance & support.

Anything else?

Both Pros and Cons seem quite inconsequential to me, don't care much either way. A low priority ticket.

[gojomo]

I'd prefer not using a wheel unless necessary for some user convenience, as my understanding is:

(1) it makes the download a little larger;
(2) in includes more opaque things (in this case .pyc) that are fully determined by the actual source.

Aesthetically and as a matter of "minimal surface area for errors/confusion", I'd prefer delivering the tiniest, most-transparent set of things as the package – given that they can become the full package on demand at any users' environment.

[menshikh-iv]

(1) it makes the download a little larger;

why? I see the opposite on practice

ivan@P50:~/release/smart_open$ python setup.py sdist bdist_wheel
ivan@P50:~/release/smart_open$ cd dist/
ivan@P50:~/release/smart_open/dist$ ls -laht
total 204K
-rw-r--r--  1 ivan ivan  89K апр 29 07:40 smart_open-1.10.0-py2-none-any.whl
drwxr-xr-x  2 ivan ivan 4,0K апр 29 07:40 .
drwxr-xr-x 13 ivan ivan 4,0K апр 29 07:40 ..
-rw-r--r--  1 ivan ivan 102K апр 29 07:40 smart_open-1.10.0.tar.gz

you point have sense for wheels that includes "binaries" (like gensim)

(2) in includes more opaque things (in this case .pyc) that are fully determined by the actual source.

No, .pyc never includes to wheels

[gojomo]

Interesting! If it's not including the .pyc files, then what's the practical difference between the .whl and the .tar? (How does the wheel get smaller, if it's doing more build-like stuff up-front? What's it leaving out?)

[menshikh-iv]

what's the practical difference between the .whl and the .tar?

Only in "installation process":

• when you install wheel, pip just copy content to site-packages, nothing else
• when you install tarball, modern pip build wheel (i.e. run smth from setup.py) for them and goto step above

How does the wheel get smaller, if it's doing more build-like stuff up-front? What's it leaving out?

Because tarball includes root of the repo (with CI files, data files for testing, other irrelevant stuff), wheel includes only smart_open/smart_open dir + little metadata (extracted from setup.py)

[gojomo]

For a source-only package, isn't the "run something from setup.py) step essentially "just copy content to site-packages", so the install speed difference should be negligible?

Couldn't the sdist also discard files of no value to pip-install users? On the other hand, sometimes testing data-files will still be of interest to end-users. As a policy, should people who "pip install" be able to run a library's unit-test suite?

(My default answer would be yes, so users can verify all aspects are working in their local install, or narrowly identify those that are not. Also, at least in gensim, some of the demo/tutorial notebooks make use of the test-data files as small illustrative data sets the user is assumed to have in their gensim installation.)

[menshikh-iv]

For a source-only package, isn't the "run something from setup.py) step essentially "just copy content to site-packages", so the install speed difference should be negligible?

Not really :)

Couldn't the sdist also discard files of no value to pip-install users?

Depends on MANIFEST.in and options of setup(...) function

As a policy, should people who "pip install" be able to run a library's unit-test suite?

Good question, idk. Not sure that this really useful (and somebody uses that). More than that, user should install test deps with smart_open[test], otherwise, this fails.

upd: checked - this doesn't work for now (and never works I guess)

[nc9]

This project is starting to stand out now as an ever-lone holdout on adopting wheel packages for distribution:

https://pythonwheels.com/

[kokes]

So, here is all needed changes #492 (1 line)

That's incorrect, the PR is not sufficient. I'm not terribly familiar with the project's layout, but from the get go, there are at least two things to add:

1. Tests only cover sdist builds
2. The PyPI upload script never uploads the wheel, it only covers the source distribution.

(I'm not advocating wheels or sdists, just pointing out it's not a one-liner.)

[vlcinsky]

quoting @gojomo

Aesthetically and as a matter of "minimal surface area for errors/confusion", I'd prefer delivering the tiniest, most-transparent set of things as the package ..

Well, this sounds like definition of wheel distribution format.

Anyway, reminding part of the quote:

given that they can become the full package on demand at any users' environment.

is a bit confusing and sounds contradicting the first part. If "full package" is "all it's source code", why not using git clone of given package repository? sdist format might always omit some file.

Conversion to using wheel format in this project would require:

• change of two lines in push_pypi.sh (it could even upload both sdist and wheel format)
• accept, that today de-facto standard for package distribution format (wheel) is worth spending 30-120 minutes on research of it to learn, there are no problems associated with it

[rdb]

There are some valid reasons why having wheels would be helpful for even a Python-only package:

• It is the simplest, most elegant, least-moving-parts solution for getting a Python package installed.
• There are security considerations due to the fact that wheels require no code execution during installation, which may be a factor in restricted environments.
• Some build tools, one example being Bazel, require wheels to be available, because they do not have the much more complex machinery needed to be able to build sdists.

[piskvorky]

Thanks, these look like valid reasons.

Updated summary from above:

---

Pros:

• The pip client no longer has to build wheels itself when installing. (that's an observation; what's the actual benefit?)
• The install process is marginally faster
• There are security considerations due to the fact that wheels require no code execution during installation, which may be a factor in restricted environments.
• Some build tools, one example being Bazel, require wheels to be available, because they do not have the much more complex machinery needed to be able to build sdists.

Cons:

• Extra maintenance & support.
• Anything else?

---

@mpenkov WDYT? Worth the effort?