Protobuf: Lack of binary Python packages on PyPI

[public]

There is a protobuf wheel on PyPI, but it is an "any" wheel. This is great, but it does mean that if you want your protobufs to be fast you need to compile your own wheels using --cpp-implementation.

https://github.com/pypa/manylinux wheels are now a thing that works for shipping binary wheels on Linux. Should this project ship fast binary wheels for Linux and other supported platforms?

[TeBoring]

@anandolee Can we have separate wheel for python2 and python3 and upload binaries for them?

[public]

@TeBoring Anything I can do to help keep this ticket moving?

[ryan-lane]

Agreed with @public. We also need this. Any update here?

[TeBoring]

We are working on this.

[public]

@teboring I made a PR that will build the wheels for you. Is this unwanted?

[TeBoring]

Uploaded: https://pypi.python.org/pypi/protobuf/3.2.0
I also have a PR: https://github.com/google/protobuf/pull/2693 Feel free to leave comments there.

[ryan-lane]

Is there any way you can also upload a wheel for 3.0.0?

[TeBoring]

Will do that. Could you help check 3.2.0 works?

[public]

@TeBoring We are using the 3.2.0 wheels and they seem to work as expected. Thanks a lot. :+1:

[ewalker11]

@TeBoring Just checking on the 3.0.0 wheels, would it be possible to upload those this week? Thanks for your responsiveness on this issue!

[TeBoring]

I'll do that.
On Tue, Feb 21, 2017 at 11:02 Eliot Walker [e-mail] wrote:

@TeBoring https://github.com/TeBoring Just checking on the 3.0.0
wheels, would it be possible to upload those this week? Thanks for your
responsiveness on this issue!

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/2623#issuecomment-281443808,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5dngQpVc6LwXlCHUbccXHZZsQtKIks5rezTfgaJpZM4LtTKs
.

[TeBoring]

done

[tebriel]

@TeBoring this causes Protobuf to now have a different behavior for 3.0.0 than it did before.

This new version changes the behavior of protobuf such that running SomeProtobuf.FromString(‘abc’) now results in a “TypeError: a bytes-like object is required, not 'str'” where before it was: “google.protobuf.message.DecodeError: Truncated message.”. I'm concerned about the behavior change here and what other potential issues may arise from this upload.

[tebriel]

We cannot help but download this version, as the bdist wheel takes precedence over the any wheel. So even requirements.txt requiring protobuf==3.0.0 still causes subsequent builds to take the new wheel.

[TeBoring]

If your code is not compatible with cpp implementation, you can avoid wheel when you do pip install.
I think in the end your code should be modified to be compatible with both python and cpp implementation. Currently, the python implementation may allow too many invalid cases.
@anandolee to take a look of this behavior change.

[crohling]

@TeBoring The issue is less with the changes that need to be made, but more about the shifting of dependencies. There has been a stable version of this code within PyPi for several months, and now it is different. IMO a package hard pinned at a specific version should be installable today, and 2 years from now, and the artifact received should not have changed at all. This is why version pinning is used, to guarantee idempotence.

[TeBoring]

I think the exact old version can still be installed by avoid wheel.
On Thu, Feb 23, 2017 at 11:48 Christian Rohling [e-mail]
wrote:

@TeBoring https://github.com/TeBoring The issue is less with the
changes that need to be made, but more about the shifting of dependencies.
There has been a stable version of this code within PyPi for several
months, and now it is different. IMO a package hard pinned at a specific
version should be installable today, and 2 years from now, and the artifact
received should not have changed at all. This is why version pinning is
used, to guarantee idempotence.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/2623#issuecomment-282100099,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5bM4eeQwquNYliJdaRP18hcm6V40ks5rfeKKgaJpZM4LtTKs
.

[tebriel]

@TeBoring Yes, it can, and does work in this one case. But my greater concern is releasing software and then re-releasing the same version later but with different code and potentially different behaviors. There is no way for me to guard against this in the future and this took more than a day to diagnose because the versions were marked as the same. Ideally these new wheels would've been released as 3.0.0-1 or 1-3.0.0 to indicate that they were new versions of 3.0.0.

[TeBoring]

Sorry for the trouble. Both python and cpp implementation came from the
same commit.
On Thu, Feb 23, 2017 at 12:07 Chris M [e-mail] wrote:

@TeBoring https://github.com/TeBoring Yes, it can, and does work in
this one case. But my greater concern is releasing software and then
re-releasing the same version later but with different code and potentially
different behaviors. There is no way for me to guard against this in the
future and this took more than a day to diagnose because the versions were
marked as the same. Ideally these new wheels would've been released as
3.0.0-1 or 1-3.0.0 to indicate that they were new versions of 3.0.0.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/2623#issuecomment-282105578,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5TrXeD_lhayOIzaQvjObWcRXQX0Tks5rfebvgaJpZM4LtTKs
.

[tebriel]

Yeah I understand, I guess I'm more asking for a forward looking policy where re-released versions get different version tags to prevent this happening again. Is this something that would be in line with the google/protobuf polices? Thanks.

[TeBoring]

I agree that 3.0.0.post1 would have been a better solution. It's also our
usual way to release a new build if previous pypi release had problems.
For this case, I think it's a one-time case. For future release, we will
always release both at the same time. So user code has to conform to the
new change if they want upgrade. (Even though, there won't be broken API
change)
Besides, I think the broken code is to do using our undefined behavior.
(Coding based on defined API can avoid being broken by such case)In the
long way, it's better to fix them early than late.
On Thu, Feb 23, 2017 at 12:15 Chris M [e-mail] wrote:

Yeah I understand, I guess I'm more asking for a forward looking policy
where re-released versions get different version tags to prevent this
happening again. Is this something that would be in line with the
google/protobuf polices? Thanks.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/2623#issuecomment-282107839,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5UmG8HCVHD8ToJTOA8vqQmpfKXqNks5rfejpgaJpZM4LtTKs
.

[tebriel]

Okay, thanks, we'll update our tests and code and will check for any other issues. Appreciate the fast response.

[public]

I think the root cause here is not releasing a wheel but that the cpp and py versions have such divergent behaviour?

[tebriel]

@public I'm definitely concerned about that. My guess is that the cpp version has type checking before we actually get to the decode. So we get a different error because the type is expected to be bytes and I'm passing string.

[TeBoring]

These divergent behavior are undefined usage. We leave them there to give
us freedom of implementation. Some of them may be easily fixed. But some
are due to language differences, and are hard to change.
On Thu, Feb 23, 2017 at 12:32 Alex Stapleton [e-mail]
wrote:

I think the root cause here is not releasing a wheel but that the cpp and
py versions have such divergent behaviour?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/2623#issuecomment-282112331,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5ZxqE6WKev1cKPj462DjrgEgRl75ks5rfez5gaJpZM4LtTKs
.

[public]

It might be too late to fix the wheel disruption but it sounds worth making a dedicated ticket for the cpp/py variance to help avoid this kind of issue for people in future?