Protobuf: Make Python fast C++ implementation available on PyPI

[xfxyjwf]

Currently "pip install protobuf" will only install the pure python version. To install the fast C++ version, users have to install it from source. We can potentially make it a lot easier for users if we can distribute the C++ version in PyPI directly.

[xfxyjwf]

Just to keep track of things...

@haberman

[ye]

@anandolee I am curious as to why you close this ticket without any explanation? Is this resolved or there are some other existing tickets addressed this issue?

Just test it now and the pip package in PyPI is still using python implementation AFAIK, as seen below.

$ mkvirtualenv temp
Using base prefix '/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5'
New python executable in /Users/ye/.envs/temp/bin/python3.5
Also creating executable in /Users/ye/.envs/temp/bin/python
Installing setuptools, pip, wheel...pipdone.
virtualenvwrapper.user_scripts creating /Users/ye/.envs/temp/bin/predeactivate
virtualenvwrapper.user_scripts creating /Users/ye/.envs/temp/bin/postdeactivate
virtualenvwrapper.user_scripts creating /Users/ye/.envs/temp/bin/preactivate
virtualenvwrapper.user_scripts creating /Users/ye/.envs/temp/bin/postactivate
virtualenvwrapper.user_scripts creating /Users/ye/.envs/temp/bin/get_env_details
(temp) $ pip install protobuf
Collecting protobuf
  Using cached protobuf-3.2.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /Users/ye/.envs/temp/lib/python3.5/site-packages (from protobuf)
Requirement already satisfied: six>=1.9 in /Users/ye/.envs/temp/lib/python3.5/site-packages (from protobuf)
Requirement already satisfied: packaging>=16.8 in /Users/ye/.envs/temp/lib/python3.5/site-packages (from setuptools->protobuf)
Requirement already satisfied: appdirs>=1.4.0 in /Users/ye/.envs/temp/lib/python3.5/site-packages (from setuptools->protobuf)
Requirement already satisfied: pyparsing in /Users/ye/.envs/temp/lib/python3.5/site-packages (from packaging>=16.8->setuptools->protobuf)
Installing collected packages: protobuf
Successfully installed protobuf-3.2.0
(temp) $ python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
python

[xfxyjwf]

@TeBoring How to install the python C++ version using pip?

[TeBoring]

For 3.0.0 and 3.2.0, just pip install.
On Wed, Mar 8, 2017 at 10:53 Feng Xiao [e-mail] wrote:

@TeBoring https://github.com/TeBoring How to install the python C++
version using pip?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1332#issuecomment-285132399,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5eAqCGKVrawAoMOKb3o1um95O_lJks5rjvkpgaJpZM4H0SYj
.

[TeBoring]

If still not get, please check you system. We have already provided whl for
manylinux.
On Wed, Mar 8, 2017 at 10:54 Paul Yang [e-mail] wrote:

For 3.0.0 and 3.2.0, just pip install.
On Wed, Mar 8, 2017 at 10:53 Feng Xiao [e-mail] wrote:

@TeBoring https://github.com/TeBoring How to install the python C++
version using pip?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1332#issuecomment-285132399,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5eAqCGKVrawAoMOKb3o1um95O_lJks5rjvkpgaJpZM4H0SYj
.

[ye]

@TeBoring this ticket is NOT about installing PyPI via wheel on many distros. This is about having the wheel package to default to use C++ implementation of the protobuf, as the title reads.

[TeBoring]

By default, the wheel package is already c++ implementation for 3.0.0 and
3.2.0. pip install should bring you the default, which is c++.
On Wed, Mar 8, 2017 at 13:24 Ye Wang [e-mail] wrote:

@TeBoring https://github.com/TeBoring this ticket is NOT about
installing PyPI via wheel on many distros. This is about having the wheel
package to default to use C++ implementation of the protobuf, as the title
reads.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1332#issuecomment-285173854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5bgpplldSRL_kEJf2etuY9Jp4xRiks5rjxyHgaJpZM4H0SYj
.

[TeBoring]

BTW, what we can do is only to provide a whl with cpp implementation on
pypi . The pip decides which is default.
On Wed, Mar 8, 2017 at 13:27 Paul Yang [e-mail] wrote:

By default, the wheel package is already c++ implementation for 3.0.0 and
3.2.0. pip install should bring you the default, which is c++.
On Wed, Mar 8, 2017 at 13:24 Ye Wang [e-mail] wrote:

@TeBoring https://github.com/TeBoring this ticket is NOT about
installing PyPI via wheel on many distros. This is about having the wheel
package to default to use C++ implementation of the protobuf, as the title
reads.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1332#issuecomment-285173854,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5bgpplldSRL_kEJf2etuY9Jp4xRiks5rjxyHgaJpZM4H0SYj
.

[ye]

By default, the wheel package is already c++ implementation for 3.0.0 and
3.2.0. pip install should bring you the default, which is c++.

Check out my install log from today https://github.com/google/protobuf/issues/1332#issuecomment-285087905 It wasn't default to C++ after pip install.

BTW, what we can do is only to provide a whl with cpp implementation on
pypi . The pip decides which is default.

Forgive me, I didn't get the memo. And I didn't find any .whl from the release page of this repo.

[TeBoring]

https://pypi.python.org/pypi/protobuf/3.2.0
.whl is not provided through our release page. They are put on pypi, which is the default place to host whl files.
What system are you using? If you are not using linux, we don't provide cpp wheel. It's also possible that you are using some old version of pip. Could you also update your pip if so?

[ye]

@TeBoring I develop on a Mac, just like majority of the Google engineers do, from what I heard.

[TeBoring]

We haven't provided the cpp whl on Mac. Maybe in future.

[ye]

¯_(ツ)_/¯

If you ask me, I would prefer keeping this ticket open until Mac w/ cpp impl is provided; otherwise not all major platforms are covered.

[TeBoring]

SGTM
On Thu, Mar 9, 2017 at 10:50 Ye Wang [e-mail] wrote:

¯_(ツ)_/¯

If you ask me, I would prefer keeping this ticket open until Mac w/ cpp
impl is provided; otherwise not all major platforms are covered.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/protobuf/issues/1332#issuecomment-285443400,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE9H5Z7409e69UlGjfK77Dfzmnaz_ZGZks5rkEncgaJpZM4H0SYj
.

[mikeml123]

I'm having trouble using protobuf python cpp implementation - I tried to download protobuf 3.3.0.0 both from pip and conda(separately) on windows10 . Before running my script I have included
PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=cpp
but I keep getting the below message every time I run the program
"from google.protobuf.pyext import _message
ImportError: cannot import name _message"

python -c "from google.protobuf.internal import api_implementation; print(api_implementation._default_implementation_type)"
python

When I'm running my program without this flag everything runs smoothly.

How do I solve this? How do I make sure that protobuf runs in "cpp" version?
And what is the reason for making it trouble to turn on PB cpp; why isn't it the default behavior ?

[xfxyjwf]

@mikeml123 The package you get from pip doesn't contain C extension for windows. To use the cpp version you have to build it from source for now. We would also like to make Python C++ the default, but that's not easy.

[TeBoring]

done. https://pypi.org/project/protobuf/#files

[tweakimp]

Is there a version for python 3.7 on windows?

[invisibleroads]

@TeBoring
Thanks for making the default implementation of protobuf be C++

from google.protobuf.internal import api_implementation 
print(api_implementation.Type()) 

https://stackoverflow.com/questions/39079065/tensorflow-check-which-protobuf-implementation-is-being-used

[thisch]

@invisibleroads This is interesting. Didn't expect that, because this is neither mentioned in the release notes nor in the readme https://github.com/protocolbuffers/protobuf/blob/master/python/README.md

[invisibleroads]

Must have been a stealth upgrade

[laike9m]

A related question.

The documentation says:

There is also a C++ implementation for Python messages via a Python extension for better performance. Implementation type is controlled by an environment variable PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION (valid values: "cpp" and "python"). The default value is currently "python" but will be changed to "cpp" in future release.

May I ask is "change default value to cpp" something already planned? And if so, is there an estimate when it will happen?