Protobuf: Python 3.4+ support ...

[CyanVladimirS]

Hello everybody,

I want to run Caffe with Python 3, which needs Protobuf, but I'm unable to run Protobuf with Python 3.

When I use protobuf from: sudo pip3 install protobuf

I got this error:

Traceback (most recent call last):
File "/home/lada/diplomka/deep-learning/caffe/python/PyCaffeTest.py", line 10, in
import python.caffe
File "/home/lada/diplomka/deep-learning/caffe/python/caffe/init.py", line 1, in
from .pycaffe import Net, SGDSolver, NesterovSolver, AdaGradSolver, RMSPropSolver, AdaDeltaSolver, AdamSolver
File "/home/lada/diplomka/deep-learning/caffe/python/caffe/pycaffe.py", line 15, in
import caffe.io
File "/home/lada/diplomka/deep-learning/caffe/python/caffe/init.py", line 4, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/home/lada/diplomka/deep-learning/caffe/python/caffe/proto/caffe_pb2.py", line 7, in
from google.protobuf import reflection as _reflection
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/reflection.py", line 68, in
from google.protobuf.internal import python_message
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 848
except struct.error, e:
^
SyntaxError: invalid syntax

I found somewhere, that I need to download Protobuf BETA, so I did:
sudo pip3 install protobuf==3.0.0b2 (I tried also b1 version)

However this version still doesn't support Python 3 completlly, because when I run my Caffe script I got this error:

Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 1091, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 1113, in InternalParse
(tag_bytes, new_pos) = local_ReadTag(buffer, pos)
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/decoder.py", line 181, in ReadTag
while six.indexbytes(buffer, pos) & 0x80:
TypeError: unsupported operand type(s) for &: 'str' and 'int'

During handling of the above exception, another exception occurred: ....

Any thoughts? Any help appreciated, I'm kind of desperate now...

[xfxyjwf]

Can you check the version of six installed on your computer? Try "pip3 install --upgrade six"?

[CyanVladimirS]

Output from that command is: "Requirement already up-to-date: six in /usr/local/lib/python3.4/dist-packages"

Thanks for reply.

[CaffeBin]

Hi there,
Some issue here. Did you manage to solve it?

[CyanVladimirS]

No I did not solve it, I had to use Python 2.7....

[xfxyjwf]

Could you check the six library's version? Protobuf requires six version above 1.9.0 to work.

[CaffeBin]

Already have, six version is 1.10.0

From: Feng Xiao [mailto:[email protected]]
Sent: Thursday, March 10, 2016 7:52 PM
To: google/protobuf
Cc: Doron Shuldiner
Subject: Re: [protobuf] Python 3.4+ support ... (#1276)

Could you check the six library's version? Protobuf requires six version above 1.9.0 to work.

—
Reply to this email directly or view it on GitHubhttps://github.com/google/protobuf/issues/1276#issuecomment-194975347.

[Coldmooon]

facing the same problem...

[CaffeBin]

Reinstalling with python 3.3 worked for me

From: Coldmooon [mailto:[email protected]]
Sent: Monday, March 21, 2016 8:38 AM
To: google/protobuf
Cc: Doron Shuldiner
Subject: Re: [protobuf] Python 3.4+ support ... (#1276)

face the same problem...

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHubhttps://github.com/google/protobuf/issues/1276#issuecomment-199145981

[Coldmooon]

@CaffeBin Thank you! I installed python3.5 through anaconda. After downgrading to python3.3, all the packages including jupyter and ipython notebook didn't work because these packages are installed with python3.5 ...

[Coldmooon]

@CaffeBin After lots of attempts, finally I can letcaffe work with python3.5 in Ubuntu 14.04. These are what I did:
1) Download anaconda3-4.0 for python 3.5, then install it
2) Download protobuf 3.0 beta2 of c++ version and python version , then compile and install the two versions manually and successively.
3) Run sudo apt-get install libboost1.55-all-dev, then make a soft link

sudo ln -s /usr/lib/x86_64-linux-gnu/libboost_python-py34.so.1.55.0 /usr/local/lib/libboost_python3.so

4) Edit Makefile.config. Change PYTHON_INCLUDE to the following:

PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
                       $(ANACONDA_HOME)/include/python3.5m \
                       $(ANACONDA_HOME)/lib/python3.5/site-packages/numpy/core/include 

5) make all, make pycaffe

Updated in 2016.05.17

Note that libboost_python-py33.so.1.55.0 is not compatible with python3.5.
If you use libboost_python-py33.so.1.55.0 instead of libboost_python-py34.so.1.55.0, the python kernel will die without any error prompts when running caffe commands like solver.net.forward().
So, it is crucial to use libboost_python-py34.so.1.55.0 or later version.

[tpwrules]

I have fixed this issue with the very latest protobuf from Google's repository and building it as the c++ extension vs pure python. Make sure you rebuild Caffe after installing the new protobuf because the protobuf compiler will be updated.

[BKZero]

@Coldmooon
hi~ i try to follow your hints. but when i install protobuf 3 beta2 mannually, make and make install seems good, but when i enter the python directory and run python setup.py build, i got this:
google/protobuf/any.proto:31:10: Unrecognized syntax identifier "proto3". This parser only recognizes "proto2".
it seems wrong with something. but i don't know how to fix it. i am a newer to linux. could you give me some hint?
or should i reinstall the ubuntu cause i have installed protobuf2 before. although i uninstall it, but i am not sure if it has effects.

[andrey-iliyov]

@Coldmooon Thanks! I installed python3.5 with conda 4.0.5. Steps 2,3 help me! Thanks for your posts!

[Coldmooon]

@andrey-iliyov You are welcome~ I'm glad to have helped you.

[Coldmooon]

@BKZero To install protobuf, you need to install both the protocol compiler and the protobuf runtime. See overview section: https://github.com/google/protobuf#overview.

It seems that you have installed the protocol compiler protobuf2 before and you are trying to use the protobuf2 compiler to compile the protobuf3 python runtime.

If you follow my hints, the protobuf3 compiler will be installed. So both protobuf2 and protobuf3 exist in your system. If this is the case, you should set $PATH to point to the protobuf3 compiler
, or you may get that error.

[autosquid]

can we just redef it as:

def indexbytes(buf, i):
        return ord(buf[i])

?

[autosquid]

Hi @Coldmooon , it seems you have solved your original problem (i.e. TypeError: unsupported operand type(s) for &: 'str' and 'int') -- is that true？

[autosquid]

ok, I figured it out in my case.

Note the ParseFromString is error prone for a pythoner -- it is in fact ParseFromBytes.

[arthur-tacca]

I had this problem but this fixed it for me: The problem for me is that the C++ version of protobuf on my system was too old; it was version 2.5. The solution was to download version 3.1 (the version matching the Python library installed in my virtual environment) and build and install that (as is standard: ./configure then make then sudo make install). If you do that right it should end up in /usr/local/, and protoc --version should show 3.1. Don't forget to add /usr/local/lib to your LD_LIBRARY_PATH environment variable. You then need to rebuild caffe. (The native library is important because it affects the protocol buffer compiler protoc, as @Coldmooon said, and caffe uses that to generate the caffe_pb2.py file.)

I think this might actually be what others are talking about earlier in this thread, but I was confused by @tpwrules saying "building it as the c++ extension vs pure python" because there are two versions of the Python protobuf library, one is C++ and the other pure Python, and I thought that was the distinction being made. In fact what is important is the native code protobuf library, which is nothing to do with Python at all.

[momozizi]

hello

I have installed Anaconda, cafee, cuda and Visual Studio 2015 on windows 7. I want to start with caffe. So I run "open with Ipython" from Anaconda.When I type "import caffe" I have this error:

In [1]: import caffe
Failed to include caffe_pb2, things might go wrong! ImportError Traceback (most recent call last)
<ipython-input-1-1cca3aa1f8c5> in <module>()
----> 1 import caffe
C:\Users\mohsen\Anaconda3\envs\firstEnvironment\caffe\__init__.py in <module>()
2 from ._caffe import init_log, log, set_mode_cpu, set_mode_gpu, set_devic
e, Layer, get_solver, layer_type_list, set_random_seed, solver_count, set_solver
_count, solver_rank, set_solver_rank, set_multiprocess, has_nccl
3 from ._caffe import __version__
----> 4 from .proto.caffe_pb2 import TRAIN, TEST
5 from .classifier import Classifier
6 from .detector import Detector

C:\Users\mohsen\Anaconda3\envs\firstEnvironment\caffe\proto\caffe_pb2.py in <mod ule>()
4 import sys
5 _b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
----> 6 from google.protobuf.internal import enum_type_wrapper
7 from google.protobuf import descriptor as _descriptor
` 8 from google.protobuf import message as _message``

ImportError: No module named 'google'

In [2]:

I tried another way too.I run CMD on caffe root. then I typed "python". But when I typed "import caffe" I have this error:

>>> import caffe
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\mohsen\Anaconda3\envs\firstEnvironment\caffe\__init__.py", line
4, in <module>
from .proto.caffe_pb2 import TRAIN, TEST
File "C:\Users\mohsen\Anaconda3\envs\firstEnvironment\caffe\proto\caffe_pb2.py
", line 6, in <module>
from google.protobuf.internal import enum_type_wrapper
ImportError: No module named 'google'

I am beginner to caffe. I do not know what is my problem?

[trungphan9x]

@momozizi you should install google and protobuf package also