Caffe: Python3 support

[shiquanwang]

As listed in requirements.txt:

Cython>=0.19.2
h5py>=2.2.0
ipython>=1.1.0
leveldb>=0.191
matplotlib>=1.3.1
networkx>=1.8.1
nose>=1.3.0
numpy>=1.7.1
pandas>=0.12.0
protobuf>=2.5.0
python-gflags>=2.0
scikit-image>=0.9.3
scikit-learn>=0.14.1
scipy>=0.13.2

With my limited search, Python3 support of protobuf and python-gflags are not known.

It is promising, anyone got an idea or a successful build.

[shelhamer]

We have no official plan to support python3. However, any adventurous souls are welcome to comment with their installation attempts.

[inferrna]

@shiquanwang
Ubuntu 15.04, got it working. Steps:

1. Install alpha version of python3-protobuf:
   ( https://github.com/google/protobuf/issues/9 )
   sudo pip3 install --pre protobuf
2. Install other requirements. In my case:
   python3-gflags python3-skimage
3. Select python version in cmake:
   cmake-gui ../
   then set "Advanced" checkbox to see and modify python related parameters.

[adnan15110]

@inferrna
I am following your instruction to install it python3. everything compiled successfully but when I tried to import caffe it thows following error.

Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "", line 1, in
File "/library_install_dir/caffe/python/caffe/init.py", line 3, in
from .proto.caffe_pb2 import TRAIN, TEST
File "/library_install_dir/caffe/python/caffe/proto/caffe_pb2.py", line 977, in
has_default_value=True, default_value=unicode("constant", "utf-8"),
NameError: name 'unicode' is not defined

It looks like some python2 and python3 unicode compitability error. anyone having similar issues?

[inferrna]

Check your cmake options. The file caffe/proto/caffe_pb2.py is generated by build process and my version haven't any call to unicode().

[CyanVladimirS]

@inferrna

which version of protobuf did you use? Because current version doesn't work with Python 3, I got this error:
Failed to include caffe_pb2, things might go wrong!
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 1091, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/python_message.py", line 1113, in InternalParse
(tag_bytes, new_pos) = local_ReadTag(buffer, pos)
File "/usr/local/lib/python3.4/dist-packages/google/protobuf/internal/decoder.py", line 181, in ReadTag
while six.indexbytes(buffer, pos) & 0x80:
TypeError: unsupported operand type(s) for &: 'str' and 'int'

when I use protobuf version: 3.0.0b2

[inferrna]

First try to import it in python console:

$ python3.4
Python 3.4.3+ (default, Oct 14 2015, 16:03:50) 
[GCC 5.2.1 20151010] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from google import protobuf
>>> protobuf.__version__
'3.0.0b2'

My opinion it's more like caffe problem than protobuf. Actually I'm on this fork https://github.com/naibaf7/caffe, it also might affect.

[Algomorph]

Just tried it on Ubuntu 16.04, succeeded with CMake compile & import in python 3 shell.

Installed the latest protobuf bindings, which was 3.0.0b3 at time of this post. I'm not sure everything will work out, since Caffe on my platform still links against the Ubuntu-standard protobuf 2.6.1 binaries, and I'm hesitant to install 3.0.0b3.

Command to get the bindings:
sudo pip3 install protobuf==3.0.0b3

Relevant CMake options:
Boost_PYTHON_LIBRARY_DEBUG=/usr/lib/x86_64-linux-gnu/libboost_python-py35.so
Boost_PYTHON_LIBRARY_RELEASE=/usr/lib/x86_64-linux-gnu/libboost_python-py35.so
PYTHON_EXECUTABLE=/usr/bin/python3
PYTHON_INCLUDE_DIR=/usr/include/python3.5
PYTHON_LIBRARY=/usr/lib/x86_64-linux-gnu/libpython3.5m.so

[CSJDeveloper]

@adnan15110
Hey, I have a same problem as you. "NameError: name 'unicode' is not defined"....
Have you solved your problem in the end, please help me out.
Thanks a lot.
And I have already checked my Makefile.config. It is correctly setted.
ANACONDA_HOME := $(HOME)/languageCode/anaconda3
PYTHON_LIBRARIES := boost_python-py34 python3.4m
PYTHON_INCLUDE := $(ANACONDA_HOME)/envs/py34/include \
$(ANACONDA_HOME)/envs/py34/include/python3.4m \
$(ANACONDA_HOME)/envs/py34/lib/python3.4/site-packages/numpy/core/include ....

[Laowai01]

i have compiled caffe on Ubuntu 16.04 LTS both with python 2 and python, it is really a hectic process for beginners, for caffe with python 2.7 make sure that your /.local/lib/python2.7/site-packages only contains a version same as you protoc version. protoc version can be checked by entering

protoc --version

for me i got problems with python 2.7 because by mistake i installed protbuf 3.0 alpha in python 2.7 site packages, so there was a protobuf error after removing it from python 2.7 site packages everything worked fine
also install python-dev version

apt-get install the python3.x-dev

and for python 3.5 make sure you have python 3.0 alpha, if you have both python 2.7 and python 3.x in the system use pip3 otherwise use pip

pip3 install protobuf==3.0.0-alpha-3

or another protobuf such as 3.5 version of protobuf.
also do this

pip3 install python-dateutil --upgrade

after that do

make clean

make all -j8
make runtest -j8

make pycaffe

after performing make all make runtest steps don't forget to do that

sudo ldconfig

hope everything works OK for you.