Models: object_detection API protoc installation step fails

[radzfoto]

System information

• What is the top-level directory of the model you are using: ~/tensorflow/models/research
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): v1.7.0-3-g024aecf414 1.7.0
• Bazel version (if compiling from source): N/A
• CUDA/cuDNN version: Cuda compilation tools, release 9.0, V9.0.176, CUDNN: 7.0.4.31-1+cuda9.0
• GPU model and memory: nVidia Titan Xp, 12GB, driver 390.48
• Exact command to reproduce: protoc object_detection/protos/*.proto --python_out=.

Describe the problem

I followed object_detection installation instructions step by step. When I tried protoc command, .proto files could not be compiled to .py files.

I issued this command:

protoc object_detection/protos/*.proto --python_out=.

And obtained these errors:

object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.

Note that all the .proto files exist including ssd.proto.

Source code / logs

No custom source code. Just following object_detection API installation instructions

[he44]

I encountered the same issue and I think it's because of the update/latest commit. I bypassed the protoc error by checking out an older version (2018-04-13).

git reset --hard 490813bdb3499290633919a9867eb0bb6d346d87 

(How to checkout previous commit: https://stackoverflow.com/questions/3555107/git-clone-particular-version-of-remote-repository/18037563 )

[denglixi]

Upgrade your protoc.

[derekjchow]

@denglixi is correct. Update your protobuf compiler/runtime to 3.x.x

[radzfoto]

@denglixi and @derekjchow, you were both correct. I removed the 2.6 protobuf-compiler and installed the 3.5.1 protobuf-compiler from github. What is frustrating is that the installation instructions in the object_detection api github still say to use the 2.6 version. I hope that this gets updated. Thank you!

[derekjchow]

Will update the docs for next release

[lechatthecat]

related?
https://github.com/tensorflow/models/issues/4047

[lechatthecat]

I have protobuf 3.5.2 and still fail to compile. Why?

object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.

[sandunadhikari]

Make sure you grab the latest version

curl -OL https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip

Unzip

unzip protoc-3.2.0-linux-x86_64.zip -d protoc3
change your protobuf to protobuf3

Move protoc to /usr/local/bin/

sudo mv protoc3/bin/* /usr/local/bin/

Move protoc3/include to /usr/local/include/

sudo mv protoc3/include/* /usr/local/include/

Optional: change owner

sudo chwon [user] /usr/local/bin/protoc
sudo chwon -R [user] /usr/local/include/google

[AliceDinh]

@radzfoto: please show how did you install the 3.5.1 protobuf-compiler from github please!
I grab the latest version from:
https://github.com/google/protobuf/releases/download/v3.2.0/protoc-3.2.0-linux-x86_64.zip
then I don't know what to do next

[radzfoto]

Hi @AliceDinh, here are some basic steps that may help to get you started:
To download version 3.5.1 you need to go here:
then download file protoc-3.5.1-osx-x86_64.zip. Unzip the file in some directory, let's say something like ~/protobuf/

If you are using Ubuntu or similar, then use the archive manager to unzip everything in the file, or install unzip. Make sure that you put the protoc zip file in your target directory, then with unzip in a terminal window

$ cd ~/protobuf
$ sudo apt-get install unzip
$ unzip protoc-3.5.1-osx-x86_64.zip

You will now see a readme.txt file and two directories, bin and include. The protobuf compiler is a file in the bin directory called protoc. It doesn't hurt to make sure that protoc is executable, so do

$ cd bin
$ chmod +x protoc

Now you could add this directory to your PATH so you can execute from anywhere, for example:

$ export PATH=$PATH:~/protobuf/bin

If you want this to be permanent, add that line to your .bashrc file

To use the protobuf compiler now, all you have to do is

$ protoc /path/to/the/protobuf/file/you/want/to/compile

I hope this helps.

[Thaiph1308]

@radzfoto
I follow you but only 3.3 version have bin and include folder, and when I using protoc --version , it still returns
libprotoc 2.6.1
so I still got the same issue like #4223.
It's mean that I failed when installing new protoc right?
Is there any way I can check I follow exactly what you say?
And how can I install protobuf 3.5.1 version (newest version)? It's doesn't contain bin and include folders.
I tried install like @sandunadhikari but it still return libprotoc 2.6.1
i'm using ubuntu on Window 10.
Thank for your help.

[radzfoto]

@Thaiph1308, if you download the protoc zip file, it should have everything inside. Please make sure you are downloading the correct file and unzipping it correctly. It is very important that you then execute this new version of protoc. You can make sure by putting the directory in the PATH environment variable. And make sure that you remove any other versions of protoc.
I am very confused about your comment that you are using "ubuntu on Windows 10". What does that mean? Are you using Ubuntu Linux? Or are your using Windows 10? If you are using a virtualized version of Ubuntu running on something like VirtualBox on Windows, then you are just using Ubuntu.

[huyutong]

@radzfoto
I use "which protoc ,sudo rm /usr/local/bin/protoc" , uninstall the protoc2.6.1 , then follow the way you give as above to install protoc3.2.0. when I run " protoc object_detection/protos/*.proto --python_out=." ,it shows "sudo apt install protobuf-compiler", but even if I run ''sudo apt install protobuf-compiler'', it still does not work, it means the protoc was't installed correctly. so,do u have any idea about this ? thans very much

[radzfoto]

Hi @huyutong, if you read my suggested procedure more carefully, I suggest that you do NOT use apt to install the protobuf compiler. Instead go directly to the download website here:
https://developers.google.com/protocol-buffers/docs/downloads
And then click on the "release page" to select the version for your operating system. Once you download the zip file, just unzip it to a folder, look for the "bin" subdirectory and you will find the protoc executable there. Put that in your PATH and you should be all set.

[huyutong]

@radzfoto
I have followed the way you given, but it still shows ''protoc is not installed，you can install by sudo apt install protobuf-compiler'' ,then I tried ''sudo apt install protobuf-compiler'', it didnot work

[Thaiph1308]

@radzfoto
Thanks for your help, I have successfully installed protoc.
But I have some concern about file protobuf-python-3.5.1.tar.gz. It's used in Window OS right? How can we know a file is used for what OS?
And I'm using Ubuntu from Microsoft Store, direct Ubuntu from Window.
@huyutong
What OS did u use? If Ubuntu, u have to install some library (make,gcc ...) to compile protobuf-compiler before use it.
More details here https://github.com/google/protobuf/blob/master/src/README.md

[edisonchensy]

I got the same problem, please give me a help. my protobuf is 3.5.2 version.

edison@amax-server:~/google_TF_object_detection/models/research$ protoc object_detection/protos/*.proto --python_out=.
object_detection/protos/ssd.proto:87:3: Expected "required", "optional", or "repeated".
object_detection/protos/ssd.proto:87:12: Expected field name.
object_detection/protos/model.proto: Import "object_detection/protos/ssd.proto" was not found or had errors.
object_detection/protos/model.proto:12:5: "Ssd" is not defined.
edison@amax-server:~/google_TF_object_detection/models/research$ pip install protobuf
Requirement already satisfied: protobuf in /home/edison/anaconda3/lib/python3.6/site-packages (3.5.2.post1)
Requirement already satisfied: setuptools in /home/edison/anaconda3/lib/python3.6/site-packages (from protobuf) (39.0.1)
Requirement already satisfied: six>=1.9 in /home/edison/anaconda3/lib/python3.6/site-packages (from protobuf) (1.11.0)

[ETucks]

I had the same problem, and attempted resolve things by updating to protobuf 3.5.2, but was unsuccessful.
However, commmenting out line 87 of object_detection/protos/ssd.proto resolved the issue (As mentioned by mastash3ff in issue #4047 : )

[HyperWayne]

Following @sandunadhikari steps, I check my protoc version, it's still 2.6.1 (--version).
After

source ~/.bashrc

error message for this compilation disappear.

protoc object_detection/protos/*.proto --python_out=.

Update version of protoc to 3.2.0 works for me.

[dogod621]

try this pyscript... change the researchRoot for yourself.
It works perfectly for me ( I use protobuf 3.6 ).
you can execute the script at any folder.

import sys
import os
import subprocess

researchRoot = os.path.normpath( "D:/Program Files (x86)/Microsoft Visual Studio/Shared/Python36_64/Lib/site-packages/tensorflow/models/research/" )
protoRoots_ = os.listdir(researchRoot)
protoRoots = []

for protoRoot_ in protoRoots_:
    if ( protoRoot_ in protoRoots ):
        continue

    for root, dirs, files in os.walk(os.path.join(researchRoot, protoRoot_)):
        if ( protoRoot_ in protoRoots ):
            break

        for name in files:
            if ( os.path.splitext(name)[1] == ".proto" ):
                protoRoots.append( protoRoot_ )
                break


for protoRoot in protoRoots:
    for root, dirs, files in os.walk(os.path.join(researchRoot, protoRoot)):
        for name in files:
            if ( os.path.splitext(name)[1] == ".proto" ):
                cmdStr = \
                        "protoc " + name + \
                        " --proto_path=\"" + root + "\" " + \
                        " --proto_path=\"" + os.path.join(researchRoot, protoRoot) + "\" " + \
                        " --proto_path=\"" + researchRoot + "\" " + \
                        "--python_out=."

                print(cmdStr)

                with subprocess.Popen(cmdStr, stdout=subprocess.PIPE, shell=True, cwd=researchRoot) as proc:
                    stdout, stderr = proc.communicate()
                    cmdExit = proc.returncode

                    if ( cmdExit != 0 ):
                        print ("cmdExit == " + str(cmdExit) )
                        print ("stdout")
                        print (stdout)
                        print ("stderr")
                        print (stderr)
                    else:  
                        print ("cmdExit == 0")

[bozcani]

I downloaded and installed protoc from source, and it worked for me:

# You should be in folder tensorflow/models/research/
$ wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
$ unzip protobuf.zip
$ ./bin/protoc object_detection/protos/*.proto --python_out=.

Then, test the installation:

python object_detection/builders/model_builder_test.py

[upasana-mittal]

I downloaded and installed protoc from source, and it worked for me:

# You should be in folder tensorflow/models/research/
$ wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
$ unzip protobuf.zip
$ ./bin/protoc object_detection/protos/*.proto --python_out=.

Then, test the installation:

python object_detection/builders/model_builder_test.py

It is not working for me @bozcani
I tried with the version you have given 3.0.0 also with 3.4.0 but it gave me same error

object_detection/protos/*.proto: No such file or directory

Can you please help?

[AliceDinh]

@upasana-mittal
Try to run these:
protoc object_detection/protos/anchor_generator.proto --python_out=.
protoc object_detection/protos/argmax_matcher.proto --python_out=.
protoc object_detection/protos/bipartite_matcher.proto --python_out=.
protoc object_detection/protos/box_coder.proto --python_out=.
protoc object_detection/protos/box_predictor.proto --python_out=.
protoc object_detection/protos/eval.proto --python_out=.
protoc object_detection/protos/faster_rcnn.proto --python_out=.
protoc object_detection/protos/faster_rcnn_box_coder.proto --python_out=.
protoc object_detection/protos/grid_anchor_generator.proto --python_out=.
protoc object_detection/protos/hyperparams.proto --python_out=.
protoc object_detection/protos/image_resizer.proto --python_out=.
protoc object_detection/protos/input_reader.proto --python_out=.
protoc object_detection/protos/keypoint_box_coder.proto --python_out=.
protoc object_detection/protos/losses.proto --python_out=.
protoc object_detection/protos/matcher.proto --python_out=.
protoc object_detection/protos/mean_stddev_box_coder.proto --python_out=.
protoc object_detection/protos/model.proto --python_out=.
protoc object_detection/protos/multiscale_anchor_generator.proto --python_out=.
protoc object_detection/protos/optimizer.proto --python_out=.
protoc object_detection/protos/pipeline.proto --python_out=.
protoc object_detection/protos/post_processing.proto --python_out=.
protoc object_detection/protos/preprocessor.proto --python_out=.
protoc object_detection/protos/region_similarity_calculator.proto --python_out=.
protoc object_detection/protos/square_box_coder.proto --python_out=.
protoc object_detection/protos/ssd.proto --python_out=.
protoc object_detection/protos/ssd_anchor_generator.proto --python_out=.
protoc object_detection/protos/string_int_label_map.proto --python_out=.
protoc object_detection/protos/train.proto --python_out=.