Protobuf: Python generated code import paths

[kimvais]

By default the imports of other .proto-generated files are translated to import <module> as <alias>

They probably should be from . import <module> as <alias> as now all the _pb2.py files need to be in a top-level pythonpath.

[christophlingg]

I just run into the same problem! Let's make that a bit clearer: I have several .proto files that have dependencies between each other, let's say file1.proto and file2.proto. Once you compile them to python files the imports will look like:

file1_pb2.py

import file2

The directoy where these file will end up is not necessary in the pythonpath. Relative imports worked for python2 but apparently they don't for python3. For python3 we need from . import file2

[kimvais]

Also, relative imports are supported already in Python2.5 with from __future__ import absolute_imports
https://www.python.org/dev/peps/pep-0328/#rationale-for-relative-imports

[xdmiodz]

Hi,

I had similar issues with the new python3 importing rules, but resolved them by changing the way file2.proti is being imported from file1.proto

Suppose you have the following project tree:

|---setup.py
|---mymodule
  |--__init__.py
  |--file1.proto
  |--file2.proto

where file2.proto is imported from file1.proto like:

# file1.proto
import 'file2.proto';

Indeed in this case I had python3 complaining about the absolute imports.

But if you can replace the import directive by the following one:

# file1.proto
import 'mymodule/file2.proto';

the file1.proto will be compiled like this:

#file1_pb2.py
from mymodule import file2

and python3 will be happy with that.

Hope it will help

[awan1]

tl;dr: can we talk about the mapping of proto packages to Python packages? I'm having trouble with including different proto packages together in one Python package because of how the imports get translated.

I believe this problem still applies when using more than one subdirectory. Consider this structure:

myschema/
  myapi/
    set1/
      resource1.proto  # import "myapi/set1/resource1.proto"
      trait1.proto

In resource1.proto, trait1 is imported with import "myapi/set1/trait1.proto". resource1 and trait1 are both declared to be part of the myapi.set1 package. I compile the myschema folder so that I get these pb2 files:

codegen/
  myapi/
    set1/
      resource1_pb2.py  # from myapi.set1 import trait1_pb2 as myapi_dot_...
      trait1_pb2.py

Note that the import expects the package root level to be available on the Python path. This is a problem for me because the outer directory structure looks like this:

myproject/
  __init__.py  # init files are generated with the same script used to build codegen
  codegen/
    __init__.py 
    ...
  service/
    __init__.py
    server.py  # from ..codegen.myapi.set1 import resource1_pb2

I want to run the server. In order to make the relative imports work (in accordance with PEP 328), I run it from outside the myproject folder as

python -m myproject.service.server

and promptly get an ImportError for the resource1_pb2.py file, saying "No module named myapi.set1".

Now, I've managed to solve this by doing a path-wrangle in the codegen/__init__.py file¹, which is acceptable. However, the dream solution is to have protoc turn proto packages into proper python packages, with init files and relative imports. This would allow the compiled proto package to be included in other projects, or even pip installed as-is.

Update: this path wrangle doesn't work and causes some strange errors. Am working on a workaround to this workaround.

It's possible/likely that my use-case falls into the category of "not supported", since I believe protobufs are currently designed to be used to make single modules rather than whole packages. However, I'd like to hear what others think about this use-case, and having more "package-style" proto projects. Am I the only one wanting to do this?

---

¹ I wrote sys.path.insert(0, os.path.abspath(os.path.dirname(__file__))) in the codegen/__init__.py file. This causes the server.py import of resource1_pb2 to trigger that __init__ file, which adds the codegen folder to the python path, so that the from myapi.... import is able to discover the codegen/myapi folder and succeed.

[zhongql]

@awan1 ， have you solve the problem? now , I encountered this problem too, I split the .proto files into different packages, this relative imports is a big problem

[awan1]

@zhongql for now I've added the different generated folders into one larger folder, and I've put that folder directly on my Python path, using the sys.path.insert code in the footnote of my last comment.

[zhongql]

@awan1 ,thank you , i also through script to handle the generated file *_pb2.py to solve the relative imports error,now the program can work!

[daicoden]

I think this is related... I don't know python too well, but I'm trying to set up python packages for other developers to use GRPC from python.

I'm using GRPC and a few of google service protos.

pip install protobuf adds a python package containing google/protobuf. However, if you reference the http annotation in your protofiles, the generated python code also needs google/api.

This makes the generated code unusable because python is looking in the protobuf egg for google/api.

The only way I have figured out how to fix this is post-process the generated code.

add from __future__ import absolute_import to the top of all the files.
replace all from google.api import (.*) as (.*) with from ..google.api import $1 as $2

This only works if your package is one level deep... I'm sure it's going to get unmanageable as we add more services. Directory structure looks like:

script/
  __init__.py
  google/
    __init__.py
    api/
      __init__.py
      annotations_pb2.py
      http_pb2.py
  domain/
    __init__.py
    generated-domain with import substitution
  fetch_targets.py

and run with: python -m script.fetch_targets

Any thoughts on how to fix this? For now I've just copied the protobuf files from google/api I'm using and changed the packages.

[anandolee]

See #1491 for discussion of relative imports

[kiranmantri]

It is weird this problem was not fully solved yet !!
sure @awan1 workaround works, however it should be easy to implement this

from .module import <protobuff object>

and for python 2 compatibility:
from __future__ import absolute_import

Am I missing something ?

[wvxvw]

This is not solved yet!

If you generate the code for a package you want then to distribute (so you effectively cannot tell users they need to add extra directories to the path) you must put generated files under your package directory... but then you can no longer import them, because the proto files have no knowledge of that top-level directory.

The "solution" which involves moving around source directories is NOT a solution, it's an attempt at temporary patch. What would constitute a solution needs to be something like

option (pythonpb.options) = {
  package_name: "foo.bar.baz"
};

[dendisuhubdy]

Any update on this? Seems like a [maybe simple] problem that should be fixed right?

[wvxvw]

Actually, after researching this issue somewhat, I realized that all those "option" things are handled by plugins. Those plugins don't even need to be part of the main protoc distribution, and can be developed independently. I hope, maybe devs will comment on this and shed some light on how plugins are loaded / executed.

As an aside, and, sort of, self-promotion (not really). I started to work on an alternative Protobuf implementation: https://github.com/wvxvw/protopy but... I encountered some issues with it, and as of right now, I don't have time to work on the project, but, if anyone wanted to take over, I wouldn't mind, and would provide some support to clue into how the project works / supposed to work.

[danielgtaylor]

I've run into a similar issue as well. I went with a custom __init__.py that modified sys.path and then packaged all that up as a library, but that solution seems really hacky to me. In the end I spent a few weeks writing a new protoc plugin that handles this a bit better if you are using a recent Python version. Check it out:

https://github.com/danielgtaylor/python-betterproto

[davidgacc]

Hi all, I follow the sugestion from @awan1

└── apps
 └── myproject_1
  └── src
    └── myproject_1
        └── proto
            ├── myproject_1
               ├── v1
                    └── __init__.py
                    └── a_bar_pb2_grpc.py
                    └── a_bar_pb2.py
               └── __init__.py
            ├── myproject_2
               ├── v1
                    └── __init__.py
                    └── b_bar_pb2_grpc.py
                    └── b_bar_pb2.py
               └── __init__.py
            └── __init__.py
└── source_protos
      └── myproject_1
            ├── v1
               └── a_bar.proto
      └──  myproject_2
          ├── v1
               └── b_bar.proto

in a_bar_pb2_grpc.py , I have from myproject_1.v1 import something as somenthing_2
in apps/myproject_1/src/myproject_1/proto/__init__.py :

import sys
import os

sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))

However when I run , I get this error in a_bar_pb2_grpc.py and a_bar_pb2.py : ModuleNotFoundError: No module named myproject_1.v1

this is sys.path: ['apps/myproject_1/src', ...]
when I append os.path.abspath(os.path.dirname(__file__)) to the sys.path = ['apps/myproject_1/src/myproject_1/proto','apps/myproject_1/src', ...]. However I still get the error, if I remove apps/myproject_1/src from sys.path I got ModuleNotFoundError again but from project_1.proto. Any ideas to fix it?