Protobuf: [python] Add type hints for generated classes

[cbornet]

It would be great to generate type hints (PEP484/PEP526) for the python generated class so we have IDE completion and type incompatibility warnings.

Eg for :

message Foo {
  string bar = 1;
}

generate

@overload
class Foo:
  bar = None  # type: str

  def __init__(self, bar=None):  # type: (str) -> None
    pass

Foo = _reflection.GeneratedProtocolMessageType('Foo', (_message.Message,), dict(

Note: this way of writing types works with both python 2 and 3. With python >= 3.6, the Foo class could be documented more nicely.
The type annotations are well handled at least by pycharm.

[melledijkstra]

This would be the best!

[sleshep]

+1

[selimyoussry]

+1, check mypy-protobuf by Dropbox in the meantime

[alkuzman]

+1

[anandolee]

As this future is not critical (not a bug or wrong behavior or relate to performance), protobuf team may not have time for it right now. If anyone is interested and willing to contribute on it, feel free to create a PR and assign to me for review.

Note: some field accept other types, for example bool field accept bool and numbers.Integral

[lexrj]

In response to the note: Python's type hinting allows for nuanced typing to support the indicated case

[hanabi1224]

+1

[robin-xl]

+1

[henribru]

The mypy-protobuf team is open to integration of their plugin: https://github.com/dropbox/mypy-protobuf/issues/95

Their plugin is written in Python though, so it couldn't integrate directly in the C++-based Python compiler plugin. Would a solution where the Python compiler plugin calls out to the mypy-protobuf plugin be a possibility? Perhaps with a C++ wrapper if calling it as a separate process isn't acceptable? Or would it have to be completely reimplemented in C++?

[danielgtaylor]

First off, mypy-protobuf is great. Getting it integrated into the main project here would be awesome.

That said, if you are using a recent Python version (3.7+) and willing to try another plugin I've built https://github.com/danielgtaylor/python-betterproto which generates dataclasses with proper types among other improvements. For example, given this input:

syntax = "proto3";

package repeatedmessage;

message Test {
  repeated Sub greetings = 1;
}

message Sub {
  string greeting = 1;
}

You would get this generated output:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# sources: repeatedmessage.proto
# plugin: python-betterproto
from dataclasses import dataclass
from typing import List

import betterproto


@dataclass
class Test(betterproto.Message):
    greetings: List["Sub"] = betterproto.message_field(1)


@dataclass
class Sub(betterproto.Message):
    greeting: str = betterproto.string_field(1)

This then provides proper type hints in your IDE:

[hozn]

@danielgtaylor, I think I cried a little bit when I saw this comment. This is beautiful. The standard protoc python-generated code has been absolutely the worst thing about protobuf and grpc. We currently end up with a marshalling system to turn these into plain python objects, which defeats many of the values of protobuf (e.g. single message description), but allows the messages and grpc methods to actually be usable with python typehinting, etc. Anyway, I'll be adopting your work ASAP. Thank you!

[cbows]

@danielgtaylor, are you aware of pydantic? You could use pydantics BaseModel as a substitute for dataclasses and get some more benefits like runtime validation in addition to type support in IDEs etc.
pydantic even has it's own variant of dataclasses, which might also be interesting here.

[nipunn1313]

Hey friends! Maintainer of dropbox/mypy-protobuf here! As mentioned earlier by @henribru - we're happy to move toward integrating mypy-protobuf into a feature of protobuf in python.

Not actively looking into it, but if someone wants to take a crack at it, I imagine it would look something like this.

• Start by adding a dependency on mypy-protobuf
• Add a --mypy-out default implementation - using mypy-protobuf's codegen.py
• Add documentation around this - perhaps marking it as unstable feature until maintainers choose to stabilize

If this takes hold successfully, then eventually, we can look into moving the mypy-protobuf repository under protocolbuffers project on github

Also, perhaps eventually, we could integrate entirely - using https://www.python.org/dev/peps/pep-0561/ - to do the mypy typing inline. This would unlock benefits - like better base types on enum values - with the NewType available in the *_pb2.py itself (ugh - https://github.com/dropbox/mypy-protobuf/blob/master/generated/testproto/test3_pb2.pyi.expected#L45). However, it seems like a larger project.

Happy to do reviews on the mypy-protobuf side - if there are changes required there

Also - big shoutout to @danielgtaylor's solution python-betterproto - which looks amazing as well - if you're willing to use totally different generated output.