Protobuf: //:protobuf_python with use_fast_cpp_protors does not compile with python3.7 due to python's breaking changes in C API

[qzmfranklin]

How to reproduce:

git clone https://github.com/qzmfranklin/torrent
git submodule update --init --recursive
bazel build third_party/cc/protobuf:protobuf_python --define use_fast_cpp_protos=true

The bazel -s output:

ERROR: /home/zhongming/git/torrent/third_party/cc/protobuf/BUILD:668:1: C++ compilation of rule '//third_party/cc/protobuf:python/google/protobuf/pyext/_message.so' failed (Exit 1): clang failed: error executing command 
  (cd /home/zhongming/.cache/bazel/_bazel_zhongming/252308e7db9d6378926a9d84bb91921c/execroot/torrent && \
  exec env - \
    PWD=/proc/self/cwd \
  /opt/clang/5.0.1/bin/clang -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -fno-omit-frame-pointer -isystem/opt/clang/5.0.1/include/c++/v1 -nostdinc++ '-std=c++17' -MD -MF bazel-out/linux_clang-fastbuild/bin/third_party/cc/protobuf/_objs/python/google/protobuf/pyext/_message.so/third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.pic.d '-frandom-seed=bazel-out/linux_clang-fastbuild/bin/third_party/cc/protobuf/_objs/python/google/protobuf/pyext/_message.so/third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.pic.o' -fPIC -iquote . -iquote bazel-out/linux_clang-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/linux_clang-fastbuild/genfiles/external/bazel_tools -isystem third_party/cc/protobuf/python -isystem bazel-out/linux_clang-fastbuild/genfiles/third_party/cc/protobuf/python -isystem third_party/cc/protobuf/src -isystem bazel-out/linux_clang-fastbuild/genfiles/third_party/cc/protobuf/src -isystem external/bazel_tools/tools/cpp/gcc3 -isystem third_party/cc/cpython/Include -isystem bazel-out/linux_clang-fastbuild/genfiles/third_party/cc/cpython/Include -isystem third_party/cc/cpython -isystem bazel-out/linux_clang-fastbuild/genfiles/third_party/cc/cpython -DHAVE_PTHREAD -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-writable-strings '-DGOOGLE_PROTOBUF_HAS_ONEOF=1' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.cc -o bazel-out/linux_clang-fastbuild/bin/third_party/cc/protobuf/_objs/python/google/protobuf/pyext/_message.so/third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.pic.o)

Use --sandbox_debug to see verbose messages from the sandbox
third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.cc:172:13: error: assigning to 'char *' from incompatible type 'const char *'
        if (PyString_AsStringAndSize(key, &name, &name_size) < 0) {
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.cc:69:22: note: expanded from macro 'PyString_AsStringAndSize'
       ((*(charpp) = PyUnicode_AsUTF8AndSize(ob, (sizep))) == NULL? -1: 0): \
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.cc:189:13: error: assigning to 'char *' from incompatible type 'const char *'
        if (PyString_AsStringAndSize(key, &camelcase_name, &name_size) < 0) {
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
third_party/cc/protobuf/python/google/protobuf/pyext/descriptor_containers.cc:69:22: note: expanded from macro 'PyString_AsStringAndSize'
       ((*(charpp) = PyUnicode_AsUTF8AndSize(ob, (sizep))) == NULL? -1: 0): \
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 errors generated.
Target //third_party/cc/protobuf:protobuf_python failed to build

After digging through the documentation of cpython at its master, I found this commit: https://github.com/python/cpython/commit/2a404b63d48d73bbaa007d89efb7a01048475acd

It basically says cpython made a breaking change in its C API, i.e., Python.h.

I know that python 3.7 is not released yet. What is the stance of protobuf group on this matter?

[artyompal]

This breaks the latest TensorFlow build.

[anandolee]

Python 3.7.0 was released last week on 2018-06-27. We need to add support for 3.7 and add test for it in test.sh: https://github.com/google/protobuf/blob/00d32539c6aa0c4efe9577561a00f443edd9a417/tests.sh#L244

[qzmfranklin]

Awesome to see some progress on this!

[anandolee]

4862 should have fixed this issue. I am closing it for clean up. Feel free to reopen if needed