Bazel: Support Python/C++ interoperability (native Python modules)

[davidzchen]

According to the documentation for py_library.deps:

The list of other libraries to be linked in to the binary target. See general comments about deps at Attributes common to all build rules. These can be py_binary rules, py_library rules or cc_library rules,

However, when I try to build a py_library target that depends on a cc_library, I get the following error:

❯❯❯ bazel build //external:markupsafe
ERROR: /private/var/tmp/_bazel_dzc/a4c09a1fcefb8b21c5b0e7d7db6f08ba/external/markupsafe_archive/BUILD:9:12: in deps attribute of py_library rule @markupsafe_archive//:markupsafe: cc_library rule '@markupsafe_archive//:speedups' is misplaced here (expected py_binary or py_library).
ERROR: Analysis of target '//external:markupsafe' failed; build aborted.
INFO: Elapsed time: 0.213s

According to BazelPyRuleClasses.java, the only allowed rule types in py_library.deps is, in fact, py_binary and py_library:

  public static final String[] ALLOWED_RULES_IN_DEPS = new String[] {
      "py_binary",
      "py_library",
  };

What is the plan for Python-C interoperability? Should we update the documentation to remove the mention of depending on cc_library in the meantime?

The background for this is that I am trying to use Jinja for generating HTML for the Skylark docgen tool that I am working on. However, Jinja depends on Markupsafe, which is partly implemented in C.

[davidzchen]

Some related discussion here: https://groups.google.com/forum/#!searchin/bazel-discuss/pex/bazel-discuss/-DFOrVqK8aY/-7h4e69hfkEJ

[ulfjack]

Recategorizing this as a feature request to support Python-C++ interop. I marked the other report that asked for that as a dupe.

[evanj]

A workarounds I found: you can include pre-compiled .sos in the py_library's data attribute. No idea if you can somehow get the output of a cc_library to work in this way.

[benley]

The trick to building a .so that works as a python extension seems to be making it a cc_binary instead of a cc_library. I think you have to build it with linkshared = 1 to make it work, but I'd love to be wrong about that.

[MarkusTeufelberger]

I stumbled upon this one also recently. Please at least change the documentation or (ideally) increase the priority to get this fixed for good. Python libraries and binaries can and do depend on C libraries. This is currently not really easy to model in bazel.

In the meantime it would also be nice to have this workaround documented somewhere.

[buchgr]

@MarkusTeufelberger we love contributions 😃

[ulfjack]

The problem is that we don't know how to fix it. We could export our internal model, but we're unsure if that's what people want. Our internal model is that all cc_library rules in deps of py_library rules get merged into a single .so at the py_binary level, together with all cc_library dependencies of py_extension rules, except the py_extension code itself, which gets compiled into a separate .so. It sounds awkward, and it is, but there doesn't seem to be any other principled approach to avoid ODR violations in the C++ code.

[ThomasColthurst]

+1 for exporting the internal model. My project is considering switching from bazel to Makefiles solely to avoid the ODR violations we are currently seeing when using bazel + clif + protobufs.

[brandjon]

See related discussion in #1475.

[pipefail]

Is there any update on this / roadmap to look at?

Trying to plan ahead as would like to use Bazel for building x-plat Docker images which include the Postgres (psycopg2) driver that relies on the host system having libpq available.

Cannot currently do this is in a cross-platform / hermetic fashion as far as I can tell currently.

Host machine = OSX, Docker image = Debian

Work-around suggestions welcome though.

@brandjon

[quval]

I have a draft of a Starlark implementation of something like the internal Google model (all native deps are linked together into a statically-linked shared object; the modules themselves are dynamically linked to it). Would it be interesting to share this? Perhaps on rules_python?

[pvcnt]

@quval It would certainly be interesting yes. :)

[quval]

OK, here goes: https://github.com/quval/rules_native_python/
Got it to work on both Linux and Mac, but only with a toy setup, so I'd still consider this a draft. More importantly, I'm not sure I like the hack I did to work around the circular dependency (native deps -- compiled into native module -- linked against native deps). Any thoughts and suggestions are welcome.