Bazel: incompatible_load_python_rules_from_bzl: Python rules now need to be loaded from @rules_python

[brandjon]

Flag: --incompatible_load_python_rules_from_bzl
Available since: 0.29
Will be flipped in: 2.0
Feature tracking issue: #8893

Motivation

The "core" Python rule logic is currently bundled with Bazel as a combination of native symbols and symbols loaded from @bazel_tools//tools/python. Eventually the native rules will be ported to Starlark, and all symbols will live in bazelbuild/rules_python. In the meantime, we're making it so the user loads these symbols from the rules_python repository, even though they're largely still implemented inside of Bazel itself. This will help make future migration of rule logic to rules_python less painful.

Change

Enabling this flag causes the four native Python rules,

• py_library
• py_binary
• py_test
• py_runtime

to fail at analysis time when they are used directly instead of via rules_python.

Migration

Any usage of the these rules as a built-in symbol should be replaced with a use of the macros defined in rules_python as follows. (All four macros are defined in defs.bzl.)

# My BUILD file

py_binary(
    ...
)

should become

# My BUILD file

load("@rules_python//python:defs.bzl", "py_binary")

py_binary(
    ...
)

In the case of macros in .bzl files,

# My .bzl file

def my_macro(...):
    native.py_binary(
        ...
    )

should become

# My .bzl file

load("@rules_python//python:defs.bzl", "py_binary")

def my_macro(...):
    py_binary(
        ...
    )

To access the @rules_python repo you'll need to make sure you have the following in your WORKSPACE file:

# My WORKSPACE file

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "rules_python",
    remote = "https://github.com/bazelbuild/rules_python.git",
    commit = "4b84ad270387a7c439ebdccfd530e2339601ef27",  # (2019-08-02 or later)
)

load("@rules_python//python:repositories.bzl", "py_repositories")
py_repositories()

If you're using rules_python for pip support (which is separate from the core Python rules), you'll also need to call the pip_repositories() function.

Note that until recently, @rules_python went by the workspace name @io_bazel_rules_python. If you have a definition for the old name in your WORKSPACE file then you'll need to rename it, and ensure your commit is updated to a recent version.

[brandjon]

The downstream incompatible change pipeline won't test this until 0.29 is released, but in the meantime I did a custom downstream run with the flag flip here. Looks like several dozen failures, maybe almost half of all configurations in the pipeline.

[dslomov]

Is there a tool to facilitate this migration? (it is a mechanic change, so buildifer should help)

[brandjon]

Not at the moment, I'd like to add one but not sure how long that'll take. The changes amount to 1) adding WORKSPACE boilerplate, 2) adding load() statements to BUILD files, 3) (rarer) adding load() statements to .bzl files and replacing native.<py_rule> with <py_rule>.

[brandjon]

Will be added by bazelbuild/buildtools#691. It doesn't currently do WORKSPACE files though -- looks like the java and cc bzl migrations don't do that either?

[brandjon]

Update: We will not be flipping this flag (or the analogous flags for java and cc) in Bazel 1.0. The main reasons are:

• The blast radius is high. Pretty much every BUILD file in every project needs to be updated. There is tooling to do this automatically in buildifier, however...

• The tooling does not yet automatically update WORKSPACE files. Users instead have to manually copy/paste the right incantation into their WORKSPACE from rules_python/java/cc's readme.md page. The Bazel federation is also not in a position to make this change easier yet.

• The benefit is not large enough to overcome these issues. Although it's a step toward migrating the rules out of Bazel, as long as the actual rule implementations still live in Bazel we still have to support them for the duration of the 1.x development cycle.

With this in mind we'll look at targeting Bazel 2.0 instead. In the meantime we can improve tooling and add new flags for more related deprecations/migrations.

[aaliddell]

What's the timescale on 2.0? It 'sounds' a long way off but that depends on the release cycle chosen for Bazel after 1.0.

Also, is the intention to replace the native implementations with the Starlark implementations (similar to @bazel_tools, but without the load) during the 1.0 lifetime, but not force the manual addition to workspace and load in BUILD until 2.0? That doesn't solve having to support both, but starts getting the Starlark implementations battle tested.

[brandjon]

At this point the only thing set in stone about 2.0 is that it'll be at least 3 months.

Note that there is no Starlark implementation of the native Python rules -- only redirects in rules_python that export the native implementations as if they were defined in Starlark. Migrating the implementation to Starlark is not necessarily blocked on getting people to load from rules_python first. Only deleting/modifying the native rules is blocked on that.

[Yannic]

What's the status of this flag? Is it going to be flipped eventually or has it been abandoned? If it's the former, can we add migration-X labels for 2.x and 3.x so it'll be detected by Bazelisk and tested on Buildkite?

[wchargin]

+1 to @Yannic’s question: this flag was supposed to be flipped in Bazel
2.0, but still isn’t flipped in Bazel 3.7, which released almost a year
later. Is it still realistically planned? This matters for downstream
rules_* maintainers; we need to know whether we have to go to the
trouble of pulling in a rules_python dependency for helper scripts.