Bazel: Feature request: Documentation on the prelude

[davidsantiago]

I just came across the concept of the bazel prelude while trying to resolve another issue (https://github.com/bazelbuild/bazel/issues/1673), and I could not find any explanation of how the prelude is supposed to work anywhere. Searching bazel.io turns up nothing, and the google groups have only two passing mentions. It does appear to be used in another bazel project, as the Skydoc documentation instructs users to insert a snippet into their prelude. I think it would be helpful to get an explanation of what the prelude is for and how to use it effectively.

[ahyangyi]

Not really related but I wish could be fixed together: in-bazel error messages of BUILD files are messy when prelude is used.

Steps to reproduce:

• Get a working workspace that uses prelude.
• Find a BUILD file, add the following lines:

cc_library(

You get error messages that look like:

ERROR: /your/path/to/your/build/file/BUILD:51:9: syntax error at 'newline': expected ,
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:1:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:2:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:3:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:4:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:5:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:6:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:7:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:8:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
ERROR: /your/path/to/your/prelude_bazel/prelude_bazel:9:6: file '@//path/to/some/bazel/file/loaded/by/prelude.bzl' was not correctly loaded. Make sure the 'load' statement appears in the global scope in your file.
...

which is extremely messy, and I doubt the users will notice that the correct diagnosis is in the first line of hundreds lines of error messages.

[laurentlb]

@ahyangyi Thanks, we've now fixed the error messages.
Please file a new bug if you still see confusing messages.

[ahyangyi]

Thank you, @laurentlb !

[mfarrugi]

What are some use cases for a prelude? Is this a feature bazel is trying to get rid of?

I have seen adding load statements for language rules, but I would rather be able to put those rules into global scope in a WORKSPACE instead.

Is it free to have an unused load in a BUILD file?

[laurentlb]

The prelude file is not very well supported. Inside Google, we rarely use it (we used it a few times to assist us during a migration). The prelude file doesn't play well with incremental builds: if you modify it (or any of its dependencies), every package will have to be reloaded.

To automatically removed unused load statements from you BUILD files, try buildozer (https://github.com/bazelbuild/buildtools/tree/master/buildozer):
buildozer 'fix unusedLoads' path/to/BUILD

[alandonovan]

Laurent, could you confirm my understanding of the prelude?

• The prelude is an optional file whose contents are effectively textually inserted at the start of every BUILD (but not .bzl) file loaded by Bazel. It is thus a place to stuff workspace-global definitions.
• The location of the prelude is fixed, but the contents may be edited by users.
• Bazel's prelude is found at //tools/build_rules:prelude_bazel. By default there is no such file.
• Exoblaze uses the prelude to define various rules such as cc_binary, using built-in implementations.
• Blaze does not look for a prelude.
• In the future, implicit textual insertion will be replaced by implicit load-like semantics. Few users should notice the effects of that change.
• The prelude was never intended to be a feature. It was designed as a crutch to aid the Bazel team with their migrations. Ideally it would be removed.

Any advice on where this should be documented? Perhaps where we explain the default environment of a BUILD file?

[laurentlb]

All correct. We should recommend using only load statements in the prelude.

In the website, we can have a new page under Reference > BUILD files.

[alandonovan]

The prelude should not have its own page---we do not want to draw attention to it. It should be a footnote in a page describing the predeclared environment of a BUILD file: Starlark core operators, functions (glob, select ...), build rules, and so on.

Where in the source code would one add such a page? //site/_layouts/...?

[laurentlb]

The documentation of BUILD files is here: https://docs.bazel.build/versions/master/build-ref.html#BUILD_files

[alandonovan]

That page is the introduction to the core concepts. It is not the appropriate place for reference material on the predeclared environment of a BUILD file.

Where would one add such a page?

[laurentlb]

We can create a new page under Reference > BUILD files with all the semantics of BUILD files?

[alandonovan]

Yes! But where in the repository would the page's file reside?

[laurentlb]

In https://github.com/bazelbuild/bazel/tree/master/site/docs

[ahyangyi]

In the future, implicit textual insertion will be replaced by implicit load-like semantics. Few users should notice the effects of that change.

I would add that the current implicit textual insertion creates confusing error messages in case of bugs... which can only be solved when you replace it with implicit load-like semantics.

Hence, a change like that would definitely be noticed by me, but it's for the better!

[alandonovan]

Hence, a change like that would definitely be noticed by me, but it's for the better!

I meant to say "noticed by Starlark logic". But it's good to know the change will make at least one person happy. :)

[alandonovan]

I was about to send a CL to add this new file, but I realized it would be a case of "all the existing X are bad, let me create another one", where X in this case is "pages documenting the environment". This is a step in the wrong direction.

We desperately need to carefully reorganize our API reference docs (ideally without breaking URLs). The problem is that they are partly generated by a program, so this is actually a somewhat subtle logic change not a doc edit. We cannot entrust the task of organizing docs to our doc tooling (e.g. stardoc). It can generate parts of the docs but a human editor must decide how they are stitched together. Also, every document should explain how to edit it, even if that means editing .java code.

Ideally we would have one page with a complete index or ToC, organized into several logical sections, with each section being an explanation of the grouping and a set of links to a single page that explains all functions in that section fully.

1. Core Starlark built-ins: None, True, False, len, etc.
2. Common predeclared built-ins (select, depset, ...)
3. BUILD environment (native.package, native.glob, ... + all native.rules).
   Explain (or link to explanation) that BUILD files are "data" not "code" (no if/for/def/f(**kwargs).
   Explain that the native package is opened in BUILD files. Each name can be link to native.foo.
   Explain the prelude.
4. .bzl predeclared built-ins (rule, aspect, + hundreds of providers and suchlike)
5. native.xyz functions (package, glob, exports_files, etc).
   Can only be called from a BUILD thread.
6. native rules (these are also in the native package, and thus the BUILD environment.

Core built-ins are specified at https://github.com/bazelbuild/starlark/blob/master/spec.md#built-in-constants-and-functions. That doc needs a table of contents. (The one at go.starlark.net from which it was copied has one; not sure how it got lost.) They are also documented (from Java code) at https://docs.bazel.build/versions/master/skylark/lib/globals.html, but this doc also contains lots of things that don't belong. Also, we should call them predeclared built-ins, not globals.

Common functions (select, depset) are included here, along with core built-ins and .bzl things:
https://docs.bazel.build/versions/master/skylark/lib/globals.html

The native package is documented here
https://docs.bazel.build/versions/master/skylark/lib/native.html
'package' is currently missing but that will be fixed by CL 311541013.
This doc also needs a better explanation that native.foo is accessed as foo in a BUILD file but as native.foo in a .bzl file.

Built-ins common to BUILD and .bzl files: select, depset, ...
We don't have a good page for this. They are documented here, but mixed up with providers and all the .bzl stuff: https://docs.bazel.build/versions/master/skylark/lib/globals.html

This doc is a mix of native.x functions plus select (but no depset). Makes no sense.
https://docs.bazel.build/versions/master/be/functions.html

We should prefer fewer, larger pages, over many smaller ones (especially many non-intuitive groupings---I spent half my time clicking to find the seemingly arbitrary page on which a given function appears.) Browsers are quite capable of rendering big docs, they work best on slower networks, and users can jump around more easily.

[brandjon]

A couple corrections to the above:

It is thus a place to stuff workspace-global definitions.

Ever since c93ef7ba6443d7546294eacfe9d21573a11aad70, it's been local to a repo, so that the prelude definitions don't bleed between your dependencies and your main repo.

In the future, implicit textual insertion will be replaced by implicit load-like semantics. Few users should notice the effects of that change.

11940.

[aiuto]

cc: @floriographygoth @gregestren @jin