Bazel: move rules_pkg out of Bazel

[aiuto]

//tools/build_defs/pkg:{pkg_deb, pkg_rpm, pkg_tar} are being pulled out of Bazel core and moved to https://github.com/bazelbuild/rules_pkg/tree/master/pkg. Users should migrate to the new versions.

Migration:

1. Add WORKSPACE stanza as specified in a rule_pkg release
   https://github.com/bazelbuild/rules_pkg/releases

2. Replace instances of

   load("//tools/build_defs/pkg:pkg.bzl", "pkg_tar", "pkg_deb")
   load("//tools/build_defs/pkg:rpm.bzl", "pkg_rpm")
   

   with

   load("@rules_pkg//:pkg.bzl", "pkg_tar", "pkg_deb")
   load("@rules_pkg//:rpm.bzl", "pkg_rpm")
   

   One line solution (for some users)

   find . -name 'BUILD' -o -name '*.bzl' | xargs sed -i -e '/^load/s#//tools/build_defs/pkg:#@rules_pkg//:#'
   

Expected timeline:

• The bazelbuild/rules_pkg repository is available for use now. Issues against packaging rules are being transferred to that repository and will be addressed there.
• Flag to be released in Bazel 0.29.
• Enabled by default in next Bazel incompatible flag flip release after 0.29
• //tools/build_defs/pkg/... removed from Bazel by EOY 2019

[dslomov]

Will the migration tooling be available to do this change?

[aiuto]

Will the migration tooling be available to do this change?

I could do special purpose tooling with a small python script. OR.. I could wait for a migration tool that can update WORKSPACE IFF any of the BUILD or bzl files were transformed.

The other thing we can do is not bother flipping now and get people to migrate by only adding new features to the new repo.

[hlopko]

Why did you decide to implement the incompatible flag as a build_setting? Our whole infrastructure relies on incompatible flags being native options. The most notable problems are that our buildkite pipeline showing the progress of the migration will not include this flag, and bazelisk --strict will not flip this flag so users relying on bazelisk won't notice this incompatible change at all.

I mean it's fine to improve the infrastructure to support build_settings in the future, and start using build_settings after that, but not the other way around. I'm afraid you'll have to either manually collect the migration progress data or replace the build_setting with native option.

[aiuto]

We need to address flags in build settings soon, so it will be possible to update Starlark based rules without modifying the Bazel binary.

I think this should never have been a flag in the first place. It should have been a "we are dropping support for this package at Bazel 2.0". Since it is low priority to move this (we have already stopped maintaining it), we could experiment with other migration paths.

• We could add a deprecation warning and sit on it for a year, then drop it
• We could simply take it out of the 2.0 prerelease and let CI run. People who use the migration tooling will break, in just the same way they would if it were a native flag.

[aiuto]

I started work on https://github.com/bazelbuild/bazelisk/issues/118, which should allow the migration CI to pick this up automatically. It may be late for 3.0, but at least it is moving forward.

[aiuto]

This keeps slipping for a variety of reasons involving pkg_tar being deeply intertwined into integration tests. I split out the removal of pkg_deb and pkg_rpm to distinct flag flips. This way we can get those removed early to decrease chance of user backsliding.

See: #11217 (deb) #11218 (rpm)

[laurentlb]

Shall we close this issue?

[aiuto]

No. It still should happen. It is just going to take a while because of the way we entangled zip and tar building into our integration tests. So I'll split out rpm and deb first, then come back to tar after fixing the dist-build process and various workspace tests.

[siddharthab]

Will it be possible to rename the title of this issue to have the prefix @bazel_tools? bazelisk uses issue titles and labels to check for migration readiness for users of bazel. Because most users will operate in their own workspace, this flag will be available to them only through the @bazel_tools workspace.

[aiuto]

I'm not understanding how this causes a problem for you. When you use bazel
3.3 with that flag are you seeing a command line parsing error?

On Tue, Aug 4, 2020 at 8:30 PM Siddhartha Bagaria [e-mail]
wrote:

Will it be possible to rename the title of this issue to have the prefix
@bazel_tools? bazelisk uses issue titles and labels to check for
migration readiness for users of bazel. Because most users will operate in
their own workspace, this flag will be available to them only through the
@bazel_tools workspace.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/8857#issuecomment-668899482,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHCA426P3UOMUFYK3RDR7CR3DANCNFSM4H75TRQA
.

[siddharthab]

Yes, it is a command line parsing error.

Take any external project, say, bazelisk, just for an example.

$ git clone [email protected]:bazelbuild/bazelisk.git
$ cd bazelisk

$ bazel version
Build label: 3.4.1
....

$  bazel build --//tools/build_defs/pkg:incompatible_no_build_defs_pkg //:bazelisk
...
ERROR: Skipping '//tools/build_defs/pkg:incompatible_no_build_defs_pkg': no such package 'tools/build_defs/pkg': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /Users/sbagaria/Workspace/bazelisk/tools/build_defs/pkg
WARNING: Target pattern parsing failed.
...
ERROR: //tools/build_defs/pkg:incompatible_no_build_defs_pkg :: Error loading option //tools/build_defs/pkg:incompatible_no_build_defs_pkg: BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /Users/sbagaria/Workspace/bazelisk/tools/build_defs/pkg

$ bazel build --@bazel_tools//tools/build_defs/pkg:incompatible_no_build_defs_pkg //:bazelisk
...
INFO: Found 1 target...
Target //:bazelisk up-to-date:
  bazel-bin/darwin_amd64_stripped/bazelisk

[aiuto]

This has no chance of being completed for 4.x.

• I was able to remove pkg_rpm
• pkg_deb can happen over the next few weeks
• getting rid of pkg_tar is going to take a lot of time. It is blocked on issues that are low priority for others.
  
  
  
  • https://github.com/bazelbuild/bazel/issues/11183
  
  
  • https://github.com/bazelbuild/bazel/issues/11184
  
  

[lberki]

@aiuto , are you actively working on this? (otherwise I'll demote this to P3)

[aiuto]

It's a 5% project. It should be p3 or p4.
I expect pkg_deb will go in the next 3 months, but that is covered
elsewhere.

On Wed, Nov 18, 2020 at 10:38 AM lberki [e-mail] wrote:

@aiuto https://github.com/aiuto , are you actively working on this?
(otherwise I'll demote this to P3)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/8857#issuecomment-729761697,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAXHHHELOU2VWLYQ4CE3LBDSQPS6ZANCNFSM4H75TRQA
.

[lberki]

Okie. If it's actively worked on (even as 5%), I'll keep it at P2 and assigned to you.

[limdor]

I guess that I am having the same issue like @siddharthab. If I take a very simple bazel example and try to run it with the incompatible flag or with bazelisk migrate it fails.
However I do not have any mention of //tools in the code.
@aiuto is this ready for the users to be migrated? I saw that @philwo added the migrate flag for 4.0 but I do not see anything that I can do as a user to fix the failure with the migrate flag.

[aiuto]

No one should be using @bazel_tools/rules_pkg any more. They should use github.com/bazelbuild/rules_pkg.
I think we can fix this by removing the migration labels and/or changing the title of this. I'll do both.
That said, it is just going to disappear from Bazel at one point, with no migration warning.

[aiuto]

Status update
github.com/bazelbuild/rules_pkg now has .deb packaging working on linux, macos and Windows.
I plan to remove pkg_deb from Bazel around March 2021.

No timeline for pkg_tar removal.