Bazel: Design a publish / deploy story

[damienmg]

We should be able to provide easy way to deploy / publish artifacts. That include publish to artifactory / maven central, push to GCR or GCS, Android Play Store and so on...

We should also think of release process.

The order of priority:

• Artifactory / Maven central is requested by several people including the Gerrit team and Twitter heron (https://github.com/twitter/heron/issues/893)
• GCR / GCS for ci.bazel.io
• Others.

[damienmg]

Internal ref: b/25764843

[ronshapiro]

The dagger team is also very interested in this, because the Maven Android plugin doesn't suit our needs and we'd rather switch to bazel than gradle. Can you let me and/or @gk5885 know if there's anything we can do to help?

[damienmg]

Getting further in the though process.

For GCR for ci.bazel.io, we actually moved to use a run rules that calls gcloud docker push, and we do bazel run //gcr:deploy. That works pretty well so I think there is no Bazel side update to the story (but I may be wrong).

The only topic left right now would then be to implement a maven_deploy rules that would be able to talk to artifactory or maven central. @ronshapiro: wdyt?

[ittaiz]

@damienmg is 0.5 realistic? any chance to see a deploy story for maven jars landing this release?

[ronshapiro]

The actual maven_deploy logic would be nice, but simple enough to implement on your own since they're just HTTP PUT calls. I think the more interesting side of the deployment process is the packaging and creating of a maven pom.xml. I have a rough-and-dirty script which parses the WORKSPACE file for artifacts and then scans the deps+exports lists of our deps in Dagger, and then omits deps tagged "compile_time_dep". But it's still very much patchy. I'll post here when I sync that first version externally to github.com/google/dagger.

[damienmg]

@ittaiz I don't think we can do it by 0.5 moved to 0.6. But the design should be pretty straightforward as @ronshapiro described.

[ittaiz]

Fair enough. I suspected as such as didn't see any activity about it on bazel-dev.
@ronshapiro do you have any estimate when you'll have that first very rough version synced?

[johnynek]

I'm pretty skeptical publishing will be very easy.

Pants has actually tackled this quite a bit. The issue is: each internal node in the graph needs to be given a maven coordinate, or you need to somehow do some combining. Combining is really tough because future changes in the build graph could invalidate the clustering.

I guess you could do a naive thing and make one groupid for the whole repo, then publish every target as a separate artifact and everything gets one version, but you might not want to publish everything.

For small repos, it is probably easy, for big repos, not so sure.

pants at least, I think, gets this reasonable right:
http://www.pantsbuild.org/publish.html

[eaftan]

We'd like this for Error Prone as well. We'd like to switch from Maven to Bazel, but we need a way to publish artifacts to Maven Central.

[ronshapiro]

This is what I have so far. It's scrappy and doesn't send everythign to sonatype yet, but it generates the poms based on deps: https://github.com/google/dagger/blob/4bc66063f4b1a44323630e5bfae5649de13219df/util/maven/generate_poms.py

[ronshapiro]

Here's the script we're using to execute the deploy to maven central, invoking the previously mentioned script to generate the poms and associate them with the built artifacts: https://github.com/google/dagger/blob/cdff29ba11c92921fc44b9c57dda30a19c016fd6/util/execute-deploy.sh

[ronshapiro]

The generate_poms.py script that I wrote relies on parsing the WORKSPACE file for maven_jar rules, but it doesn't have visibility to rules that are defined in skylark macros. Is there any way to get access to those via bazel query, or can that be added?

[abergmeier-dsfishlabs]

We have a simpler case where we deploy via ssh + rsync. Our problem is that we want to execute the deploy in the build graph so we do not waste precious time. Also one cannot be sure what state is on the remote. Problem with that is, currently there seems no way of forcing execution of an action.

The workaround for us is to have a sub-graph _ActionFoo_ --depends--> _ActionShadow_.
_ActionShadow_ here generates 1 output file, which we then in _ActionFoo_ remove just so that we make sure that it gets invalidated.

[damienmg]

So docker_push is there. We need to work out maven export but I believe it falls under @kchodorow's hat now. Moving to 0.7 anyway.

[ittaiz]

We migrated a maven codebase to Bazel and needed to solve this so current
clients can still consume it. We've solved it by assuming targets map back
to the maven module they originated from, repackaging Bazel output to maven
granularity and then generating POMs. We hope to OSS that in the next few
months.
On Thu, 29 Jun 2017 at 12:28 Damien Martin-Guillerez <
[email protected]> wrote:

So docker_push is there. We need to work out maven export but I believe it
falls under @kchodorow https://github.com/kchodorow's hat now. Moving
to 0.7 anyway.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/1372#issuecomment-311913217,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFwTXqE4LGNMmlDfdnFvGoSK7bAqjks5sI25NgaJpZM4IzE-g
.

[btbuxton]

@ittaiz What is the status of the maven codebase conversion to Bazel that you talked about? Did it publish to a maven repository? Is there any place that I could go to look at the code?

[sffc]

My team is working on a new open-source project, and we'd like to use Bazel, and publishing to Maven Central, npm, pip, and other repositories is an important feature for us.

[ssuchter]

So I was just encounting this problem as well. Because Bazel doesn't have support for publishing artifacts, I of course was going to write my own. Seems like bazel query -output graph + pydot.graph_from_dot_file will be the way I go.

It seems like the chief challenge is in creating a synthetic pom with the right dependencies. There are certainly dependencies declared in the java_library rule that is used to create the jar I want to publish. But those dependencies travel through a chain until they get to the actual thing created by a maven_jar WORKSPACE entry. (I think these are the java_import rules that I see from bazel query -output label_kind)

When you traverse down that chain, you want to make sure that you only get the direct dependencies of the java_library rule (and get to their artifacts), rather than any indirect dependencies that might exist (presumably due to tools like generate_workspace or site-specific things).

@kchodorow, @damienmg - do you imagine your auto-publish mechanism will somehow solve this? If so, how?

[dinowernli]

+1 here, I am looking after a few bazel-piwered open source projects and having bazel generate the pom files (and perhaps even publish) would be very useful.

[davido]

Gerrit Code Review uses Bazel for publishing its artifacts to Maven Central. We have hand written pom.xml files, e.g.: [1], and we have maven_package rule, that uses mvn.py, that invokes mvn's deploy_file/install_file behind the scenes. To install the artifacts locally:

  davido@wizball:~/projects/gerrit2 (stable-2.15 %>)$ VERBOSE=1 tools/maven/api.sh install
  + bazel build //tools/maven:gen_api_install
INFO: Analysed target //tools/maven:gen_api_install (1 packages loaded).
INFO: Found 1 target...
Target //tools/maven:gen_api_install up-to-date:
  bazel-genfiles/tools/maven/api_install.sh
INFO: Elapsed time: 23.493s, Critical Path: 23.09s
INFO: Build completed successfully, 79 total actions
+ ./bazel-genfiles/tools/maven/api_install.sh
+ bazel build //gerrit-acceptance-framework:acceptance-framework_deploy.jar //gerrit-extension-api:extension-api_deploy.jar //gerrit-plugin-api:plugin-api_deploy.jar //gerrit-plugin-gwtui:gwtui-api_deploy.jar //gerrit-acceptance-framework:liblib-src.jar //gerrit-extension-api:libapi-src.jar //gerrit-plugin-api:plugin-api-sources_deploy.jar //gerrit-plugin-gwtui:gwtui-api-source_deploy.jar //gerrit-acceptance-framework:acceptance-framework-javadoc //gerrit-extension-api:extension-api-javadoc //gerrit-plugin-api:plugin-api-javadoc //gerrit-plugin-gwtui:gwtui-api-javadoc
INFO: Analysed 12 targets (0 packages loaded).
INFO: Found 12 targets...
INFO: Elapsed time: 0.315s, Critical Path: 0.13s
INFO: Build completed successfully, 1 total action
+ python tools/maven/mvn.py -v 2.15-rc2 -s gerrit-acceptance-framework:jar:bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/acceptance-framework_deploy.jar -s gerrit-extension-api:jar:bazel-out/k8-fastbuild/bin/gerrit-extension-api/extension-api_deploy.jar -s gerrit-plugin-api:jar:bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api_deploy.jar -s gerrit-plugin-gwtui:jar:bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api_deploy.jar -s gerrit-acceptance-framework:java-source:bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/liblib-src.jar -s gerrit-extension-api:java-source:bazel-out/k8-fastbuild/bin/gerrit-extension-api/libapi-src.jar -s gerrit-plugin-api:java-source:bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api-sources_deploy.jar -s gerrit-plugin-gwtui:java-source:bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api-source_deploy.jar -s gerrit-acceptance-framework:javadoc:bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/acceptance-framework-javadoc.zip -s gerrit-extension-api:javadoc:bazel-out/k8-fastbuild/bin/gerrit-extension-api/extension-api-javadoc.zip -s gerrit-plugin-api:javadoc:bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api-javadoc.zip -s gerrit-plugin-gwtui:javadoc:bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api-javadoc.zip -a install
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-acceptance-framework/pom.xml -Dpackaging=jar -Dfile=bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/acceptance-framework_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-extension-api/pom.xml -Dpackaging=jar -Dfile=bazel-out/k8-fastbuild/bin/gerrit-extension-api/extension-api_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-api/pom.xml -Dpackaging=jar -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-gwtui/pom.xml -Dpackaging=jar -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-acceptance-framework/pom.xml -Dpackaging=java-source -Dfile=bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/liblib-src.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-extension-api/pom.xml -Dpackaging=java-source -Dfile=bazel-out/k8-fastbuild/bin/gerrit-extension-api/libapi-src.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-api/pom.xml -Dpackaging=java-source -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api-sources_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-gwtui/pom.xml -Dpackaging=java-source -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api-source_deploy.jar
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-acceptance-framework/pom.xml -Dpackaging=javadoc -Dfile=bazel-out/k8-fastbuild/bin/gerrit-acceptance-framework/acceptance-framework-javadoc.zip
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-extension-api/pom.xml -Dpackaging=javadoc -Dfile=bazel-out/k8-fastbuild/bin/gerrit-extension-api/extension-api-javadoc.zip
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-api/pom.xml -Dpackaging=javadoc -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-api/plugin-api-javadoc.zip
mvn install:install-file -Dversion=2.15-rc2 -DpomFile=/home/davido/projects/gerrit2/gerrit-plugin-gwtui/pom.xml -Dpackaging=javadoc -Dfile=bazel-out/k8-fastbuild/bin/gerrit-plugin-gwtui/gwtui-api-javadoc.zip
Version: 2.15-rc2

To publish the artifacts on Maven Central: tools/maven/api.sh deploy is called.

• [1] https://github.com/GerritCodeReview/gerrit/blob/master/tools/maven/gerrit-plugin-api_pom.xml

[davido]

FWIW, Buck has a dedicated publish command to do the job. How does this work and know what Maven's coordinates and version to publish to? The concept relies on maven_coord attribute of java_library. I don't find the documentation, but it can be seen with this publish test fixture https://github.com/facebook/buck/blob/master/test/com/facebook/buck/cli/testdata/publish/BUCK.fixture#L4

java_library(
    name = "foo",
    srcs = glob(["*.java"]),
    maven_coords = "com.example:foo:1.0",
)

Then running buck publish --to-maven-central //:foo should just work. The integration test part is here.

[netdpb]

+Ron Shapiro [e-mail]

On Sun, Feb 4, 2018 at 2:32 PM David Ostrovsky [e-mail]
wrote:

FWIW, Buck has a dedicated publish command
https://buckbuild.com/command/publish.html to do the job. How does this
work and know what Maven's coordinates and version to publish to? The
oncept relies on maven_coord attribute of java_library. I don't fine the
documentation, but it can be seen with this publish test fixture
https://github.com/facebook/buck/blob/master/test/com/facebook/buck/cli/testdata/publish/BUCK.fixture#L4

java_library(
name = "foo",
srcs = glob(["*.java"]),
maven_coords = "com.example:foo:1.0",
resources = [
],
visibility = [
"PUBLIC",
],
deps = [
],
)

Then running buck publish
https://github.com/facebook/buck/blob/master/test/com/facebook/buck/cli/PublishCommandIntegrationTest.java#L124-L131
should just work.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/1372#issuecomment-362932783,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEG1yX6zN1e0OSPd1o77y2SWXglHZPx4ks5tRgXTgaJpZM4IzE-g
.

[ronshapiro]

I have more to write on this, but I think having that if maven_jar/java_import_external added "maven_coordinates=com.example.foo:foo-bar:version" to the generated java_import rule, we could go a long way with a combination of bazel query and some skylark macros to generate the pom file.

A nice-to-have would also to have a way to add your own tags, like maven:ignore, maven:shaded, etc.

[ittaiz]

maven_jar has this info.
You mean that dependencies don't have access to this (as opposed to this
being exposed in an aspect for example)?
On Mon, 5 Feb 2018 at 21:43 Ron Shapiro [e-mail] wrote:

I have more to write on this, but I think having that if maven_jar/
java_import_external added a tags =
["maven_coordinates=com.example.foo:foo-bar:version"], we could go a long
way with a combination of bazel query and some skylark macros to generate
the pom file.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/1372#issuecomment-363198641,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIFy62kFcUi9g1_sKiHlcc6z9dxd-zks5tR1negaJpZM4IzE-g
.

[ronshapiro]

@ittaiz I believe that's only accessible if you use the //external package. Those won't show up in bazel query deps(//foo:bar) that depends on a maven_jar (unless I'm doing something wrong).

[ittaiz]

I misread your comment, I now understand you suggested adding a property to the generated java_import rule.
I think one of the biggest issues I have with java_import_external is that it's missing a level of abstraction (no artifact attribute, no support for maven_server) and so also the attribute you're suggesting sounds a bit mismatched there. I think we need a maven_import_external that should probably write a new rule with this attribute since AFAIUjava_importis much more generic than maven. We (at rules_scala) have our own variant ofjava_importto work around some ijar issues and are considering copyingjava_import_externalso that we'll have ascala_import_external` that writes scala_import rules.
Now that you've pointed it out then maybe we'll try to take it up a level of abstraction.
cc @natasil who'll work on this

[ronshapiro]

I have a POC using Skylark to generate pom files assuming we can get the tags coming in. Is this something the Bazel team is open to? I'll try to clean up the POC and publish it out for review, and I'm happy to make the change to add the tags. It seems like valuable metadata even if we decide to go a different route for pom generation

[ittaiz]

Are you adding it to java_import?
If so,
Is that really the correct place? Since one can use java_import to import
any kind of files
On Thu, 8 Feb 2018 at 6:48 Ron Shapiro [e-mail] wrote:

I have a POC using Skylark to generate pom files assuming we can get the
tags coming in. Is this something the Bazel team is open to? I'll try to
clean up the POC and publish it out for review, and I'm happy to make the
change to add the tags. It seems like valuable metadata even if we decide
to go a different route for pom generation

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/bazel/issues/1372#issuecomment-364001669,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABUIF-FJ6waHs29ktJly8OT-DrFcELj0ks5tSnymgaJpZM4IzE-g
.

[ronshapiro]

This is what I'm suggesting: https://github.com/ronshapiro/bazel/commit/e85c79aa400173e72d14706a995bf46eb617382c, i.e. propagate the artifact attribute to the generated java_import.tags attribute. tags already exists in the set of "common definitions" so this would contain no changes to java_import itself.

Regarding what you mentioned about java_import_external - you're right, I made a mistake on that. I'm not sure what the right thing to do, but something that allows you to specify an artifact and a list of servers without hardcoding the full urls (since the extension is duplicated) is my hunch. Whether that's a change to java_import_external or a maven_import_external that delegates to java_import_external is secondary discussion.

[ronshapiro]

@jart would you be open to a tags attribute to java_import_external? And we could define a repository rule for maven that would delegate to java_import_external, follow the recommended repository order, and add the maven artifact to the tags?

[ronshapiro]

Here's a PR that shows an approach to generating pom files in Skylark. It would greatly benefit from the previous few comments about having maven_jar/a maven-flavor of java_import_external automatically adding the maven_coordinates=foo:bar:ver flags.

https://github.com/google/dagger/pull/1068

[ronshapiro]

FYI that https://github.com/google/bazel-common/commit/2782531da81d4002bce16e853953d9e8117a6fc1 added a pom_file rule. Would be happy to field any questions people may have about it.

[jart]

@ronshapiro Yes I absolutely be open to approving contributions to all the java_import_external copies that exist, adding any _PASS_PROPS you need, which would ideally be all of the standard ones. Please note that some of those standard properties, e.g. testonly needed to be specified as testonly_ since the repository rule itself accepted them for some reason.

[jin]

Grakn Labs open sourced a collection of rules to assemble and deploy to Maven, Pip, Homebrew, Packager and RPM. https://github.com/graknlabs/bazel-distribution

[amit-traiana]

Anyone figured a way to publish generated protobuf/grpc jars? Both the Grakn Labs solution and the pom_file rules supplied by @ronshapiro require you to use java_library. But java_proto_library can't be the src of java_library because java_proto_library doesn't output .java, .srcjar or .properties (what both Grakn Labs and the Dagger team do is to use the java_grpc_compile rule by rules_proto. Which this works for Java, I'm also trying to figure a way to make it work with Scala (and the Scala rule in rules_proto repo doesn't compile).

[jin]

We're in the process of adding Maven publishing rules to rules_jvm_external: https://github.com/bazelbuild/rules_jvm_external/pull/413

[jin]

rules_jvm_external 3.3 has the ability to publish Maven artifacts with java_export: https://github.com/bazelbuild/rules_jvm_external#publishing-to-external-repositories

API docs: https://github.com/bazelbuild/rules_jvm_external/blob/master/docs/api.md#java_export
Example project: https://github.com/bazelbuild/rules_jvm_external/tree/master/examples/java-export

cc @shs96c

I think we can consider this issue closed, since various ecosystems (Maven, Docker) now have publish/deploy rules using bazel run and runfiles deps.