Bazel: Please implement output format 'graph' for cquery and aquery

[konste]

Description of the problem / feature request:

Bazel 'query' command supports output format 'graph' which is very useful for visualization of dependencies. Unfortunately cquery and aquery commands introduced later do not support 'graph' output format and it would be great to have it implemented.

Feature requests: what underlying problem are you trying to solve with this feature?

Graph visualization with 'query' gets very complex for multi-platform and multi-configuration builds. It can be made ten-fold less busy and more readable if produced by cquery for particular platform and configuration, which would make it substantially more usable.

What's the output of bazel info release?

2.1.1

[gregestren]

Hi @konste,

For cquery, We recognize this as a valuable feature request. The cquery core devs currently have the resources to do probably one or two cquery features per quarter. We're seriously considering this one for next quarter (starting April). We'll update as we're able to prioritize.

FWIW, I don't think this is technically too hard. If anyone's willing to volunteer to make the change we're happy to consult and review.

I can't speak for aquery, which I'm not as familiar with.

[gregestren]

We've (Bazel configurability) officially prioritized this for Q2.

[gregestren]

@meisterT - any thoughts on aquery?

[konste]

@gregestren I am not a Java developer and cannot contribute directly, but as a stop-gap solution for this problem I have a tiny Python script which converts cquery jsonproto output to .dot graph. You can find it at the end of this thread: https://groups.google.com/forum/#!msg/bazel-discuss/NMoRLSthQHY/quk9E-ACCAAJ It works and confirms that graph generation from cquery is not technically challenging.

[meisterT]

I have a similar script like @konste somewhere lying around. I can brush that up and make it work with the v2 proto output of aquery and then publish it. However it turned out to be less useful (at least internally) than I expected since even subsets of action graphs are so large that they don't render with graphviz anymore.

[konste]

For the actual visualization of .dot I don't use Graphviz. Instead, I apply dot2dgml and use Visual Studio to render resulting .dgml. It can render bigger graphs and adds much needed interactivity to help research it.

[mihaimaruseac]

TensorFlow would greatly benefit from this. I'll see if I can use the scripts mentioned in recent commits, though wfh makes this somewhat harder :)

[konste]

@gregestren
I noticed that any cquery on the target which uses toolchain produces a new WARNING pointing at the toolchain. For instance:

bazel cquery "deps(tests/cpp:hello_world)" --noimplicit_deps --notool_deps
INFO: Analyzed target //tests/cpp:hello_world (33 packages loaded, 3486 targets configured).
INFO: Found 1 target...
WARNING: Targets were missing from graph: [ConfiguredTargetKey{label=@tab_msvc_toolchain_repo_64//:tab_msvc_cc_toolchain_64, config=BuildConfigurationValue.Key[5abb50d874a0ab83869e8c0d685014657df6f34d02bb0ebbb59b2108192fcfff]}]

Any clue why toolchain related rules may be causing it and how to fix it?

[gregestren]

@konste - interesting. Is this easy to reproduce? If so, could you file a dedicated issue for it?

[konste]

@gregestren It took me some time to make a minimized repro, but now I have it and here is the dedicated issue: https://github.com/bazelbuild/bazel/issues/11993

[gregestren]

Tentatively good news: I prototyped cquery --output=graph yesterday successfully, and I think the approach is feasible and not too hard. I'm going to try to put up a proper PR today. I'll update this thread by EOD either way.

[gregestren]

EOD update: didn't get time to put in tests, but basic PR at https://github.com/bazelbuild/bazel/pull/12248.

[gregestren]

Done, but I must emphasize again this only covers cquery, not aquery.

In case anyone missed the comment on the PR, check out this extremely cool graph visualization demo from another tool: https://youtu.be/GDbaBOCDwrQ?t=98.

[konste]

Is the feature released? In what Bazel version? I don't see it in 3.7.0.

[gregestren]

3.71's baseline commit was on October 5 and this commit was October 19. So I suppose not.

It looks like 4.0 will have it (much newer baseline). Since that's the first iteration of Bazel's long term support policy, I don't know what that means for future 3.x releases. (i.e., you'd have to upgrade to 4.x to get this?)

[konste]

I plan to keep up with the monthly releases and upgrade to 4.0 and 4.x. Don't have the patience to wait for the new features for 9 months. :-)

[mihaimaruseac]

Bazel 4.0 has the graph output format, just tested with the fresh release.