Elixir: mix xref graph should provide more information about compile time dependencies

[liamwhite]

Environment

• Elixir & Erlang/OTP versions (elixir --version): Elixir 1.9.4
• Operating system: Linux

Behavior

I am trying to untangle a complicated module dependency graph in my project due to slow recompilations. When I change certain files, large portions of the project will be recompiled. There are over 3000 edges in the output of mix xref graph, and the dot graph is entirely unreadable due to the large amount of edge crossings.

In order to facilitate the process of fixing this, mix xref graph needs the following, which it currently does not have:

• Filtering for indirect compile dependencies. When the --label compile flag is used in combination with --sink, mix xref graph only produces portions of the dependency graph that have a direct compilation dependency on the sink file. If other files which indirectly depend on the sink file will also be recompiled, mix xref graph does not show what they are, and the command will generally produce no output. Other debugging techniques like the inotify technique presented here are also unable to narrow down the source of recompilation, because the parallel compiler outputs the beam files in an order which is not necessarily the dependency ordering
• Identification of cycles (strongly connected components) in the dependency graph.

[josevalim]

They are both good ideas. I would suggest for those to be new xref commands, especially to make the API easier to use.

[josevalim]

@wojtekmach I think you had some ideas here?

[wojtekmach]

Filtering for indirect compile dependencies.

Yeah that's exactly what I had in mind too. I'm not sure exactly how this would work but it's very important information to surface.

[josevalim]

Maybe “mix xref dryrun file” which emits a tree emulating what would be recompiled if something changed?

[wojtekmach]

Yes, showing a tree sounds good to me. FWIW I currently do the following: touch lib/foo.ex && mix compile --verbose which is obviously much slower. That would be a nice improvement.

[epilgrim]

just FYI, when faced with analyzing the vast data of the mix xref dot file content because of the same compilations slowdowns, I used this python script to narrow down the cycles:
https://github.com/jantman/misc-scripts/blob/master/dot_find_cycles.py

using ./dot_find_cycles.py --only-shortest xref_graph.dot

some pip dependencies are there in the file, so don't forget to install them

[josevalim]

We have made --label compileto be transitive by default and we have also added --format cycles. See changelog in master.