Elixir: mix xref --list file

[josevalim]

Hi @tonini (from Alchemist) and @msaraiva (from Atom),

@antipax is working on a feature in Elixir v1.3 that would allow you to list all functions being called on a given file. For example:

mix xref --list lib/foo.ex
Foo.bar/1:7
Bar.baz/1:19

Do you think such feature would be useful for Emacs and/or Atom? If there is an interest, let's discuss how it should work and its format. If there is no interesting, it is less code for us to write. :)

Have a good weekend!

[ericentin]

I think this could enable a "find all references" type of search, which would be cool, IMO.

[DiogoNeves]

I am new to Elixir so, apologies in advance if this already exists.
Wouldn't it be more useful to have the opposite? A list of all the files and functions that called a given file/function?

What's the use case in mind for the current feature suggestion?

Personally I'd like to be able to easily refactor code. This would be made easier if I could quickly list all the places that call the Function or File I'm updating and update them.

As a side effect, listing the callers instead of the suggested "callees" would make it easier to generate a call graph from a given function.

Thanks 😄

EDIT: Would also be cool to have, in Atom, a "context" of where a given function is called in my project. I can easily see what the current function calls but it's harder to visualise how it is currently used (called).

[josevalim]

@DiogoNeves that's a great suggestion which we could also implement. Let's wait for feedback from @msaraiva and @tonini. :)

[josevalim]

@DiogoNeves although I believe finding where a function is called can be easily done today using search. Search won't catch dynamic calls but neither will this feature.

[DiogoNeves]

@josevalim that's a very good point! 😄
I think both cases could be achieved using search but the list of called functions is definitely more complex and having it available helps. You changed my mind haha

As for dynamic calls, that sounds complex but incredibly useful! 👍

[ericentin]

For applys that we have all the information about at compile time (i.e. an apply in a macro with module, func and arity present), it might be a good idea to track (and check) those as well. Even for applies where we only have the module, or the func and arity, or the module and arity, we could do additional checks that there is at least some function somewhere that fulfills any compile-time constraints. Could be cool!

[ericentin]

Also, one nice thing about having the fully-expanded list of callers is that even if the module is aliased you can find all references to it/a function. Might miss some with simple text search otherwise.

[DiogoNeves]

@antipax those are great points!

would this fit under the same command? Something like:

mix xref --list-callers lib/foo.ex
mix xref --list-callers lib/foo.ex bar/1
# and the suggested behaviour
mix xref --list lib/foo.ex

I don't like my suggestion --list-callers but it's a way to start brainstorming names 😄

[ericentin]

I think any information about the call graph that we end up deciding to provide will end up under xref, yes. But as @josevalim says, let's let @tonini and @msaraiva weigh in before we make any decisions :)

[josevalim]

We probably want:

# Shows help
mix xref
# Prints unreachable modules and calls as warnings (used by mix compile.xref)
mix xref --unreachable
# Prints all file:line that call M.F/A
mix xref --callers Foo.bar/1
# Prints all calls (M.F/A) in the given file
mix xref --calls lib/foo.ex

But again, let's wait for use cases, otherwise feature creep is easy. :)

[tonini]

Hi @josevalim and others in this discussion :-)

I start to think about a refactoringtool if I read about the xref functionality, if you ask me.

For example for mix xref --callers Foo.bar/1 which would prints all file:line that call M.F/A could be useful. Yes there are search/replace tools in most IDE/Editors available but these mostly work with regex and getting the informations much more precise by Elixir it self is something I would like definitely more.

For the case of mix xref --callers Foo.bar/1 the output format could be extended with not just file:line but also the column informations like start:end. (That something I also would love to get from the tokenizer like @jfis once suggested ;-) https://github.com/elixir-lang/elixir/pull/3554)

I'm really appreciate it that you guys thinking about people like @msaraiva and me which try to develop some sort editor support/tools which should help coders to get an easier daily grind with Elixir. But I really won't steal you guys time with asking for such functionality if there are more important things to in the Elixir language project.

No matter which feature would be implemented with xref the output format which would be nice to have is something like the following:

Bar.baz/1:19:5:12 - M.F/A:LINE:START:END

Thanks

[josevalim]

@tonini including column information is really complicated in Elixir because it is very common for information to get lost through AST processing. We can still give it a try though, but that should be a separate discussion.

But I really won't steal you guys time with asking for such functionality if there are more important things to in the Elixir language project.

Don't worry about this. :) The hard work is already done to power other features and adding such listings will be really straight-forward. The answer we need from you is if they will be something useful for editors or not.

[msaraiva]

@josevalim, @antipax and others, I agree with @tonini and I think the mix xref --callers Foo.bar/1 would be an awesome feature. It would be very useful to develop much better search/refactoring tools. I never liked the idea of having such tools based on regex using the editor's native language, actually that's one of the reasons I developed the Elixir parser for Atom in Elixir itself!

I'm really glad you're taking the time to develop such tools.

Cheers.

[josevalim]

Perfect, we can work on --callers Foo.bar/1 next. Would --calls lib/foo.ex be useful to anyone at this point or not really?

[josevalim]

We have merged the callers feature. If there is an interest in changes or more features, just let us know!

[fishcakez]

https://github.com/lpil/mix-test.watch automatically runs tests when files in a mix project change. It would be nice to only run a subset of these tests based on the files that changed. If it was possible to build the callgraph of each test file with mix xref --list test/foo_test.exs then it would possible to only run test files when the relevant modules are compiled/changed.

/cc @lpil

[ericentin]

I think we could look in the manifest for the provided file, and then if it's not present we could try to compile it and dump the compile/runtime dispatches. Thoughts, @josevalim?

[josevalim]

I don't think we should use the compile.elixir manifest for storing test information. I don't think we need xref either. We would need to:

1. extend the parallel require (used by tests) and start storing this information from tests themselves
2. for every run, see on disk which beam files have changed since the last test manifest was wrote
3. go through the test manifest looking for tests that reference any of the modules found in step 2
4. re-run those test files

That said, we don't need the xref information. Everything we had in the manifest before the xref feature is enough for implementing this, since we basically want to handle stale tests in the same way we handle stale code in the Elixir compiler. We also don't need to build the callgraph because the compiler does it for us (by recompiling the beams that matter). Notice though the granularity here is test files: we re-run test files and not tests per se. Test-level granularity would require parsing beam files to build callgraphs.

@antipax if you are interested on this, you should definitely go ahead. Maybe we can have a mix test --stale flag. Once this flag is used for the first time, we build the manifest. Next time you run it, we compile the manifest time with the beam times, as I described above and run only the stale ones. It would be a really neat feature.

[josevalim]

We also don't need to build the callgraph because the compiler does it for us (by recompiling the beams that matter).

Actually, this is wrong. Elixir's compiler is smart and recompiles only compile time dependencies and in this case you also want to run tests for runtime dependencies. But this information can be retrieved from the compile.elixir manifest as well (in the runtime column). So overall it is 5 steps:

1. extend the parallel require (used by tests) and start storing this information from tests themselves
2. for every run, see on disk which beam files have changed since the last test manifest was wrote
3. traverse the compile.elixir manifest looking for any module that depends on the modules found on step 2 both at runtime and compile time recursively until no more tests are added
4. go through the test manifest looking for tests that reference any of the modules found in step 3
5. re-run those test files

[josevalim]

We should probably open a new issue for this as it is not xref related. :)

[ericentin]

@josevalim this sounds like a pretty awesome feature so I'm interested :)