Numba: coverage.py support for Numba

[seibert]

It would be extremely helpful to be able to capture line coverage in Numba-compiled functions the same as in Python functions with coverage.py.

Doing this would most likely require something like the following:

• Numba compiler pipeline detects when coverage.py is active and injects a bit of code after each line to register execution of "line X in file Y" in a data structure. We might need to switch on compiler debug mode to avoid code relocation that would confuse the measurement.
• We could make the data structure hang off the Environment object that is passed to every function call. It is append-only, but needs to be thread-safe.
• When transitioning back to the interpreter, we should run down the log of executed lines, populate the relevant data structure in coverage.py, and clear the log.

@cdeil noted on the mailing list that Cython has done this already. Relevant links:

• http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html
• https://cython.readthedocs.io/en/latest/src/tutorial/profiling_tutorial.html#enabling-coverage-analysis
• https://github.com/cython/cython/blob/master/Cython/Coverage.py

[stuartarchibald]

I think this needs a bit of research (some of which is done already) but I'm wondering if we can just lean on LLVM's existing tooling to do the work and then parse the output.

[sklam]

I figured out how to make this work. No need to do any plugin for coverage.py. It's about faking a Frame and triggering a trace event on each line. That's how cython does it.