Pytest: FEATURE: run pytest in Jupyter notebooks

[pylang]

nose.tools offers functions for testing assertions, e.g. assert_equal(). These functions are callable in a Jupyter notebook (REPL) and produce a detailed output if an error is raised. I've looked through the pytest docs, but those relative invocation methods seem to focus on the commandline.

I request an interactive, nose-like feature in pytest, i.e. invoke pytest in a Jupyter notebook and generate the pytest, detailed output for each cell.

# Jupyter cell
def func(x):
    return x + 1

def test_answer():
    assert func(3) == 5

# Out: pytest output

======= FAILURES ========
_______ test_answer ________

    def test_answer():
>       assert func(3) == 5
E       assert 4 == 5
E        +  where 4 = func(3)

test_sample.py:5: AssertionError
======= 1 failed in 0.12 seconds ========

[RonnyPfannschmidt]

i had prior discussions about enabling this with @userzimmermann

[pylang]

For reference, there are a few plugins that run tests on full .ipynb files ,e.g. nbval, but I am unaware of a procedures that actually run pytest inside a notebook. @akaihola attempts to resolve this barrier with ipython_pytest, an extension that uses Jupyter cell magics to express the pytest console output:

# Jupyter cell
%%pytest

def test_my_stuff():
    assert 42 == 42

His implementation appears concise. However, rather than inserting multiple cell magics throughout the notebook in various test cells, it would be favorable to initiate the pytest outputs globally. I am not certain what an efficient approach would be. Perhaps a single builtin magic, similar to %matplotlib that presently initiates different plotting backends, e.g. %pytest, %pytest <mode> --> pytest output for cells that contain valid tests ...

Ideas are welcome. Many thanks.

[akaihola]

@pylang, another shortcoming in ipython_pytest compared to my fork of ipython_nose (original by @taavi) is that I couldn't find a way to run tests in-process and inject the notebook's environment into the test.

In other words, what ipython_nose allows me to do is prepare the test environment in multiple separate notebook cells. I often import required modules in one cell, create some test data in another cell, and define the actual test in yet another cell.

This approach works great in ipython_nose and the %%nose directive, since it shoves all globals from the notebook environment into the dynamically generated test module.

With ipython_pytest, I need to include all imports, test data preparation and test functions in a single cell. Also, I can't share imports and data between tests in different cells.

I tried to study pytest internals to make ipython_pytest work like ipython_nose, but gave up pretty quickly since it seemed to either be impossible or require lots and lots of hacking. Do you think it would be possible with reasonable effort?

[userzimmermann]

Hi @pylang @akaihola :)

As @RonnyPfannschmidt pointed out I am also - and still ;) - very interested in this feature!

I will also try to come up with some concept asap... Or we try to work on some concept together somehow... Sharing effort to make it reasonable ;)

[jonnor]

Is the Assertion rewriter that introspects and annotates the exceptions thrown by the tests available?
With that one can easily collect basic tests in the notebook manually and call them to get nice errors.

[nicoddemus]

AFAIK the assertion rewriter only works on .py files, it might be possible to refactor its logic to support rewriting the assertions transparently inside Jupyter notebooks

[slayoo]

Just for the record, this solution seems to work:
https://github.com/chmp/ipytest/blob/master/example/Magics.ipynb

[chmp]

I am the author of ipytest and just stumbled upon this issue.

For reference: ipytest supports assertion rewriting by using either the run_pytest magic or the rewrite_asserts magic. As in

%%run_pytest

def test_foo():
    assert [] == [1, 2, 3]

If there is interest in integration of this functionality in pytest, I would be happy to help.

[chmp]

@pylang, for testing a whole notebook you could use:

# cell1
import ipytest.magics

# cell 2
%%rewrite_asserts
... # define tests

# cell 3
%%rewrite_asserts
... # define more tests

# cell 3
%%run_pytest -qq
pass

[nicoddemus]

Hi @chmp, thanks for chipping in!

I see that you are using the rewrite_asserts function directly:

https://github.com/chmp/ipytest/blob/master/ipytest/magics.py#L43

If you managed to use that successfully, I believe that's the way to go. It also doesn't need any changes to pytest itself. 👍

[chmp]

While I have to admit, that it did seem a bit hacky to use pytest internals, it works quite nicely in practice. The hardest part was getting stack traces to work :) (I typed up the details here). I have been using pytest from inside notebooks pretty regulary the last couple of months and did not run into any problems so far.

Edit: I changed the implementation. It is now using the ast_transformers and works transparently for the user once activated. The use of magics is no longer required.

[jfthuong]

@slayoo

Just for the record, this solution seems to work:
https://github.com/chmp/ipytest/blob/master/example/Magics.ipynb

FYI: link is now a 404.

[slayoo]

Here it is: https://github.com/chmp/ipytest/blob/299623a1a94fd58ed7a5230a53bc7219c8ae2ce6/example/Magics.ipynb

[pylang]

Perhaps this should be a new issue, but I noticed the new subtests plugin. This part caught my attention:

        ...
            with self.subTest("custom message", i=i):
                self.assertEqual(i % 2, 0)

The result is pytest-flavored tracebacks by way of context managers.

---

Might it be possible to make a context manager that hooks into the traceback rewriter? Example:

with pytest.assertions:
    def test_a():
        assert 1
        assert 0

test_a()

================================== FAILURES ===================================
___________________________________ test_a ____________________________________

    def test_a():
        assert 1
>       assert 0
E       assert 0

Having a way to hook into the assertion writer, decoupled from the test runner, could more generally extend pytest-flavored tracebacks to interactive Python, i.e. the REPL, jupyter notebooks (not solely .py files).

[chmp]

Having worked a bit on this issue, my guess is a context manager will not work, as you do need to to inject the rewriter before the cell is executed. Therefore it's most likely going to be either a magic or spreading out the code between cells.

If you're happy to install ipytest as an additional dependency, you could do theses things with the current version as follows:

# cell 1
import ipytest
ipytest.config(magics=True)

# cell 2
%%rewrite_asserts

def test_a():
    assert 1
    assert 0

or

# cell 1
import ipytest
ipytest.config(rewrite_asserts=True)

# cell 2
def test_a():
    assert 1
    assert 0

# cell 3
ipytest.config(rewrite_asserts=False)

In both cases, the asserts will be pytest flavoured. You can execute test_a either in the cell it was defined in or anywhere else.

If installing ipytest does not work for you for some reason, feel free to copy out the relevant parts from it. It is not too much code and it is licensed under MIT.

As an aside: I fundamentally changed the way rewrites work. The implementation is now much more lightweight by using jupyter's ast_transformers.

[sashgorokhov]

Stumbled upon similar case. But my environment is a bit special - i am working with databricks and they have their custom "notebooks" based on ipython. So, ipytest doesnt really works for my case. What i would like to see is in general more "programmatic" approach to run tests with pytest. Like, just pass it some test function objects and let it run them. This would help to solve many similar cases where tests are done not by static files but rather generated or created in dynamic way.