Pytest: No longer can extract message from exceptions in 5+ versions

[cscetbon]

I can't extract the message of an exception using str(my_exception) when using pytest 5+. Here is an example :

E       AssertionError: assert 'blabla' in '<ExceptionInfo Exception tblen=1>'
E        +  where '<ExceptionInfo Exception tblen=1>' = str(<ExceptionInfo Exception tblen=1>)

That test succeeds in previous versions (4.6.4 for instance)

I've been able to test it using the following example :

import pytest

def test_ex():
    with pytest.raises(Exception) as e:
        raise Exception('blabla')

    assert 'blabla' in str(e)

[banderson5586]

assert 'blabla' in str(e.value)

Haven't looked at release notes yet to see why but accessing the exception text changed slightly.

[cscetbon]

@banderson5586 but why accessing the error message using pytest is different from the way it's done in python 2/3 ? it does not make any sense to me.

[nicoddemus]

Hi,

Here's the related CHANGELOG entry:

5412: ExceptionInfo objects (returned by pytest.raises) now have the same str representation as repr, which avoids some confusion when users use print(e) to inspect the object.

So it was mostly to avoid confusion. Note that pytest.raises returns a ExceptionInfo object when used in a with context, not the exception object itself.

Here are the docs for ExceptionInfo: https://docs.pytest.org/en/latest/reference.html#exceptioninfo

As can be seen in the docs, ExceptionInfo objects contain a lot of other attributes, so what happened before is that it was a little confusing when someone printed an ExceptionInfo object and it behave as if it was the exception value.

[cscetbon]

Understood. This change is probably gonna break a ton of unit tests ...

[nicoddemus]

Let's see, we are hopíng not, as this was not really documented behavior.

I'm closing this for now, feel free to follow up with further questions though.

[stadelmanma]

@nicoddemus I understand the change but it did smash a good bit of my test suite out of the blue when my CI suite pulled v5 instead of v4.X. I didn't realize I was using undocumented behavior which in the end is on me but I suspect several others will get hit in the face by this as well.

Maybe adding an extra note somewhere that statements such as assert "string" in str(err) will no longer work would help with the pain. I read the CHANGELOG prior to finding this issue but I didn't realize the impact of #5412 until seeing it detailed out here.

[cscetbon]

I don't know @stadelmanma how much time you spent in order to find this issue but I still think it would make more sense to support the old behavior and avoid breaking uncountable tests.

[nicoddemus]

Maybe adding an extra note somewhere that statements such as assert "string" in str(err) will no longer work would help with the pain

Sorry this caused problems, we usually try to avoid that as much as possible.

I would be glad to review/merge a PR with improved docs and/or to the README! 👍

[blueyed]

(Just for information / as a data point: I've also seen this rather often in test suites (but not my own ;)) upgrading to pytest 5 by now. But usually without much effort (AFAICS).)

[wjwwood]

Broke quite a few of our tests too, I agree with @stadelmanma's comment (https://github.com/pytest-dev/pytest/issues/5579#issuecomment-513936649). I didn't connect the CHANGELOG content to this consequence either.

Edit: README -> CHANGELOG

[stadelmanma]

I agree the README would provide better visibility but I don't think this kind of notice fits well on it. The CHANGLOG is the more appropriate place and maybe somewhere in the general docs but I'll need to scan over those tomorrow to see if any pages make sense.

@cscetbon When I saw all my tests break I took the following steps:

1. Check to see if there was a major version change in the pytest installed (check)
2. Seeing a new major version, check the CHANGELOG for any obvious breaking changes (miss)
3. Check if any relevant issues seem to describe the problem (check)

All in all it only took me about 5-10 minutes to figure out the problem since you already found it. I imagine most devs will probably take the same steps unless they've never been burned by a dependency changed. (Eventually Google SEO will probably funnel people here directly)

@nicoddemus, perhaps there are a few options to allow this behavior back:

1. Allow this as deprecated behavior, disabled by default but enabled via config opt? This gives people time to migrate away from the pattern.
2. Allow "in" to work like str(err) used to by having the error text as part of the output when str/repr is called.
3. (still requires code refactors but pairs well with opt 1.) Allow "in" to work on an ExceptionInfo object as if we were doing str(err) in v4.x but we would do "foo" in err instead of "foo" in str(err).

[wjwwood]

Sorry, README was a typo, I meant CHANGELOG too.

[nicoddemus]

Definitely meant CHANGELOG as well, not README, sorry about that.

@nicoddemus, perhaps there are a few options to allow this behavior back

Those are all sensible solutions, thanks, but I'm not sure adding a new option (and now having to maintain it until at least pytest 6.0) will actually help things. I believe it will probably only add to the confusion.

[cscetbon]

@cscetbon
All in all it only took me about 5-10 minutes to figure out the problem since _you_ already found it. I imagine most devs will probably take the same steps unless they've never been burned by a dependency changed. (Eventually Google SEO will probably funnel people here directly)

Neat. I guess you’re right, that issue helps understanding it faster. I’m not sure everybody will look into the dependencies but yeah Google will hopefully funnel them here.

@nicoddemus, perhaps there are a few options to allow this behavior back:

1. Allow this as deprecated behavior, disabled by default but enabled via config opt? This gives people time to migrate away from the pattern.

Could be interesting and that’s usually what I expect from a breaking change ^. I wouldn’t vote for a config option though. It could be deprecated but supported for a few versions before being unsupported.

[nicoddemus]

Could be interesting and that’s usually what I expect from a breaking change ^. I wouldn’t vote for a config option though. It could be deprecated but supported for a few versions before being unsupported.

Agree, in retrospect I think that would be the best course of action. At the time we didn't thought it would bring much breakage... 😞

I opened https://github.com/pytest-dev/pytest/pull/5648 to at least make that more explicit on the changelog.