Pytest: Change junit_family default to xunit2 in pytest 6

[nicoddemus]

In order to do that, we also need to show a warning if the user does not have junit_family configured stating that the default will change in 6.0

[Zac-HD]

Per https://github.com/pytest-dev/pytest/issues/6265#issuecomment-557777175 we already have such a warning - that issue is just to expand and clarify the message.

[Zac-HD]

@blueyed - I think it's worth keeping this open so we remember to change it in pytest 6?

[blueyed]

@Zac-HD
GitHub failed me when I only wanted to close my issues, sorry.

[Zac-HD]

No worries, easily fixed 😄

[bittner]

Running tests with pytest now shows the warning you mention. But it's unclear what the impact of this change is, hence what should I as a developer actually do?

• _Is it required to add the junit_family=xunit1 line to my pytest configuration?_
• _Will my CI build be affected or something (aka, can I ignore this warning if I don't use Jenkins)?_

.tox/py37/lib/python3.7/site-packages/_pytest/junitxml.py:417
  <redacted>/_pytest/junitxml.py:417: PytestDeprecationWarning: The 'junit_family' default value will change to 'xunit2' in pytest 6.0.
  Add 'junit_family=xunit1' to your pytest.ini file to keep the current format in future versions of pytest and silence this warning.
    _issue_warning_captured(deprecated.JUNIT_XML_DEFAULT_FAMILY, config.hook, 2)

-- Docs: https://docs.pytest.org/en/latest/warnings.html

The link provided doesn't seem to have information related to the impact of the change, but rather on how warnings are handled and can be controled.

➜ Where can I find information about the impact that this change will have?

[nicoddemus]

Hi @bittner,

Actually the important bit is here (emphasis mine):

Add 'junit_family=xunit1' to your pytest.ini file to keep the current format in future versions of pytest

So yes the answer is to add that line to your pytest.ini to avoid having your suite breaking when upgrading to pytest 6. 👍

[bittner]

I did read (and understand) _that_, but why should my test suite break for a jUnit output format? Note that I guess it's all about the --junitxml option. My tests won't break, will they?

So, if they won't, what _will_ break? What are you warning about? Should I (and others) care? What would the "upgrade path" look like?

[nicoddemus]

I did read (and understand) that, but why should my test suite break for a jUnit output format? Note that I guess it's all about the --junitxml option. My tests won't break, will they?

Oh I see. Sorry, I misspoke: no, the test suite won't break of course, but related tools that expect the format to be JUnit1 will break, because the default will be JUnit2 unless specified to be JUnit1 in the pytest.ini file.

What would the "upgrade path" look like?

If currently you expect the XML format generated by pytest to be JUnit1, you should add junit_family=xunit1 to your pytest.ini.

[bittner]

Do we know which popular CI systems or tools would _not_ support JUnit2?

For example, I could not find a mention of JUnit2 in the related GitLab documentation. If we do know, or if we at least know which ones will _continue to work_ after the change, it may be worth noting this somewhere. Somewhere related to the deprecation notice.

[RonnyPfannschmidt]

there is no mention of junit2 anywhere else since this numbering is purely for the identification of the pytest internals

[jaraco]

I have the same question as bittner.

It's weird to me that all the docs talk about how to retain the legacy behavior, but there's very little information on how to update to the preferred behavior. I'd much rather update to xunit2 now if I can, rather than pin to some legacy default and have to address this transition again in the future. What I'd like to know:

• When was xunit2 introduced in Pytest (when is junit_family=xunit2 supported)?
• Will older versions of Pytest silently ignore the junit_family setting?
• What is the effect of the change to xunit2?

this numbering is purely for the identification of the pytest internals

o_O

Are you saying that the output targets the same spec, just has different internal processing? In that case, how is compatibility a concern?

What was the motivation for making this change in pytest?

[jaraco]

What was the motivation? When was the change? What was the effect?

Looks like #3547 introduced the change, released with v4.2.

Based on my reading, it seems likely that many (most) backends will support xunit2, so I'm going to attempt to upgrade my environments.

[jaraco]

I tried adding junit_family=xunit2 and downgrading a tox environment to pytest<4.2, but pytest 4.1.1 appears to be buggy (#5903).

[jaraco]

Will older versions of Pytest silently ignore the junit_family setting?

Appears yes.

[jaraco]

I've confirmed that Azure pipelines appears to support the xunit2 format.

[nicoddemus]

Hi @jaraco,

What was the motivation for making this change in pytest?

See https://docs.pytest.org/en/latest/deprecations.html#junit-family-default-value-change-to-xunit2

Given that this message seems to be confusing some people, I will extend the docs a bit and make the message point to the docs instead. 👍

[bittner]

Note that both here, in this conversation, as well as in the docs you just mentioned, you (as in "pytest" and "the docs") seem to assume that people would want to stick to the old format. It might be likely that the opposite is true, though.

When will the deprecation warning disappear? Is it possible to silence the warning without sticking to "legacy"? Please also take care of people wanting to go the upgrade path.

[nicoddemus]

Indeed the warning assumed people were happy with the current default (legacy) and would not like to have this changed when pytest 6.0 comes around.

I've opened a PR hopefully addressing those points, please let me know what you think in the comments. 👍

[bittner]

I hope it will not be necessary to explicitly define the junit-family in pytest >=6.0, so we can keep the configuration file small.

[nicoddemus]

It's not, the default will be xunit2 if it is not defined.

[nicoddemus]

Oh and the warning will be gone too.

[nicoddemus]

Done in https://github.com/pytest-dev/pytest/pull/7660