Pytest: implement support for PEP-518 - the tool.pytest key in pyproject.toml

[RonnyPfannschmidt]

as per https://www.python.org/dev/peps/pep-0518/#specification

the file has a section for tools using the pypi name for the tool,

so this is the one file where we can properly and in a supported way put metadata

i propose adding support asap and deprecating the other config files for the 3.x series so we can possibly remove support starting with the 4.x series

[The-Compiler]

Big -1 on deprecating pytest.ini if you also suggest that.

My config files are pretty big, which is why I want to keep them in their own file rather than stuffing everything into one - line count:

   18 .coveragerc
   59 .flake8
   13 .pydocstylerc
   69 .pylintrc
  245 tox.ini
   41 pytest.ini

[RonnyPfannschmidt]

i see, good point!

i change my proposal to deprecating all files other than pytest.ini and pyproject.toml

[hynek]

Any plans on moving this forward? 😇

[RonnyPfannschmidt]

@hynek yes, but no time/volunteer

[tadeoos]

Hey guys, I gave it a go in #3686. It adds support for pyproject.toml but doesn't deprecate anything.

[nicoddemus]

Any ideas on how we the file would look like? #3686 was an attempt made by @tadeoos, but @RonnyPfannschmidt mentioned that we should take advantage of the additional layers offered by TOML.

I suggest we focus on the file format first, then after a consensus move to the API to get to that file format.

My quick take on this is that we don't need many sections other than separating built-in options from those defined by plugins. For brevity I think built-options could go to a root tool.pytest table, and plugin options in sub-tables named after the plugin.

Quick example:

[tool.pytest]
junit_family = "xunit2"
doctest_encoding = "UTF-8"

  [tool.pytest.qt]
  qt_api = "qt5"

  [tool.pytest.timeout]
  timeout = 10

  [tool.pytest.xprocess]
  timeout = 10

This avoids ini-options from different plugins to clashing.

[RonnyPfannschmidt]

that would intermix pytest and plugin tables in the most unforetunate manner,

tool.pytest.x nests inside tool.pytest

plugins should use tool.pytest_x, matching the distribution name with dashes replaced by underscore perhaps

the real problem isnt the new config file, the real problem is layering, integration of the old config options and transition of the apis

[nicoddemus]

that would intermix pytest and plugin tables in the most unforetunate manner,

Why would that be a problem?

The real problem isnt the new config file, the real problem is layering, integration of the old config options and transition of the apis

Oh I know, but I thought the discussion should start from the file format first. I'm well aware that the hard work will be the integration and backward compatibility. 😓

[njsmith]

Right, @nicodemus's example is equivalent to

[tool.pytest]
junit_family = "xunit2"
doctest_encoding = "UTF-8"
qt = { qt_api = "qt5" }
timeout = { timeout = 10 }
xprocess = { timeout = 10 }

If you wanted to avoid collisions you could use [tool.pytest.plugin.pytest-whatever], but that seems like it might be excessive, given that [tool.pytest-whatever] is already available.

with dashes replaced by underscore perhaps

Dashes are fine in toml keys, so if packages want to use [tool.pytest-whatever] then that's fine.

[RonnyPfannschmidt]

@nicoddemus i believe that without the integration strategy, we will end up with a massive mess

right now config initialization is pretty much fundamentally broken to begin with

[nicoddemus]

For illustration:

[tool.pytest]
junit_family = "xunit2"
doctest_encoding = "UTF-8"

  [tool.pytest-qt]
  qt_api = "qt5"

  [tool.pytest-timeout]
  timeout = 10

  [tool.pytest-xprocess]
  timeout = 10

i believe that without the integration strategy, we will end up with a massive mess

"massive mess" on the file format or the code to support the file format in the end? We can always can come back to the drawing board if we can't find a clean way to implement the desired file format.

Or do you want to change the discussion from the file format over to the API/integration?

[RonnyPfannschmidt]

the format is easy, the integration/semantics are __hard__

[marco-silva0000]

I would also like to see this feature added.

[nicoddemus]

What does the approach taken by tox to have a legacy session inside the toml file where it contains a plain string with the current pytest.ini contents? This would at least allow people to move their configuration there, while we try to brainstorm how to move this forward.

[RonnyPfannschmidt]

That one would be good, we could also support a mapping of options to strings or lists as the legacy value to take advantage of the file format for Syntax, which may be a help to the desired semantics

[nicoddemus]

we could also support a mapping of options to strings or lists as the legacy value to take advantage of the file format for Syntax, which may be a help to the desired semantics

Interesting, care to elaborate, perhaps with examples?

[RonnyPfannschmidt]

Format strings come to mind, and native lists

[RonnyPfannschmidt]

I won't make examples as I'm currently afk

[RonnyPfannschmidt]

[tool.pytest.LEGACY]
addopts = ["-p", "test"]

or

[tool.pytest]
LEGACY = """
addopts = -p test
"""

if a multiline string is used, we should allow a missing [pytest]
if a mapping is used, the type of each value should be enforced to exactly match whats used/declared (to avoid having to be unreasonably smart about the transformations in a format thats only supposed to help with transitions

(perhaps we should not support the mapping form to begin with)

[NargiT]

will your example be able to support an hybride version of LEGACY flags and new one?
Or we should migrate everything in order to remove LEGACY completely ?

[RonnyPfannschmidt]

@NargiT personally, i'd aim for big bang changes there

[NargiT]

Big bang is good for me as long as the refactoring is easy enough to not waste to much time

[nicoddemus]

I personally like

[tool.pytest.LEGACY]
addopts = ["-p", "test"]

Because then we have the more explicit data types to use.

---

Neverthless, we should discuss the pain points, which I think is mainly that the API to declare/get options don't have any extra context other than the option name:

• parser.addini(name, help, ...)
• config.getini(name, default)

This makes it hard for those methods to know if who is asking for the option's value is pytest-qt or pytest-timeout (for example).

Any ideas to overcome this?

I have some, none of which I really like:

1. Create proxy objects to config which are aware of the context where they will be used, and pass that along to the hooks which give access to the config object, instead of the actual config object.

   Pros:

   • Configuration parity: it is possible for a plugin to support any format (ini or toml) without changes.
   • All existing plugins immediately support toml, without nameclashes.

   Cons:

   • It would require some nasty changes in pluggy for that happen.
   • While the context is obvious for third party plugins (their entry point name), that's not true for conftest.py files.

2. Introduce an optional parameter to config.addini and config.getini, context: it means the context in which the option will reside in configuration files. For example:

   parser.addini('qt_api', ..., context='pytest-qt')
   parser.getini('qt_api', default='qt5', context='pytest-qt')
   

   Then this would translate to:

   [pytest]
   pytest-qt.qt_api = "qt5"
   

   Or:

   [tool.pytest]
   
      [tool.pytest-qt] 
      qt_api = "qt5"
   

   context=None means a flat option, exactly like today. Also, if addini receives a context, every getini call would also need one.

   Pros:

   • Compatible, opt-in API
   • Plugins can start using flatter options without fear of nameclashes; in the example above, that option could be api instead of qt_api now (of course there's backward compatibility concerns for the plugin).

   Cons:

   • getini is longer to type, now we need to include context in all calls.

   • Longer option names to type in the config files

     • This can be alleviated by an extra parameter to addini, which would issue a deprecation warning when the "flat" option is used, and indicate the new option automatically:

     parser.addini('api', ..., context='pytest-qt', legacy='qt_api')
     

     Would emit a PytestDeprecationWarning:

     ini option 'qt_api' has been deprecated, use 'pytest-qt.api' from now on.
     
     

   
   

As I said, I don't like any of the above much, but no other ideas come to mind.

---

Having said all that, what are the problems with the current .ini format? Here's what I gather:

1. No proper unicode support.
2. Has some weird escaping rules.
3. Only supports strings and list of strings as values.
4. Community is migrating to pyproject.toml.
5. Flat option names.

toml would solve the first 4 options nicely and naturally.

The 5th option TBH doesn't seem to be a huge problem in practice: pytest has existed for many years now (eleven?), and plugins have found a way to not step on each other's toes.

If these are the problems we are trying to solve, then perhaps just adopting flat options in the toml format be enough for our needs?

toml [tool.pytest] addopts = ["-p", "test"] qt_api = "qt5"

This has the following real world advantages:

• Trivial to implement
• No API/integration concerns
• Full backward compatibility
• Trivial for users to understand/migrate.

To further mitigate the possibility of plugins nameclashing option names, we could explicitly recomment that they add some unique prefix to the options, like qt_api has used since its inception, and similar to the same recommendation we have for keys in the cache plugin.

[RonnyPfannschmidt]

im absolutely certain that the cost would be a horrific pain if we simply mulled the addini api to support toml
ini and tomml have different value type models, and i really would prefer declaring the real options in the new system and then declaring how a legacy ini option translates to the new system for a transition period

[nicoddemus]

ini and tomml have different value type models, and i really would prefer declaring the real options in the new system and then declaring how a legacy ini option translates to the new system for a transition period

You mean the addini's type parameter, which supports "pathlist, args, linelist or bool"?

Can you please provide a better description of what are your concerns, and you would like to see (preferably with examples)? I'm sorry, but your description is quite vague to me, so I'm left with guessing.

[RonnyPfannschmidt]

I won't be able to make a good mockup until next week, I'll put it in a todo

[nicoddemus]

Great, thanks!

[nicoddemus]

@RonnyPfannschmidt gentle ping. 😁

[RonnyPfannschmidt]

Thanks for the reminder, unfortunately I am on mobile and about to go to a vocation

[RonnyPfannschmidt]

i wanted to start with the loggin plugin, but this immeadely made me want to burn the config systme with dire, i'll strill come up with a nomerge pr for the tooling example

[nicoddemus]

I'm a bit concerned about the direction the overall discussion is going (not #5882 in particular per see, I definitely appreciate taking the time to write it up).

My impression is that this would introduce a real complexity to pytest's configuration which I don't think neither users nor plugins need.

It would also mean that users will need to understand the intricacies of the different systems, and plugin authors would have to support both ways to declare options for many years to come. Worse, users wanting to migrate to pyproject.toml would then be at mercy of the authors of the plugins they use (until the plugins support the new way), or fallback to have half the configuration in pytest.ini, the other half in pyproject.toml.

I understand this is far from a simple issue (as other projects are also in the same state), but I would like to further discuss the points I brought up in https://github.com/pytest-dev/pytest/issues/1556#issuecomment-528659962 (which I took a good time to write, would appreciate some thoughtful/elaborate replies to it).

More explicitly, I would like thoughtful/elaborate answers to:

1. What are the problems with the existing config.addini API?
2. What is the problem which we would be solving by adding support for pyproject.toml support in pytest?
3. Which features TOML has which setup.cfg doesn't, but we would like to have in pytest configuration?

I fear the discussion is stalled because it is not clear (at least to me) what problems we are trying to solve, which are the existing pain points, etc. 😬

[RonnyPfannschmidt]

thanks for calling out the lack of a clear vision, i#ll let this linger in the back of my head a while longer to come up with good answers to the points

[feluxe]

1. What is the problem which we would be solving by adding support for pyproject.toml support in pytest?

Project directories these days are stuffed with config files of various formats/conventions: requirements.txt, mypy.ini, Pipfile, .style.yapf, tox.ini, pytest.ini, etc... . These add a lot of noise to the project directories and quite some mental overhead when you work on them. This is not a big deal if you maintain a few projects, but it's taxing if you work on many projects simultaneously. To me it would be a great relief to combine them in pyproject.toml and use the same config convention for every tool.

In the long run the whole Python ecosystem will benefit from a standardized config system and I guess the toml proposal is the first one which is powerful enough to succeed. I'm cheering for every major Python tool to jump on the PEP-518 wagon. ;-)

[RonnyPfannschmidt]

That's a lovely idea, but we have to give ourselves an user lens so the feature removes user problems instead of shifting or piling them

if we introduce toml support, then I'd like to unify configuration and configuration sources in a way that makes usage better

until we actually have a sensible idea for that transition its preferable to keep the current system or just enable pytest. ini content in a legacy multiline item

[canassa]

I am following this issue for some time so I would like to give my input on this.

I am currently maintaining several Python/Django project and I try to use the same skeleton for all of them. These projects are using several tools:

• mypy
• black
• coverage
• pytest
• isort
• flake8
• pre-commit

Each tool has it's own configuration system, some used it's own format (coverage) while others used setup.cfg. Black was actually the first one that forced me to create a pyproject.toml file and what I noticed is that over 2018-2019 several of the other projects are implementing pyproject.toml support:

• mypy open
• isort implemented
• coverage implemented
• flake8 open
• pre-commit wont fix

So far, pre-commit is the only one that didn't jump into the pyproject.toml bandwagon but giving that pre-commit is a language-agnostic tool I think that decision makes sense. But as you can see, there is a clear trend in the Python community to support the pyproject.toml format.

Having a unified configuration format/file (and backed by a PEP nonetheless) is beneficial for the Python community and given that some projects are not supporting setup.cfg at all (Black) the pyproject.toml seems to be way to go.

[nicoddemus]

Thanks everyone so far.

I think we can all agree that moving to pyproject.toml to reduce the number of files is a good idea, so I believe this out of the table for now.

[nicoddemus]

@canassa

Thanks for the links, I was wondering which projects have made the switch and how they did it.

It seems isort just moved their flat options over to pyproject.toml.

For coverage I couldn't find the PR or the docs (but I didn't search much actually).

[nicoddemus]

@RonnyPfannschmidt

if we introduce toml support, then I'd like to unify configuration and configuration sources...

What do you mean "configuration sources", the various files supported currently?

... in a way that makes usage better

Usage better for who? Users or plugin authors?

If possible it would be nice to enumerate the current problems (both in API and configuration), rather than try to come up with a solution, because it still not crystal clear what the current problems are to me.

From my previous comment, I've listed this as the current problems:

1. No proper unicode support.
2. Has some weird escaping rules.
3. Only supports strings and list of strings as values.
4. Flat option names.

What else can you add to this list?

You also mention that the current configuration API is a mess... can you provide a clear list of those problems?

The community is pushing to have a pyproject.toml in the same format as a pytest.ini, which they think is reasonable.

@RonnyPfannschmidt seems to think this is problematic, but why this is problematic doesn't seem to be clear yet.

If we are going to stall this, it would be good to list the exactl reasons why this is stalled, so we can point people to the reasons when they wonder why pytest doesn't support pyproject.toml yet.

[hugovk]

I was wondering which projects have made the switch and how they did it.

https://github.com/flying-sheep/awesome-python-packaging lists projects which support it.

[nicoddemus]

Thanks @hugovk!

[merwok]

Just to clarify: PEP 518 is about build isolation for packaging tools, and incidentally defines the pyproject.toml file. It is not a spec for all Python dev tools to use the file (but mentions that they can do it — PEP authors expected packaging-related tools to do so, not necessarily all dev tools). Having a tool’s option in pyproject.toml is an emerging community practice, but it’s not accurate to say it’s backed by a PEP.

[RonnyPfannschmidt]

@RonnyPfannschmidt

if we introduce toml support, then I'd like to unify configuration and configuration sources...

What do you mean "configuration sources", the various files supported currently?

currently we have cli options and ini entries as configuration sources,
those have distinct type systems with distinct capabilities

in many plug-ins and pytest core we try to mix and coerce those manually

... in a way that makes usage better

if we add toml in a sane way, it will need yet another type system, suddenly everyone would have to juggle 3 systems, when 2 of them are already quite a mess

Usage better for who? Users or plugin authors?

"both" this one is tricky

If possible it would be nice to enumerate _the current problems_ (both in API and configuration), rather than try to come up with a solution, because it still not crystal clear what the current problems are to me.

a reasonably exhaustive listing will need a while

From my previous comment, I've listed this as the current problems:

1. No proper unicode support.

2. Has some weird escaping rules.

3. Only supports strings and list of strings as values.

4. Flat option names.

What else can you add to this list?
see above for a start

You also mention that the current configuration API is a mess... can you provide a clear list of those problems?
dito

The community is pushing to have a pyproject.toml in the same format as a pytest.ini, which they think is reasonable.

@RonnyPfannschmidt seems to think this is problematic, but why this is problematic doesn't seem to be clear yet.

the key issues i currently consider a problem are name-spacing and conflicts,
those are indeed not things you see day2day, but when you see them, you certainly wish you didnt

If we are going to stall this, it would be good to list the exactl reasons why this is stalled, so we can point people to the reasons when they wonder why pytest doesn't support pyproject.toml yet.
its going to take time to to make the "exact"

but i believe its fair to say that we currently have no idea how to do it good, and we don't want to regret doing it really horrible unfixable

[nicoddemus]

Thanks @RonnyPfannschmidt

currently we have cli options and ini entries as configuration sources,
those have distinct type systems with distinct capabilities

I agree completely, but I believe this problem is orthogonal to pyproject.toml support, is it not?

I mean, I believe the question in most people's mind thinking about this (me included) is what's the problem with adding pyproject.toml as another configuration file format, with same capabilities as pytest.ini today. By itself this would solve a number of problems already (encoding, weird escaping rules, sane syntax, etc).

if we add toml in a sane way, it will need yet another type system, suddenly everyone would have to juggle 3 systems, when 2 of them are already quite a mess

But why would the API need another type system?

Let me elaborate:

Currently we support a few data types in configuration files:

• str
• int
• bool
• list of str
• choices (list of str)

pyproject.toml would work just as well for those.

TOML has more data types, but it seems to me adding support for those later wouldn't change the support we have for the existing ones.

My feeling is that we could add support for TOML, restricting for the same types we currently support for pytest.ini and other configuration files, without changing much of the (bad) APIs we have today.

Later, we can always improve the API (by deprecations/refactorings etc).

a reasonably exhaustive listing will need a while

No need to be exhaustive, but it would be nice to get a list started. 😁

the key issues i currently consider a problem are name-spacing and conflicts,
those are indeed not things you see day2day, but when you see them, you certainly wish you didnt

Indeed, and name-spacing/conflicts is a problem, but to solve that we have to either:

1. Break the existing API so option names can have some name-spacing: definitely bad, would break the world.
2. Create a new one: less than ideal/might be useless, because then plugin authors won't use because it because users are not using, because not all plugins use it so users stick to pytest.ini, so plugins are less encourarged to support it, and so on.
3. Keep the existing one, perhaps enhancing it later in ways that don't break backward compatibility.

For an examples of the later:

# existing API
parser.addini("junit_suite_name", ...)

# implicit context using strings:
parser.addini("junit.suite_name", ...)  # this becomes an option "suite_name" under "junit" table, which its a child of [tool.pytest]

# explicit context using strings:
parser.addini(Context("junit", "suite_name"), ...)   # same as above

(To be clear: just trying to adress the name-spacing problem here)

All the above have the advantage of being backward compatible.

but i believe its fair to say that we currently have no idea how to do it good, and we don't want to regret doing it really horrible unfixable

Sure, I agree with that, but my point in the previous post is explaining the problems and get them clearly defined, not getting to the solution.

[RonnyPfannschmidt]

i had a few nights of good sleep, and figured that we can have a good start with just
having something like

[tool.pytest.ini_options]
some_name = "coercible string or coercible toml type"

with clear documentation that this subkey is to enable people to use the single config file even tho pytest itself didn't design a proper type system/integration story for first class toml support

currently i'd like to avoid/defer designing a proper toml type integration - and i certainly want to avoid letting the ini concepts bleed into it

[nicoddemus]

Thanks @RonnyPfannschmidt,

That looks reasonable, but I assume you understand that this:

[tool.pytest]
some_name = "coercible string or coercible toml type"

Might be problematic in the future, correct? Could you elaborate what are your concerns?

Perhaps (and I'm guessing here) you see something like the below might be a possibility in the future?

[tool.pytest.core]
addopts = "-x"

[tool.pytest.timeout]
timeout = 10

[tool.pytest.qt]
api = "pyside2"

[RonnyPfannschmidt]

@nicoddemus that's why i choose the ini_options subkey, that subkey will be reserved by pytest core to map ini options until the real story is happening

we should bike-shed the key naming a bit, and then take a look at making it work for the legacy transfer

if we find a mistake in that when sorting out the details about the real story we will handle it at that point, i wont speculate atm

[nicoddemus]

I see, thanks. This approach sounds reasonable, and has the advantage to be trivial to understand and port.

I'm OK with [tool.pytest.ini_options]. I think all options from the current config map directly to builtin TOML types, correct?

[RonnyPfannschmidt]

as far as i can tell yes, we need to take a look if we want to support coercion or just expect some minimal transforms on the user

im inclined to opt for starting without built-in coercion and just expecting users to do the transforms (but also leaving it open to the community to provide a pr with coercion)

[nicoddemus]

Currently the signature for addini is:

    def addini(self, name, help, type=None, default=None):
        """ 
        ...
        :type: type of the variable, can be ``pathlist``, ``args``, ``linelist``
               or ``bool``.

        ...
        """

So we should not do any coercion, in other words, the user calling config.getini shouldn't care if the config came from pytest.ini or pyproject.toml, it should return the exact same value and type.

Plugin authors make the coercion themselves, for example:

https://github.com/pytest-dev/pytest-timeout/blob/35668ab455df0155f8c8b24c28f1774617dab2f2/pytest_timeout.py#L187-L190

    if timeout is None:
        ini = config.getini("timeout")
        if ini:
            timeout = _validate_timeout(ini, "config file")

So the [tool.pytest.ini_options] should in fact convert some types to str for compatibility (which makes the ini_options section even more of a good idea).

The example above also highlights the differences between the config and command-line API (which does type-coercion), as you mentioned before.

[nicoddemus]

Now that we have a clearer way forward, I'm tentatively putting this on the roadmap for 5.5. 😁

[chadgh]

Here is the awesome-pyproject list. It would be helpful if and when pytest supports pyproject.toml to notify the list maintainer or submitting a PR to update the list indicating that pytest supports the file now. Just FYI. :)

[ofek]

Have we considered a plugin namespace to clean things up?

[tool.pytest]
addopts = "-x"

[tool.pytest.plugins.timeout]
timeout = 10

[tool.pytest.plugins.qt]
api = "pyside2"

[RonnyPfannschmidt]

@ofek we have not considered any completely new structure, as we yet have to figure how we want to unify arguments from clie, options from ini files and data from well structured toml data

we also haven't gotten't around investigating how it would actually be used - and i haven't found time to do more on my proposed api

so for now we are just going to support a legacy ini key where ini options can be put instead of the other files, better integration will come later with time or working external proposals