Setuptools: CI should fail on deprecation warnings

[pganssle]

It's hard for us to detect when we are using deprecated code, which allows things like #1798 to go undetected for a long time. In dateutil I treat all warnings as errors so that we can actually heed the warnings that are being raised by my upstream dependencies.

I think we should do the same thing here. I tried enabling warnings-as-errors on #1798 and it actually worked for a few days, then bdist_wininst was deprecated breaking us on the dev build of CPython. I think we should remove our use of bdist_wininst or suppress the warning then enable warnings-as-errors.

[jaraco]

I basically agree with all of that. Happy to enforce no warnings. Even better is to have a clear guidance on how to suppress errors/warnings when the emerge and to file a corresponding ticket to address the deprecation. Perhaps that guidance could go here.

I imagine something like:

When encountering an error from a Warning:

1. File a ticket including the text from the failure.
2. Suppress the warning by adding an entry to pytest.ini, including a link to the ticket.

Can we test a process like that with bdist_wininst?

[pganssle]

@jaraco How granular can we make the specifications in pytest.ini? I've only ever filtered by warning class, which would mean disabling the warnings for all of DeprecationWarning in this case, which would be undesirable. I think this would be a good test to demonstrate that the pytest.ini approach would work.

[jaraco]

Pytest supports the same syntax as the the -W argument to Python, which allows for arbitrary granularity, including module and line number. For example, here are some warnings I suppress as a matter of course.

I think the filter we want is ignore:distutils.command.bdist_wininst::command is deprecated (specific and pretty clear). It would be sensitive if the message changed not to include that string, but that seems unlikely.

[jaraco]

With 9fd5451, the tests now run on Python 3.8 with tox -- -W error.

[jaraco]

With 9fd5451, the tests now run on Python 3.8 with tox -- -W error.

I was mistaken. That syntax is wrong. And in fact, I've learned it's not possible to reference a warning by the module in which it's defined if stacklevel != 1 in the warning. (the module of the warning is distutils.dist and not distutils.command.bdist_wininst due to stacklevel=2)

[jaraco]

Enabling this feature is going to cause a lot of toil on this project. Every time a deprecation emerges, it's going to be blocked by this feature. Case in point #2016. That's not too bad because it means that a ticket can be filed and the error can be suppressed until the issue can be addressed (if not addressed immediately), but it's going to require additional resources to keep up with these issues in a timely manner to prevent them from blocking otherwise good pull requests.