Dependabot-core: Add support for Python dependencies in `setup.cfg`

[althonos]

Hi !

Ever since setuptools started supporting setup configuration files, I've been using them instead of hardcoding configuration values in the setup.py script. However, it does not seem like these files are supported by Dependabot right now.

Adding support would be fairly simple: check for setup.cfg, parse it as an INI configuration file, and check the value in setup_requires, tests_require, and install_requires just like a requirements.txt file.

I've been using Dependabot extensively for my Rust projects but this is stopping me from extending that to my Python projects as well. I'd love to see setup.cfg support implemented!

[greysteil]

Oh good point. We have a kind of partial support for them, where we can handle libraries that use them alongside a requirement.txt, but don't parse them or update them. We should.

[greysteil]

@althonos can you link to a repo that uses this setup? Would help a lot with adding support.

[althonos]

@greysteil : sure ! althonos/InstaLooter has that kind of structure, with every dependency listed in setup.cfg.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs within seven days. Thank you for your contributions.

[althonos]

@greysteil : bumping ? :smile:

[greysteil]

I'm not on team Dependabot anymore (I now work on security more broadly at GitHub). @hmarr, @feelepxyz and @jeffmcaffer are your go-tos now :-)

[ivelin]

@feelepxyz and @jeffmcaffer , could you please share an update on this issue.
setup.cfg is becoming popular in the python packaging community. Here is another example project that uses setup.cfg alongside setup.py for dependency management.

Thank you!

[feelepxyz]

@ivelin 👋 no progress on this yet. We're currently running a very small team maintaining dependabot alongside integrating it within GitHub so have made little progress on new features. We've got a few python improvements lined up once we get more people on the team.

[ivelin]

Thank you for the candid response, @feelepxyz . I understand and will keep an eye on the issue for updates.

[ivelin]

BTW, not sure if its an option, but if the Python setup parsing code in dependabot is open source, maybe you can provide a couple of hints and pointers. There may be contributors in the Python community with spare cycles to help out.

[feelepxyz]

@ivelin sorry for the slow update! Would love help adding support to dependabot-core. As a starting point, you probably want to hook in here and branch out to a new updater for setup.cfg: https://github.com/dependabot/dependabot-core/blob/master/python/lib/dependabot/python/file_updater.rb#L27-L43

You can use the requirements updater as a starting point: https://github.com/dependabot/dependabot-core/blob/master/python/lib/dependabot/python/file_updater/requirement_file_updater.rb

[mauvilsa]

Hi. Is there any update on plans for working on this? I see that the links that @ivelin posted as a starter for anyone willing to help are now broken. The new ones are:

• https://github.com/dependabot/dependabot-core/blob/main/python/lib/dependabot/python/file_updater.rb#L27-L43
• https://github.com/dependabot/dependabot-core/blob/main/python/lib/dependabot/python/file_updater/requirement_file_updater.rb

I also add a couple of example projects that use setup.cfg for requirements which include both install_requires and extras_require.

• https://github.com/omni-us/jsonargparse/blob/master/setup.cfg
• https://github.com/omni-us/narchi/blob/master/setup.cfg

I was considering helping out. But I don't know ruby so looking at the starting point it did not seem too easy for me to do. I also found https://github.com/datafolklabs/ruby-parseconfig which maybe is able to parse the setup.cfg files.

[akaihola]

This feature is also discussed here:
https://github.community/t/dependency-graph-does-not-support-setup-cfg-for-python/2576

PR #2281 modifies the parse_setup Python helper to use

distutils.core.run_setup(
    <path to unmodified setup.py file>,
    stop_after="init"
)

and the results are collected using the custom setup() function as before.

It also catches all exceptions. In case of errors or no collected dependencies, it reverts to the old method running a patched setup.py using exec() method.

I haven't yet added test cases for this. I'll need to install the Ruby environment and study how to run the test suite – pointers to documentation appreciated!

[graingert]

it would be better to use pep517 https://www.python.org/dev/peps/pep-0517/#prepare-metadata-for-build-wheel

[graingert]

here's a demo using the pep517.meta helper:

>>> import pep517.meta
>>> import pprint
>>> pprint.pprint(pep517.meta.load("path/to/twisted").requires)
['zope.interface (>=4.4.2)',
 'constantly (>=15.1)',
 'incremental (>=16.10.1)',
 'Automat (>=0.8.0)',
 'hyperlink (>=17.1.1)',
 'attrs (>=19.2.0)',
 'twisted-iocpsupport (~=1.0.0) ; platform_system == "Windows"',
...

[akaihola]

@graingert, have you taken a look at the other ideas thrown around in #2281? Could you add this one in that discussion?