Pipenv: Managing abstract dependencies of setup.py

[guyzmo]

You say pipenv is a marriage of Pipfile, Pip, & Virtualenv. But there's one depency related task that's left to be done manually, that's managing setup.py's install_requires.

As @dstufft wisely says in his article, abstract depencies and concrete depencies are used in different contexts. But when you do your code, you're managing them at the same time. If you change a depency, you'll have to modify both setup.py or the Pipfile. I believe, it'd be nice to make it possible to manage the abstract dependencies using pipenv.

From a pipenv user perspective, I mean something that would look like:

# in Pipfile → [dev-packages]
pipenv install --dev pytest mock delegator.py toml Sphinx
# in Pipfile → [packages]
pipenv install pew>=0.1.26
# in setup.py → install_requires
pipenv install --abstract virtualenv pip pew>=0.1.26

Maybe could we extend the syntax so it has three options:

# in Pipfile → [dev-packages]
pipenv install --dev pytest mock delegator.py toml Sphinx
# in Pipfile → [packages] (only)
pipenv install --concrete foobar
# in Pipfile → [packages] and setup.py → install_requires
pipenv install pew>=0.1.26
# in setup.py → install_requires (only)
pipenv install --abstract virtualenv pip 

That way, pipenv is the entry point tool to manipulate everything related to dependency management and associated virtualenv. And from a trouple marriage, it'll be a marriage of four tools 😁.

Now, on the question on how to implement this, I'm not much in favour of automatic edition of a .py file, because there are just too many ways it can go wrong. But instead, store the abstract depencies externally in a new [abstract-packages] section of Pipfile if it's ok with @pypa, or in an abstract-requirements.txt file or any other smarter solution I haven't think of yet.

What do you think?

[nateprewitt]

Hey @guyzmo, thanks for starting this discussion. This is definitely a decision that needs to be made by the Pipfile project itself. I'd suggest proposing this there and seeing what the response is from the pypa community. From a historical perspective, there's been pushback from Pipfile adopting any more *-packages sections at this time.

Feel free to reference this issue if you'd like.

[guyzmo]

well, the issue I'm posting here has 3 questions:

1. do you think pipenv would be the right tool to manage abstract dependencies?
2. what command line API would it be nice to expose?
3. how to implement the mutability of those abstract dependencies?
   
   
   
   • in Pipfile, by negociating with pypa?
   
   
   • in some file we parse with something looking like the following in setup.py?
     
     with open('abstract-req') as r: install_requires=r.read().splitlines()
   
   
   • by modifying the setup.py file, like it's a config file (it wouldn't be hard to parse it using ast, find a given variable and mutate the RHS of that variable assignation).
   
   

So, to me the first two questions are the most important ones we should discuss here, and on which a decision shall be made. Only then, we should discuss the third question, which is, IMHO, a technical detail. If people of the Pipfile project decide it's nice to also manage abstract dependencies in Pipfile, it's great, if they don't, then we can figure out another way.

I did not say, but I'd be happy to write a patch to implement whatever gets decided in this discussion, and ask the Pipfile project for permission, if contributors of this project share my opinion that it'd be a great adition to pipenv, and help decide what's the best approach for implementing the abstract dependencies in a way that's maintainable by a tool like pipenv.

[kennethreitz42]

I think Pipfile should leave setup.py alone completely, personally.

[guyzmo]

@kennethreitz and what do you think about pipenv managing the install_requires (independently of the implementation)?

[guyzmo]

After doing some research, it seems the right place to manage that would be indeed neither in the Pipfile or the setup.py, but rather in the setup.cfg, which supports it since setuptools 30.3.0.

[nateprewitt]

Thanks for investigating this @guyzmo! From my understanding of what you're asking, I would say setup.cfg is probably what you're looking for here. Let us know if you have any further thoughts here.

[guyzmo]

so if I understand this correctly, you're not interested to extend pipenv so it manages abstract depencies (using setup.cfg as backend) with the following syntax:

# in Pipfile → [dev-packages]
pipenv install --dev pytest mock delegator.py toml Sphinx
# in Pipfile → [packages] (only)
pipenv install --concrete foobar
# in Pipfile → [packages] and setup.cfg → install_requires
pipenv install pew>=0.1.26
# in setup.cfg → install_requires (only)
pipenv install --abstract virtualenv pip 

and if I want to see this happen, you're not definitely not interested in such a patch, so I should either write yet another tool or hard fork pipenv (which is basically the same)?

[pcattori]

@guyzmo i found myself making another tool to manage dependencies for setup.py. and actually i am just doing away with setup.py altogether and replacing it with a static file (json for now but will use toml later). its basically "what would yarn for python be like?". (disclaimer: its a pet project so far, but would love to know where to contribute to something similar)

curious to see if it addresses your use-case at all or what you have learned about this stuff so far

[guyzmo]

@pcattori sounds promising, though are you adressing the issue of abstract vs concrete deps management? Also, have you considered using the standard files (setup.cfg, Pipfile…) instead of adding yet another one?

[pcattori]

@guyzmo yes i am planning on handling abstract vs concrete dependencies. specifically, the add command will update the entry in the project's json config (which is doing the job that setup.py was doing) and though I haven't added all the functionality yet, it will allow users to specify abstract vs concrete dependencies as necessary and will keep concrete deps in a .lock file (ala Pipfile).

I have briefly considered using existing standard files like Pipfile and the new pyproject.toml but not really sure what to use. It seems like Pipfile is the replacement for requirements.txt, not setup.py. I think pyproject.toml might be what I want, but for short-term prototyping I haven't devoted the research to answer this question and instead just using something I'm familiar with (json) and something I have complete control over (my own format for that json file). Now that I've got basic functionality in, I'm trying to answer this question.

haven't explored setup.cfg much yet, which is why I asked about anything you've learned on the matter. sounds like maybe setup.cfg is another candidate to replace my custom snek.json