Pip-tools: Circular dependency between Sphinx and local Jinja development

[davidism]

Jinja lists Sphinx in requirements/docs.in. But Sphinx depends on Jinja, so jinja2==2.11.1 gets added to requirements/docs.txt (and requirements/dev.txt through -r docs.in). But this pin overwrites the locally installed version of Jinja, which is what should be documented and tested.

In most Pallets projects, I could make the instruction pip install -e . -r requirements/dev.txt to install both the dev requirements, and Jinja in editable mode. This doesn't seem to work for Jinja though, the pin in -r always overrides -e on the command line, no matter what order they're given in. This can be worked around in most places by doing pip install -e . as a separate second step. But this doesn't work in Tox, which installs the local code before the dependencies. And I'm not sure if it works in Read The Docs config.

Adding -e file:. to the requirements file did cause the editable install to take precedence. But that's still problematic for Tox, which manages building and installing the local code separately for test isolation purposes. Also, Dependabot doesn't understand the -e requirement and fails to scan the project (dependabot/feedback#936).

--exclude jinja2 (#333) might help with this, but I'm not sure. It probably wouldn't be understood by Dependabot, so I'd still miss that convenience. It might be that pip-compile is just not going to work for this type of circular dependency with the project under development. I'm willing to accept that answer and figure out another way to organize dependencies to avoid this, but I wanted to at least bring it to your attention to see if there were any suggestions.

[atugushev]

Hello @davidism,

Thank you for the report. An Interesting case! I wonder would tox --devenv work?

$ git clone [email protected]:atugushev/jinja.git

$ cd jinja

$ tox -e docs --devenv venv
docs create: /Users/albert/Projects/jinja/venv
docs installdeps: -rrequirements/docs.txt
docs develop-inst: /Users/albert/Projects/jinja
docs installed: alabaster==0.7.12,Babel==2.8.0,certifi==2020.4.5.1,chardet==3.0.4,docutils==0.16,idna==2.9,imagesize==1.2.0,-e [email protected]:atugushev/jinja.git@ef69935b37a970ecd8f91f227c8bcfd19b7b4931#egg=Jinja2,MarkupSafe==1.1.1,packaging==20.3,Pallets-Sphinx-Themes==1.2.3,Pygments==2.6.1,pyparsing==2.4.7,pytz==2020.1,requests==2.23.0,six==1.14.0,snowballstemmer==2.0.0,Sphinx==2.4.4,sphinx-issues==1.2.0,sphinxcontrib-applehelp==1.0.2,sphinxcontrib-devhelp==1.0.2,sphinxcontrib-htmlhelp==1.0.3,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-log-cabinet==1.0.1,sphinxcontrib-qthelp==1.0.3,sphinxcontrib-serializinghtml==1.1.4,urllib3==1.25.9
______________________________ summary ______________________________
  docs: skipped tests
  congratulations :)

$ source venv/bin/activate

$ pip list | grep jinja
Jinja2                        3.0.0a1    /Users/albert/Projects/jinja/src

Looks like it works. Recently I switched from make venv && pip install -e . to tox --devenv for projects which use tox. What do you think?

[atugushev]

Huge thanks to @asottile who brought this feature to tox 🎉 🎉 🎉

[davidism]

While this is a nice feature of Tox, it doesn't really solve the issue. I can't use Tox to create the env on Read The Docs, and don't want to mandate its use to set up a dev environment (and it creates another circular issue in that Tox itself is specified in dev.in).

I'm looking for a solution in pip-compile and the requirements files. I feel like this issue would come up in any situation where two projects depend on each other, regardless of what other tools the projects use.

[davidism]

It does look like Tox installs the project after the dependencies, so running tox -e docs does test the right thing. Fairly sure .readthedocs.yaml installs things in order as well, so it's mostly an issue of allowing contributors to create a dev environment with only venv and pip.

Maybe I should report this to pip instead, it seems like -e . should always take precedence over pins from -r.

[davidism]

Reported to pip as well: pypa/pip#8307

[atugushev]

In most Pallets projects, I could make the instruction pip install -e . -r requirements/dev.txt

IIUC pip install -r requirements/dev.txt should be enough to install jinja in editable mode (since -e file:. is listed in dev.txt) and dev dependencies.

Also, Dependabot doesn't understand the -e requirement and fails to scan the project (dependabot/feedback#936).

Did https://github.com/pallets/jinja/pull/1216 fix the issue with Dependabot?

[davidism]

No, nothing I tried fixed it, I reported it to Dependabot https://github.com/dependabot/feedback/issues/936

[davidism]

This is also causing issues in Click, because pip-tools depends on Click, but pip-tools is a dev requirement for Click.