Pip-tools: pip-sync yields different results depending on pre-installed package set

[AndydeCleyre]

Update: The shape of this issue has changed a bit since creation, due to discoveries in this discussion, and the addition of the --pip-args option. I'm renaming it, and adding a new summary:

---

If pip-sync has to actively install a requirement, its dependencies will end up in the environment. But if the requirement is already installed when pip-sync runs, its dependencies not present in the txt will be removed, "breaking" the environment.

$ echo "pytest==5.1.2" > requirements.txt
$ pip-sync /dev/null
$ pip-sync requirements.txt
$ pip freeze | wc -l
12
$ pip-sync requirements.txt
$ pip freeze | wc -l
4

A txt which omits dependencies is, by pip-tools standards, malformed, and therefore pip-sync's behavior is not currently guaranteed or defined. But:

1. As @Zac-HD says below:

IMO "running pip-sync a second time makes no further changes" is a very important property to maintain

1. IMO a package is not "properly" installed if its dependencies aren't present, and pip-sync should achieve "proper" installs.

2. If a user wants to "break" dependencies, they can explicitly do so with --pip-args --no-deps

3. While I don't see an advantage to the current (inconsistent) behavior, having the deps always installed (unless --no-deps is specified) would enable users to use pip-sync without issue on txts that don't come from pip-compile. Though not officially supported, it's a nice bonus to be able to use pip-sync uniformly as a complete replacement/wrapper for pip install -r. Now that #927 is merged, I don't know of any other obstacle to supporting "foreign" txts (in practice, if not policy).

My idea of a solution is #907, which keeps already-installed packages in the to-install set, as pip itself will refrain from reinstalling those anyway, and will ensure its dependencies get installed if necessary.

---

Original report:

I looked but didn't find an existing issue for this.

What's the problem this feature will solve?

I would like to use pip-sync on requirements.txt files that are not generated by pip-compile, for arbitrary (other folks') Python projects. In fact I had been doing so, and I guess I have been getting lucky, as I only recently came to a case where this failed, and was surprised to find when re-reading pip-tools's README:

Be careful: pip-sync is meant to be used only with a requirements.txt generated by pip-compile

The failure I notice is that pip-sync will uninstall hard dependencies of packages listed in the requirements.txt. I don't know if there are further problems.

Describe the solution you'd like

Until/unless there are other specifically identified problems for externally crafted requirements.txt files, the solution would seem to be for pip-sync to refrain from uninstalling dependencies of explicitly listed packages. This might be as simple as doing all the uninstalling before any of the installing.

This would enable pip-tools users to continue to use pip-sync across all Python projects they work on, even if they are not in a position to enforce use of pip-compile to manage those projects' requirements.txt generation.

Alternative Solutions

Alternatively, of course, users can just use pip install -r requirements.txt. That however does not have the advantage of stripping the environment of non-required packages. Although I guess the solution could then be as simple as pip-sync requirements.txt && pip install -r requirements.txt.

Additional context

Simple demonstration of the problem:

echo "pytest==5.1.2" > requirements.txt
pip install -r requirements.txt
pip freeze

atomicwrites==1.3.0
attrs==19.1.0
Click==7.0
importlib-metadata==0.22
more-itertools==7.2.0
packaging==19.1
pip-tools==4.1.0
pluggy==0.13.0
py==1.8.0
pyparsing==2.4.2
pytest==5.1.2
six==1.12.0
wcwidth==0.1.7
zipp==0.6.0

pip-sync requirements.txt
pip freeze



md5-4d028eb40a1544998a4ee2e572b29e82



Click==7.0
pip-tools==4.1.0
pytest==5.1.2
six==1.12.0

The result is a broken installation of pytest.

[AndydeCleyre]

@ulope Can you please comment here?

[ulope]

@AndydeCleyre I assume you pinged me because of my thumbs-down vote.
I'm just a user of pip-tools so my vote has no more weight than anyone else's, but to spell out why I think this is a bad idea:

What you request would be a complete reversal of how pip-tools works. All the 'smarts' are in pip-compile. pip-sync just ensures that the installed packages exactly match the requirements file. It seems unwise to change this.

Maybe it would be possible for your case to pass your initial (third party generated) requirements.txt through pip-compile first and that way get a pip-sync compatible output.

[AndydeCleyre]

Thanks, yes, I wanted to encourage discussion here.

On the other side, I'd say that installing a requirement's dependencies is part of installing that requirement, and that users should expect any installed package to have its dependencies also installed.

I took a look and saw that piptools/sync.py:sync already does the uninstalling before the installing, which I thought would be a solution to this issue. It turns out the current behavior is more subtle and less predictable:

echo "pytest==5.1.2" > requirements.txt
pip install -r requirements.txt
pip-sync requirements.txt
# pytest remains installed but broken, with dependencies removed
pip-sync /dev/null
pip-sync requirements.txt
# pytest is installed "properly," with its dependencies included

In other words, if pip-sync has to actively install a requirement, its dependencies will end up in the environment. But if the requirement is already installed, its dependencies not present in the txt will be removed, breaking the environment.

Here the inconsistency is more simply demonstrated:

echo "pytest==5.1.2" > requirements.txt
pip-sync /dev/null
pip-sync requirements.txt
pip freeze | wc -l
# 14
pip-sync requirements.txt
pip freeze | wc -l
# 4

So the environment state that pip-sync will achieve depends not only on the requirements files, but also the state of the environment at time of operation. I don't think this is desirable or predictable.

While my position is for not breaking dependencies, either way the project goes on that, I think we can agree that it should behave more consistently in this case.

I will submit a pull request for my idea of a fix for this, though I understand that it will likely just be an anchor for continued discussion of "correct" behavior here.

[Zac-HD]

pass your initial (third party generated) requirements.txt through pip-compile first and that way get a pip-sync compatible output.

That's my preferred solution!

Perhaps pip-sync could also use pip's --no-deps option to avoid installing non-specified dependencies? That way you'd get the same result regardless of what was installed initially.

IMO "running pip-sync a second time makes no further changes" is a very important property to maintain, while in this case it wouldn't even converge to a stable point.

[atugushev]

Perhaps pip-sync could also use pip's --no-deps option to avoid installing non-specified dependencies? That way you'd get the same result regardless of what was installed initially.

That's an interesting idea!

[AndydeCleyre]

Outdated:

Adding --no-deps is easy! I don't know if that addition merits a test, as the actual behavior change happens in pip proper, rather than pip-tools. EDIT: I've added to the existing test for options passed on to pip.

I've included that change in #907. Maybe someone can test to see if it's giving them the behavior they want/expect, with and without --no-deps, and offer feedback?

Seems good to me:

echo "pytest==5.1.2" > requirements.txt
pip-sync /dev/null
pip-sync requirements.txt
pip freeze | wc -l
# 14
pip-sync requirements.txt
pip freeze | wc -l
# 14
pip-sync --no-deps requirements.txt
pip freeze | wc -l
# 4
pip-sync /dev/null
pip-sync --no-deps requirements.txt
pip freeze | wc -l
# 4

[AndydeCleyre]

@Zac-HD @ulope

What do you think of #907? It brings consistent behavior where we currently have surprises, and it brings in --no-deps for when you want to break your dependencies.

I don't think --no-deps or default-implied "yes-deps" is out of scope for pip-sync as pip-sync is largely analogous to pip install, which behaves that way.

[AndydeCleyre]

Outdated:

I'll add here that while #907 solves this dependency-breaking, inconsistent and unpredictable behavior, I've identified another obstacle to achieving the above-proposed goal of supporting foreign requirements.txt files: #925, which will be resolved if #927 gets merged.

[AndydeCleyre]

@ulope What do you think? In order to fix the inconsistent behavior we'll have to change what pip-sync does one way or another. #907 gives us the choice of --no-deps or not, just like pip install.

[AndydeCleyre]

Alright, I've rebased #907 onto master, now that we've got --pip-args. With #907, here's what the consistent behavior looks like:

$ echo "pytest==5.1.2" > requirements.txt
$ pip-sync /dev/null
$ pip-sync requirements.txt
$ pip freeze | wc -l
12
$ pip-sync requirements.txt
$ pip freeze | wc -l
12
$ pip-sync --pip-args --no-deps requirements.txt
$ pip freeze | wc -l
4
$ pip-sync /dev/null
$ pip-sync --pip-args --no-deps requirements.txt
$ pip freeze | wc -l
4

And here's the behavior in current master, without #907:

$ echo "pytest==5.1.2" > requirements.txt
$ pip-sync /dev/null
$ pip-sync requirements.txt
$ pip freeze | wc -l
12
$ pip-sync requirements.txt
$ pip freeze | wc -l
4
$ pip-sync --pip-args --no-deps requirements.txt
$ pip freeze | wc -l
4
$ pip-sync /dev/null
$ pip-sync --pip-args --no-deps requirements.txt
$ pip freeze | wc -l
4

This shows that --pip-args --no-deps allows us to achieve consistent behavior if --no-deps behavior is desired, and that #907 will offer consistent behavior if you want installed packages to have their dependencies installed.

Please have a look and let me know what you think, @atugushev @Zac-HD @Minkey27 @ulope

[ulope]

Unfortunately my opinion hasn't changed from when I made my last comment in this issue.

For me the whole point of using pip-tools is that the generated requirements.txt exactly describes what the installed virtualenv will look like afterwards.

I don't think it's a good idea to change the default (which would also be backwards incompatible as well) and force everyone wanting to retain the current behaviour to pass --pip-args --no-deps in the future.

As for the inconsistency you note in #907 (12 vs 4 packages installed), that seems like a bug to me. It should always behave in the no-deps way and only install 4 packages.

So TL;DR:
I'm not at all against providing the "include deps" behaviour as an option but I'm very much against making it the default.

[AndydeCleyre]

I appreciate the feedback @ulope, thanks!

[AndydeCleyre]

@ulope and anyone else, more discussion please!

If folks are currently relying on the current behavior, they're going to be surprised and run in to trouble. Unless they both read this discussion and inspect/ensure their environment before each pip-sync, there's no consistency to be expected or preserved. If you want the --no-deps behavior in the current release, you already must pass that option to be sure it behaves that way.

That is just to say that I don't buy backwards compatibility as an argument in this case, as it can't be achieved while also achieving consistency, which I think everyone wants.

So I'd like to learn more about the proper behavior argument. Can you elaborate on your use cases wherein you don't want package dependencies to be satisfied?

Thanks for any info!

[Minkey27]

I'm sorry for the late reaction, been swamped with work and was unable to keep track of the discussion.

From my understanding, pip-tools is a package that allows you to pin you enviroments pip packages.
For this two work, there are basically two commands pip-compile and pip-sync. By using a requirements.in and pip-compile you specify any top level packages you need and let pip-compile sort out the dependencies and pins them in a requirements.txt. (If it's your first time running it and there is no requirements.txt present, it will take all the latest deps. If you have a requirements.txt and deps are defined that have a version that is accepted, it leaves it intact). pip-sync in its turn installs everything and only everything from requirements.txt. This to stay in line with aforementioned thought process and pip-compile. To me this is the whole reason why one should want to use pip-tools. This gives me complete rebuildable (virtual) enviroments. Makes development to production a lot easier.

If I understand correctly this discussion is solely about the behaviour of pip-sync not resolving any deps that pip-compile solves for you. If that's wanted that is fine and we should implement it. But be aware, changing the default behaviour to resolving and installing deps during a pip-sync defeats the purpose of this package IMO.

Looking back at the original comment. I'm unsure why one would change the requirements.txt directly instead of updating requirements.in and run pip-compile. Wouldn't re-creating a virtualenv and installing through pip install -r requirements do the same?

TL;DR
By default, pip-sync is supposed to be used together with pip-compile and should pip-sync should not change its default behaviour to counteract pip-compile

[AndydeCleyre]

Thanks @Minkey27 !

. . . deps that pip-compile solves for you.

Yes, but note that pip itself ultimately does the resolving (important because it's used in pip-sync as well).

But be aware, changing the default behaviour to resolving and installing deps during a pip-sync defeats the purpose of this package IMO.

This is already and AFAIK has always been the default, when starting from a clean environment. If you want to be sure to omit unpinned dependencies from the environment, you must do something like

$ pip-sync && pip-sync

or the recently made possible

$ pip-sync --pip-args --no-deps

While compile-then-sync-on-same-platform is the officially supported workflow, improvements have been made to cover "simple" cases outside that workflow, as users still want and try to do things like use a single txt for multiple platforms (see discussion at #563 about #460). This change would cover a few more cases like that: if a subset of target platforms had extra dependencies, this would prevent broken installations via pip-sync when the txt is compiled on a platform without those. This would not be making a claim to fully support mixed-platform + single txt workflows, but can support some more of them, just like #460.

Another benefit would be that pip-sync could be used with good results instead of pip install -r in pip-compiled projects and non-pip-compiled projects alike.

And I still think a package "installed" without its requirements is only partially installed -- broken.

I understand in principle the expectation of the pip-sync behavior you describe (which again I must note is not the current behavior), but can you elaborate on your actual use cases wherein you don't want package dependencies to be satisfied? It would help me greatly in understanding the practical benefit of that behavior.

Thanks for any info!

[Minkey27]

I don't think I'm the expert of all the ins and outs of this package. Personally I haven't noticed that pip-sync installs deps of packages in the txt files even though they aren't present. I try to maintain a complete txt file at all time.

As to our actual usecase:
We currently use pip-tools to pin our entire virtualenv so that we have reproducible deploys on production. We were in need of this due to our week of release testing. This means that at most, we could have 2 weeks between development and deployment. These two weeks are enough for a sub dep to be updated and / or broken. I think we all agree that's undesirable.

All our top-level deps are listed in the .in file and a txt file is created. We basically have two .in files. requirements.in and requirements-dev.in. The dev.in inherits (-c requirements.txt from requirements.txt) and contains any additional packages needed for developement (i.e. django-debug-toolbar).

Whenever we add a new package or need to update one, we edit requirements.in and let pip-compile do its work. Workflow as suggested in the README. We never touch the txt files.

We do have a dev team that run different OS's and granted that does give some issues. As far as I could tell (and remember) there were some issues regarding env markers and OS X vs Linux distro's resulting in different txt files.

We solved the enviroment marker issue by making our codebase py3 only (which was what we needed it for). The second we solved a bit hacky by adding the following to the bottom of our .in file.

##############################################################################
##### Garbage pins, used for pinning packages that are platform specific. ####
##############################################################################

# Simple package for disabling App Nap on OS X 10.9, which can be problematic.
appnope==0.1.0            # via ipython

So basically we have worked around most of the problems.

Coming back to the problem at hand. In regards to your original report. I still agree with @ulope and I hope to have illustrated in my workflow why such a thing would be a bad idea.

As to your updated report:

If pip-sync has to actively install a requirement, its dependencies will end up in the environment. But if the requirement is already installed when pip-sync runs, its dependencies not present in the txt will be removed, "breaking" the environment.

I think this is undesired behaviour. It is inconsistent to the very least. and this should be one or the other. And I prefer pip-sync to strictly listen to the txt file.

[AndydeCleyre]

@Minkey27

Thanks for sharing the workflow!

I hope to have illustrated in my workflow why such a thing would be a bad idea.

I don't see what would change about your workflow with this proposal. Where is there a case of wanting to omit hard dependencies? Can you please clarify?

---

FWIW re: mixed env markers, you may be interested in this discussion. For example, now you can have:

macos-requirements.in:

appnope ; sys_platform == 'darwin'

and on a mac, compile that to macos-requirements.txt:

appnope==0.1.0 ; sys_platform == 'darwin'    # via -r macos-requirements.in

Then a base-requirements.in:

ipython

compiled on any system to base-requirements.txt.

And then a manually composed requirements.txt:

-r base-requirements.txt
-r macos-requirements.txt

With that, pip install -r requirements.txt or pip-sync requirements.txt will do the right thing, regardless of platform pip-sync is run on.

See the platform-compile.sh script in that linked comment for more possibilities, and I encourage you to add info there on how multi-env workflows could be better for you!

[AndydeCleyre]

Can someone who is against this change please provide a real use case which this change would break? None has been provided yet, which leaves open the possibility that there is a misunderstanding that it would break something which it would not.

[AndydeCleyre]

Just checking in to note that:

• I've rebased #907 onto latest master
• despite vague suggestions of potential broken workflows, no one has yet articulated one

---

@ulope @Minkey27

Have you experienced or witnessed an actual workflow that would be negatively affected by #907?

@atugushev @jdufresne @vphilippon

Have you ever encountered such a workflow?

[atugushev]

@AndydeCleyre

Have you ever encountered such a workflow?

I've never used pip-sync outside of pip-compile and honestly using pip-sync as a separate tool looks unnatural to me. However, if you feel that pip-sync could be used solely, perhaps, it's a good idea for a side-project.

As for the inconsistency you note in #907 (12 vs 4 packages installed), that seems like a bug to me. It should always behave in the no-deps way and only install 4 packages.

I agree with @ulope here. The fact that pip-sync installs dependencies without --no-deps flag I consider as a bug.

[AndydeCleyre]

A year and a half since opening, and I'm still curious if anyone has encountered a workflow that would be broken by #907.

I believe some are concerned that existing workflows would be disrupted, but AFAICT that is untrue.

There may be good reason to reject this change, so let's not muddy the waters with bugaboos.