Pip: A way to blacklist 'provided' dependencies.

[zenlambda]

I have an appengine application, I would like to manage third party libraries with pip. I do this by adding each library to requirements.txt and executing pip -r requirements.txt -t libsubdir/. Then I arrange for appengine to add the libsubdir to the pythonpath. This works fine except when a third party package brings in a package normally provided by appengine (which may include native code that won't work if loaded into the sandbox).

E.g. say I depend on ua-parser, it will bring in pyyaml, but pyyaml (3.10) is provided by appengine. So I'd like a way to exclude PyYAML in requirements.txt.

PyYAML would then be specified in the appengine app.yaml. It would probably have to be installed as a system dependency for local development, this would be managed by the developer just as for other common appengine provided libraries such as PyCrypto, PIL, etc.

[xavfernandez]

I think you could use constraints file for that, containing the version provided by app engine.

If you have a constraint.txt file containing:

pyyaml==3.10

And a dependency requires pyyaml, pip will merge those and act as if pyyaml==3.10 was required.

Cf https://pip.pypa.io/en/stable/user_guide/#constraints-files for more infos.

[zenlambda]

@xavfernandez Thanks for your reply. I've just tried what you said. I'm afraid it didn't have the effect you described. PyYaml continues to be downloaded :( .

First I removed the explicit dependency on ua_parser==0.3.6, so that the latest ua_parser gets downloaded, which now depends on PyYAML (which I want to exclude and manage myself).

From this:

» cat requirements.txt
user-agents==0.3.2
ua_parser==0.3.6

To this:

» cat requirements.txt
user-agents==0.3.2

Then I added a constraints file as you said:

pyyaml==3.10

Then I ran pip:

pip install -c constraints.txt -r requirements.txt -t server/lib
Collecting user-agents==0.3.2 (from -r requirements.txt (line 1))
  Using cached user-agents-0.3.2.tar.gz
Collecting ua-parser (from user-agents==0.3.2->-r requirements.txt (line 1))
  Using cached ua_parser-0.4.1-py2-none-any.whl
Collecting pyyaml==3.10 (from -c constraints.txt (line 1))
/Library/Python/2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Downloading PyYAML-3.10.tar.gz (241kB)
    100% |████████████████████████████████| 241kB 2.1MB/s 
Installing collected packages: pyyaml, ua-parser, user-agents
  Running setup.py install for pyyaml
  Running setup.py install for user-agents
Successfully installed pyyaml ua-parser user-agents

And I'm afraid pip went and downloaded and installed pyaml. :-(

[xavfernandez]

Mmh are you running this in a virtualenv ? with --system-site-packages option ?
What does pip list give you on a fresh install ?

[zenlambda]

I am not using virtualenv. I'm not sure at this point how to get virtualenv to refer to the google sdk directories and what the implications are for actually uploading external libraries to appengine. I use pip in a similar fashion to this description.

It just occurred to me that when I performed the steps above:

• PyYAML was not installed in the system packages
• pip would behave differently if it was (I guess you are referring to some behaviour where pip will not install something in the --target dir if it is the system packages).

So I installed PyYAML in my system libraries (no -t option) and repeated the procedure I, but I got the same result as before.

You might ask why did I run pip install without installing PyYAML in the system directories first? Well, the ideal behaviour for me is that if pip finds a requirement to install PyYAML and it is 'blacklisted' then it should do nothing (regardless of whether it is installed anywhere else on the system or not).

Here's my current pip list output:

altgraph (0.10.2)
astroid (1.3.8)
bdist-mpkg (0.5.0)
beautifulsoup4 (4.3.2)
bonjour-py (0.3)
boto (2.38.0)
crcmod (1.7)
flake8 (2.4.1)
gaedriver (0.1.0)
gcs-oauth2-boto-plugin (1.9)
google-api-python-client (1.4.1)
google-apitools (0.4.8)
gsutil (4.13)
httplib2 (0.9.1)
logilab-common (1.0.2)
macholib (1.5.1)
matplotlib (1.3.1)
mccabe (0.3.1)
modulegraph (0.10.4)
numpy (1.8.0rc1)
oauth2client (1.4.11)
pep8 (1.5.7)
Pillow (2.9.0)
pip (7.1.2)
protorpc (0.10.0)
py2app (0.7.3)
pyasn1 (0.1.7)
pyasn1-modules (0.0.5)
pycrypto (2.6.1)
pyfakefs (2.4)
pyflakes (0.8.1)
pylint (1.4.4)
pyobjc-core (2.5.1)
pyobjc-framework-Accounts (2.5.1)
pyobjc-framework-AddressBook (2.5.1)
pyobjc-framework-AppleScriptKit (2.5.1)
pyobjc-framework-AppleScriptObjC (2.5.1)
pyobjc-framework-Automator (2.5.1)
pyobjc-framework-CFNetwork (2.5.1)
pyobjc-framework-Cocoa (2.5.1)
pyobjc-framework-Collaboration (2.5.1)
pyobjc-framework-CoreData (2.5.1)
pyobjc-framework-CoreLocation (2.5.1)
pyobjc-framework-CoreText (2.5.1)
pyobjc-framework-DictionaryServices (2.5.1)
pyobjc-framework-EventKit (2.5.1)
pyobjc-framework-ExceptionHandling (2.5.1)
pyobjc-framework-FSEvents (2.5.1)
pyobjc-framework-InputMethodKit (2.5.1)
pyobjc-framework-InstallerPlugins (2.5.1)
pyobjc-framework-InstantMessage (2.5.1)
pyobjc-framework-LatentSemanticMapping (2.5.1)
pyobjc-framework-LaunchServices (2.5.1)
pyobjc-framework-Message (2.5.1)
pyobjc-framework-OpenDirectory (2.5.1)
pyobjc-framework-PreferencePanes (2.5.1)
pyobjc-framework-PubSub (2.5.1)
pyobjc-framework-QTKit (2.5.1)
pyobjc-framework-Quartz (2.5.1)
pyobjc-framework-ScreenSaver (2.5.1)
pyobjc-framework-ScriptingBridge (2.5.1)
pyobjc-framework-SearchKit (2.5.1)
pyobjc-framework-ServiceManagement (2.5.1)
pyobjc-framework-Social (2.5.1)
pyobjc-framework-SyncServices (2.5.1)
pyobjc-framework-SystemConfiguration (2.5.1)
pyobjc-framework-WebKit (2.5.1)
pyOpenSSL (0.13.1)
pyparsing (2.0.1)
python-dateutil (1.5)
python-gflags (2.0)
pytz (2013.7)
PyYAML (3.10)
retry-decorator (1.0.0)
rsa (3.1.4)
scipy (0.13.0b1)
setuptools (18.3.1)
simplejson (3.7.3)
six (1.9.0)
SocksiPy-branch (1.1)
Twisted (13.2.0)
unittest2 (0.8.0)
uritemplate (0.6)
vboxapi (1.0)
virtualenv (12.1.1)
waitress (0.8.9)
WebOb (1.4)
xattr (0.6.4)
zope.interface (4.1.1)

[xavfernandez]

My bad, I didn't catch the fact you were using --target option. This option currently implies the activation of --ignore-installed option which obviously ignores whether PyYAML is already available...

[bscipio]

A similar question was posed on stackoverflow: Pip install to custom target directory and exclude specific dependencies.

These would be useful to me, as I has similar situation as @zenlambda :
A feature like exclude/blacklist from a file:
pip install --log pip.log -r req.txt -target /mypath/pypkgs/ --exclude exclude.txt

or exclude dependencies already available in current environment:
pip install --log pip.log -r req.txt -target /mypath/pypkgs/ --exclude-current-env

[jirutka]

I also need this feature.

[benoit-pierre]

I don't understand why --target implies --ignore-installed, something that is not documented in the help. Not implicitly enabling it would make it possible to use pip install --target plugins_directory for installing plugins for an application that already provides some of the needed dependencies.

[dstufft]

I'm going to close this because fundamentally we don't want to provide an option to users that lets them create a broken environment (and outside of a narrow use case, that is all this option will do). I think a better solution here would be a wrapper script that calls pip and then deletes the directories of the packages you don't want installed.

[thomasf]

Well. any situation where dependencies are under-specified or packages which doesn't do predictable api compatibility guarantees (like semver) also can break things badly and it's usually worse.. I'm also getting a bit tired of having to publish my own forks of packages to pypi just to replace badly specified requirements somewhere...

An alternative to blacklisting for PIL, pycropto etc. could be be to allow adding syntax for explicitly saying that pilliow provides the same thing as PIL does. That would reduce a lot of headaches I currently have with more and more python projects nowerdays since compatible forks with their own pypi packages are becoming more common than before.

[sandrotosi]

yes, exactly that! i'm precisely in the situation where 2 different modules are depending one on PIL and the other on pillow

even something to let define that a module masks/maps to/provides another one would be enough here

[pbirnie]

I could really use --exclude exclude.txt functionality - we have transient dependencies like numpy that are "provided" at runtime. I add these to the PYTHONPATH - but because they dont have exactly the same version name 1.11.1 vs 1.11-mk1 - pip wants to install them

[ashb]

I think a better solution here would be a wrapper script that calls pip and then deletes the directories of the packages you don't want installed.

Exactly as with numpy the issue is the transitive deps (or the deps of the thing you want to exclude). In my case I want to exclude/mark-as-provided pyspark, and I don't want to have to know/work out what its deps might be to remove them manually.

@dstufft Can this issue be re-opened please? The PIL/pillow use case seems very hard to solve without something like --exclude, and the "module already provided in target env" use case gets much harder to manage when dealing with deps-of-deps.

[jpuskar]

This would be useful for creating AWS lambda function zip packages. In this use case, many AWS packages are pre-installed in a layer. Therefore, we don't need to package them into the ZIP file to upload.

The current process is:
pip install -r requirements.txt
rm -rf boto3 botocore
etc.

It would be faster to be able to:

pip install -r requrements.txt \
  --force-exclude-deps boto3 botocore \
  --allow-me-to-do-bad-things \
  --i_accept_all_the_blame \
  --please_just_do_it_this_way \
  --i_admit_i_am_a_bad_person

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.