Brew: Problems with virtualenv system & dependency formulae.

[DomT4]

• [x] Ran brew update and retried your prior step?
• [x] Ran brew doctor, fixed as many issues as possible and retried your prior step?
• [ ] If you're seeing permission errors tried running sudo chown -R $(whoami) $(brew --prefix)?

It seems like things created with the new Python virtualenv system can't find dependencies that reside outside the virtualenv? That sounds incredibly obvious saying it, but I'm not sure if it was an intentional design decision & how to work around it.

For example, something that needs to import gobject typically has a depends_on "pygobject" dependency, but if you add that dependency & build something with:

    venv = virtualenv_create(libexec)
    %w[
      setuptools attrs cffi chardet cryptography enum34 idna ipaddress
      pyasn1 pyasn1-modules pycparser Pillow pyOpenSSL pyxdg six
      service_identity Twisted zope.interface MarkupSafe Mako
    ].each do |r|
      venv.pip_install resource(r)
    end
    venv.link_scripts(bin) { venv.pip_install buildpath }

and then run the software:

deluge

[ERROR   ] 02:37:24 ui:171 No module named gobject
Traceback (most recent call last):
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/ui.py", line 152, in __init__
    from deluge.ui.gtkui.gtkui import GtkUI
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/gtkui/__init__.py", line 1, in <module>
    from gtkui import start
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/gtkui/gtkui.py", line 37, in <module>
    import gobject
ImportError: No module named gobject
[ERROR   ] 02:37:24 ui:172 There was an error whilst launching the request UI: gtk
[ERROR   ] 02:37:24 ui:173 Look at the traceback above for more information.

This will presumably be problematic for anything that relies on the Python elements of _another formula_ rather than from a resource? Was this intentional? Am I doing something stupid?

[DomT4]

Note: The formula causing the problem here is proving problematic elsewhere, in terms of anything already being installed by pip globally being skipped over during the venv.pip_install resource(r) step, so I may need to check this with another formula in case it's an isolated thing.

The skipping over problem is:

Click me

==> /usr/local/Cellar/deluge/1.3.13/libexec/bin/pip install -v --no-deps --no-binary :all: /private/tmp/deluge--pyOpenSSL-20160730-52206-1afp4zf/pyOpenSSL-16.0.0
Processing /private/tmp/deluge--pyOpenSSL-20160730-52206-1afp4zf/pyOpenSSL-16.0.0
  Running setup.py (path:/var/folders/qs/rg1v7kz56wv3v9z4qwgtv3g80000gn/T/pip-dFAohc-build/setup.py) egg_info for package from file:///private/tmp/deluge--pyOpenSSL-20160730-52206-1afp4zf/pyOpenSSL-16.0.0
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/pyOpenSSL.egg-info
    writing requirements to pip-egg-info/pyOpenSSL.egg-info/requires.txt
    writing pip-egg-info/pyOpenSSL.egg-info/PKG-INFO
    writing top-level names to pip-egg-info/pyOpenSSL.egg-info/top_level.txt
    writing dependency_links to pip-egg-info/pyOpenSSL.egg-info/dependency_links.txt
    writing manifest file 'pip-egg-info/pyOpenSSL.egg-info/SOURCES.txt'
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'pip-egg-info/pyOpenSSL.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no previously-included files found matching 'leakcheck'
    warning: no previously-included files matching '*.py' found under directory 'leakcheck'
    warning: no previously-included files matching '*.pem' found under directory 'leakcheck'
    no previously-included directories found matching 'doc/_build'
    no previously-included directories found matching '.travis'
    writing manifest file 'pip-egg-info/pyOpenSSL.egg-info/SOURCES.txt'
  Source in /var/folders/qs/rg1v7kz56wv3v9z4qwgtv3g80000gn/T/pip-dFAohc-build has version 16.0.0, which satisfies requirement pyOpenSSL==16.0.0 from file:///private/tmp/deluge--pyOpenSSL-20160730-52206-1afp4zf/pyOpenSSL-16.0.0
  Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL==16.0.0 from file:///private/tmp/deluge--pyOpenSSL-20160730-52206-1afp4zf/pyOpenSSL-16.0.0 in /usr/local/lib/python2.7/site-packages
Skipping bdist_wheel for pyOpenSSL, due to binaries being disabled for it.
Cleaning up...
  Removing source in /var/folders/qs/rg1v7kz56wv3v9z4qwgtv3g80000gn/T/pip-dFAohc-build

Testing domt4/crypto/deluge
==> Using the sandbox
/usr/bin/sandbox-exec -f /tmp/homebrew20160730-51762-c3gvzs.sb /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/bin/ruby -W0 -I /usr/local/Library/Homebrew -- /usr/local/Library/Homebrew/test.rb /usr/local/Library/Taps/domt4/homebrew-crypto/Formula/deluge.rb -v
Traceback (most recent call last):
  File "/usr/local/Cellar/deluge/1.3.13/bin/deluge-web", line 9, in <module>
    load_entry_point('deluge==1.3.13', 'console_scripts', 'deluge-web')()
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/web/web.py", line 138, in start
    web.start()
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/web/web.py", line 107, in start
    import server
  File "/usr/local/Cellar/deluge/1.3.13/libexec/lib/python2.7/site-packages/deluge/ui/web/server.py", line 50, in <module>
    from OpenSSL.crypto import FILETYPE_PEM
ImportError: No module named OpenSSL.crypto
==> curl localhost:8081
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (7) Failed to connect to localhost port 8081: Connection refused
Error: domt4/crypto/deluge: failed
<0> expected but was
<7>.

[ilovezfs]

I think in the formula you need to write out an environment script around the executable to pull in the other stuff so it's in the PYTHONPATH, no?

[ilovezfs]

@DomT4 Unless I've misunderstood the scenario, see https://github.com/Homebrew/brew/pull/344#issuecomment-234242676 and Tim's response.

CC @tdsmith

[DomT4]

Unless I've misunderstood the scenario, see #344 (comment) and Tim's response.

Ah, I didn't see that. I kind of let you & Tim hash things out in that thread before I got involved 😄. I checked the new documentation but didn't find anything about needing to find the Python elements of other formulae, which left me a little confused on how to handle it best.

So instead of:

venv = virtualenv_create(libexec)
%w[
  setuptools attrs cffi chardet cryptography enum34 idna ipaddress
  pyasn1 pyasn1-modules pycparser Pillow pyOpenSSL pyxdg six
  service_identity Twisted zope.interface MarkupSafe Mako
].each do |r|
  venv.pip_install resource(r)
end
venv.link_scripts(bin) { venv.pip_install buildpath }

You'd do something like:

venv = virtualenv_create(libexec)
%w[
  setuptools attrs cffi chardet cryptography enum34 idna ipaddress
  pyasn1 pyasn1-modules pycparser Pillow pyOpenSSL pyxdg six
  service_identity Twisted zope.interface MarkupSafe Mako
].each do |r|
  venv.pip_install resource(r)
end

ENV.prepend_create_path "PYTHONPATH", Formula["pygobject"].opt_lib/"python2.7/site-packages"
(bin/"deluge").write_env_script libexec/"bin/deluge", :PYTHONPATH => ENV["PYTHONPATH"]

❓

[ilovezfs]

I kind of let you & Tim hash things out in that thread before I got involved :smile:

Yes, I realize I'm probably lucky I didn't get :facepunch:'ed.

You'd do something like

You'd also have to do

venv.pip_install buildpath

I think that should work, assuming there aren't any other executables that would have been linked by venv.link_scripts.

In general, it might be better to do the canonical

venv.link_scripts(bin) { venv.pip_install buildpath }

and then fixup what was placed in bin with bin.env_script_all_files(libexec/"bin", env), since there's magical logic in venv.link_scripts(bin) that determines what the difference is between what executables are there before yielding to the block with venv.pip_install buildpath, and what executables are there after the block returns, and then it only links the "new" ones.

[tdsmith]

Probably don't do any of that. :D

The simplest thing to do is, in the depending formula:
(libexec/"lib/python2.7/site-packages/deps.pth").write "import site; site.addsitedir('#{Formula["foo"].lib/"python2.7/site-packages"}')\n"

I think virtualenv_create can/should do this automatically.

[tdsmith]

anything already being installed by pip globally being skipped over during the venv.pip_install resource(r) step

This is some hot garbage which should not be happening >:

[ilovezfs]

I think virtualenv_create can/should do this automatically.

Even if it's from another formula? Should this be handled by superenv?

[DomT4]

This is some hot garbage which should not be happening >:

I worked around it by locally adding --ignore-installed to our pip instructions, but I'm not sure how legit a fix that is so I didn't want to swing by with a completely wrong PR, heh.

[tdsmith]

Even if it's from another formula? Should this be handled by superenv?

What do you mean by from another formula? Superenv is irrelevant since the Python modules need to be found at runtime.

I worked around it by adding --ignore-installed to our pip instructions

It's probably the necessary fix and we'll want to add that anyway if we're auto-adding dependencies to the virtualenv.

[ilovezfs]

What do you mean by from another formula? Superenv is irrelevant since the Python modules need to be found at runtime.

I mean for determining the PYTHONPATH so that modules from depends_ons get included in env scripts.

[tdsmith]

Oh, nah. Using PYTHONPATH at all ever is an antipattern in this brave new world.

[ilovezfs]

Got it. PYTHONPATH, bad. Aldous Huxley, good.

[tdsmith]

see #608