Galaxy: startup error ImportError: No module named bigbed_file on OS X

[mvdbeek]

(.venv)mariuss-MBP:galaxy marius$ sh run.sh
Activating virtualenv at .venv
Ignoring indexes: https://pypi.python.org/simple
Requirement already up-to-date: pip==8.0.2+gx1 in ./.venv/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): bx-python==0.7.3 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe==0.23 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): PyYAML==3.11 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 4))
Requirement already satisfied (use --upgrade to upgrade): SQLAlchemy==1.0.8 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): mercurial==3.4.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): numpy==1.9.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 8))
Requirement already satisfied (use --upgrade to upgrade): pycrypto==2.6.1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 9))
Requirement already satisfied (use --upgrade to upgrade): Paste==2.0.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 16))
Requirement already satisfied (use --upgrade to upgrade): PasteDeploy==1.5.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 17))
Requirement already satisfied (use --upgrade to upgrade): docutils==0.12 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 18))
Requirement already satisfied (use --upgrade to upgrade): wchartype==0.1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 19))
Requirement already satisfied (use --upgrade to upgrade): repoze.lru==0.6 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 20))
Requirement already satisfied (use --upgrade to upgrade): Routes==2.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 21))
Requirement already satisfied (use --upgrade to upgrade): WebOb==1.4.1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 22))
Requirement already satisfied (use --upgrade to upgrade): WebHelpers==1.3 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 23))
Requirement already satisfied (use --upgrade to upgrade): Mako==1.0.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 24))
Requirement already satisfied (use --upgrade to upgrade): pytz==2015.4 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 25))
Requirement already satisfied (use --upgrade to upgrade): Babel==2.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 26))
Requirement already satisfied (use --upgrade to upgrade): Beaker==1.7.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 27))
Requirement already satisfied (use --upgrade to upgrade): Cheetah==2.4.4 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 30))
Requirement already satisfied (use --upgrade to upgrade): Markdown==2.6.3 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 31))
Requirement already satisfied (use --upgrade to upgrade): bioblend==0.6.1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 34))
Requirement already satisfied (use --upgrade to upgrade): boto==2.38.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 35))
Requirement already satisfied (use --upgrade to upgrade): requests==2.8.1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 36))
Requirement already satisfied (use --upgrade to upgrade): requests-toolbelt==0.4.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 37))
Requirement already satisfied (use --upgrade to upgrade): kombu==3.0.30 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 40))
Requirement already satisfied (use --upgrade to upgrade): amqp==1.4.8 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 41))
Requirement already satisfied (use --upgrade to upgrade): anyjson==0.3.3 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 42))
Requirement already satisfied (use --upgrade to upgrade): sqlalchemy-migrate==0.10.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 45))
Requirement already satisfied (use --upgrade to upgrade): decorator==4.0.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 46))
Requirement already satisfied (use --upgrade to upgrade): Tempita==0.5.3dev in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 47))
Requirement already satisfied (use --upgrade to upgrade): sqlparse==0.1.16 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 48))
Requirement already satisfied (use --upgrade to upgrade): pbr==1.8.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 49))
Requirement already satisfied (use --upgrade to upgrade): six==1.9.0 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 51))
Requirement already satisfied (use --upgrade to upgrade): Parsley==1.3 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 52))
Requirement already satisfied (use --upgrade to upgrade): nose==1.3.7 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 53))
Requirement already satisfied (use --upgrade to upgrade): SVGFig==1.1.6 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 54))
Requirement already satisfied (use --upgrade to upgrade): Fabric==1.10.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 57))
Requirement already satisfied (use --upgrade to upgrade): paramiko==1.15.2 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 58))
Requirement already satisfied (use --upgrade to upgrade): ecdsa==0.13 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 59))
Requirement already satisfied (use --upgrade to upgrade): Whoosh==2.4.1+gx1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 66))
Requirement already satisfied (use --upgrade to upgrade): pysam==0.8.3+gx1 in ./.venv/lib/python2.7/site-packages (from -r requirements.txt (line 69))
Requirement already satisfied (use --upgrade to upgrade): WebError==0.10.3 in ./.venv/lib/python2.7/site-packages (from -r /dev/stdin (line 1))
Requirement already satisfied (use --upgrade to upgrade): Pygments==2.0.2 in ./.venv/lib/python2.7/site-packages (from -r /dev/stdin (line 2))
Requirement already satisfied (use --upgrade to upgrade): Tempita in ./.venv/lib/python2.7/site-packages (from WebError==0.10.3->-r /dev/stdin (line 1))
Requirement already satisfied (use --upgrade to upgrade): Paste>=1.7.1 in ./.venv/lib/python2.7/site-packages (from WebError==0.10.3->-r /dev/stdin (line 1))
Requirement already satisfied (use --upgrade to upgrade): WebOb in ./.venv/lib/python2.7/site-packages (from WebError==0.10.3->-r /dev/stdin (line 1))
Requirement already satisfied (use --upgrade to upgrade): six in ./.venv/lib/python2.7/site-packages (from Paste>=1.7.1->WebError==0.10.3->-r /dev/stdin (line 1))
Activating virtualenv at .venv
Traceback (most recent call last):
  File "./scripts/paster.py", line 27, in <module>
    serve.run()
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/serve.py", line 1061, in run
    invoke(command, command_name, options, args[1:])
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/serve.py", line 1067, in invoke
    exit_code = runner.run(args)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/serve.py", line 223, in run
    result = self.command()
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/serve.py", line 639, in command
    app = loadapp( app_spec, name=app_name, relative_to=base, global_conf=vars)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 292, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 316, in loadobj
    global_conf=global_conf)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 341, in loadcontext
    global_conf=global_conf)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 365, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 503, in get_context
    section)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 562, in _context_from_explicit
    value = import_string(found_expr)
  File "/Users/marius/src/galaxy/lib/galaxy/util/pastescript/loadwsgi.py", line 122, in import_string
    return pkg_resources.EntryPoint.parse("x=" + s).load(False)
  File "/Users/marius/src/galaxy/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/Users/marius/src/galaxy/.venv/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/marius/src/galaxy/lib/galaxy/web/buildapp.py", line 5, in <module>
    from galaxy.webapps.galaxy.buildapp import app_factory
  File "/Users/marius/src/galaxy/lib/galaxy/webapps/galaxy/buildapp.py", line 15, in <module>
    import galaxy.app
  File "/Users/marius/src/galaxy/lib/galaxy/app.py", line 23, in <module>
    from galaxy.visualization.data_providers.registry import DataProviderRegistry
  File "/Users/marius/src/galaxy/lib/galaxy/visualization/data_providers/registry.py", line 2, in <module>
    from galaxy.visualization.data_providers import genome
  File "/Users/marius/src/galaxy/lib/galaxy/visualization/data_providers/genome.py", line 15, in <module>
    from bx.bbi.bigbed_file import BigBedFile
ImportError: No module named bigbed_file

This is both on release_16.01 and dev on a fresh clone.

[dannon]

Hi @mvdbeek, I'm seeing this now, after experiencing https://github.com/galaxyproject/galaxy/issues/1640

(potentially unrelated, but I didn't see this error until after dumping the .venv completely and restarting)

[natefoo]

This indicates a bx-python wheel built without numpy present. Must be the case with the OS X wheels, although these are not recently changed, so this is a bit perplexing.

[dannon]

@mvdbeek Do you remember if the bx-python eggs were built locally or if the correct ones were fetched from wheels.galaxyproject.org?

[dannon]

@mvdbeek Yeah, it's a bug in our version of pip. https://github.com/natefoo/pip/pull/1 should fix it.

[mvdbeek]

Indeed, I confirm it's fixed. Thanks @dannon @natefoo !
Btw, OS X is a supported build platform on travis, would you accept a PR that just tests if run.sh works on OS X?

[mvdbeek]

Ahh, it's not (yet) fixed, I was still on the branch on which I commented out all references to BigBedFile .... which makes sense given that that the PR is not merged yet. ouch.

[dannon]

@mvdbeek Sorry, yeah, not merged quite yet. You could make the same modification to .venv/lib/site-packages/pip/ in your Galaxy dir to sort things out until it's merged (that's what I did)

[natefoo]

Could you test that it's fixed if you do the following (move/remove .venv first)?:

$ sh ./scripts/common_startup.sh --skip-wheels --no-replace-pip
$ PYTHONPATH= ./.venv/bin/pip install https://depot.galaxyproject.org/starforge/wheels/build-44/pip-8.0.2%2Bgx2-py2.py3-none-any.whl
$ sh ./run.sh --no-replace-pip

[dannon]

@natefoo Verifying now.

[dannon]

@natefoo Yep, works.

[dannon]

Closed with #1652

[dannon]

For reference, @zipho saw the same issue due to a conda install being present and the .venv being skipped. @remimarenco is updating common_startup to more explicitly display this at startup.