pants bootstrap fails to build lmdb on macos v10.12.2, python 2.7.13

[KLFrost]

18:58 scorp10 ~$ uname -a
Darwin scorp10.local 16.3.0 Darwin Kernel Version 16.3.0: Thu Nov 17 20:23:58 PST 2016; root:xnu-3789.31.2~1/RELEASE_X86_64 x86_64
18:58 scorp10 ~$ mkdir pants-test && cd pants-test
18:58 scorp10 pants-test$ curl -L -O https://pantsbuild.github.io/setup/pants && chmod +x pants && touch pants.ini
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3711  100  3711    0     0  25905      0 --:--:-- --:--:-- --:--:-- 25951
18:59 scorp10 pants-test$ ./pants -ldebug -V
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0   122    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   281  100   281    0     0    634      0 --:--:-- --:--:-- --:--:--   634
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1802k  100 1802k    0     0  2045k      0 --:--:-- --:--:-- --:--:-- 2045k
New python executable in /Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.hsbuwP/install/bin/python2.7
Also creating executable in /Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.hsbuwP/install/bin/python
Installing pip, wheel...done.
Collecting setuptools==5.4.1
  Using cached setuptools-5.4.1-py2.py3-none-any.whl
Installing collected packages: setuptools
Successfully installed setuptools-5.4.1
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting pantsbuild.pants
Requirement already satisfied (use --upgrade to upgrade): setuptools==5.4.1 in /Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.hsbuwP/install/lib/python2.7/site-packages (from pantsbuild.pants)
Collecting twitter.common.collections<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting six<2,>=1.9.0 (from pantsbuild.pants)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting futures==3.0.5 (from pantsbuild.pants)
  Using cached futures-3.0.5-py2-none-any.whl
Collecting docutils<0.13,>=0.12 (from pantsbuild.pants)
Collecting twitter.common.confluence<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting pytest<2.7,>=2.6 (from pantsbuild.pants)
Collecting Markdown==2.1.1 (from pantsbuild.pants)
Collecting twitter.common.dirutil<0.4,>=0.3.1 (from pantsbuild.pants)
Collecting ansicolors==1.0.2 (from pantsbuild.pants)
Collecting pytest-cov<1.9,>=1.8 (from pantsbuild.pants)
Collecting cffi==1.7.0 (from pantsbuild.pants)
  Using cached cffi-1.7.0-cp27-cp27m-macosx_10_10_intel.whl
Collecting requests<2.6,>=2.5.0 (from pantsbuild.pants)
  Using cached requests-2.5.3-py2.py3-none-any.whl
Collecting pywatchman==1.3.0 (from pantsbuild.pants)
Collecting coverage<3.8,>=3.7 (from pantsbuild.pants)
Collecting pex==1.1.13 (from pantsbuild.pants)
  Using cached pex-1.1.13-py2.py3-none-any.whl
Collecting setproctitle==1.1.10 (from pantsbuild.pants)
Collecting lmdb==0.89 (from pantsbuild.pants)
Collecting pystache==0.5.3 (from pantsbuild.pants)
Collecting fasteners==0.14.1 (from pantsbuild.pants)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting psutil==4.3.0 (from pantsbuild.pants)
Collecting pathspec==0.3.4 (from pantsbuild.pants)
Collecting packaging==16.7 (from pantsbuild.pants)
  Using cached packaging-16.7-py2.py3-none-any.whl
Collecting Pygments==1.4 (from pantsbuild.pants)
Collecting scandir==1.2 (from pantsbuild.pants)
Collecting twitter.common.lang==0.3.9 (from twitter.common.collections<0.4,>=0.3.1->pantsbuild.pants)
Collecting twitter.common.log==0.3.9 (from twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants)
Collecting py>=1.4.25 (from pytest<2.7,>=2.6->pantsbuild.pants)
  Using cached py-1.4.32-py2.py3-none-any.whl
Collecting cov-core>=1.14.0 (from pytest-cov<1.9,>=1.8->pantsbuild.pants)
Collecting pycparser (from cffi==1.7.0->pantsbuild.pants)
Collecting monotonic>=0.1 (from fasteners==0.14.1->pantsbuild.pants)
  Using cached monotonic-1.2-py2.py3-none-any.whl
Collecting pyparsing (from packaging==16.7->pantsbuild.pants)
  Using cached pyparsing-2.1.10-py2.py3-none-any.whl
Collecting twitter.common.options==0.3.9 (from twitter.common.log==0.3.9->twitter.common.confluence<0.4,>=0.3.1->pantsbuild.pants)
Installing collected packages: twitter.common.lang, twitter.common.collections, six, futures, docutils, twitter.common.options, twitter.common.dirutil, twitter.common.log, twitter.common.confluence, py, pytest, Markdown, ansicolors, coverage, cov-core, pytest-cov, pycparser, cffi, requests, pywatchman, pex, setproctitle, lmdb, pystache, monotonic, fasteners, psutil, pathspec, pyparsing, packaging, Pygments, scandir, pantsbuild.pants
Successfully installed Markdown-2.1.1 Pygments-1.4 ansicolors-1.0.2 cffi-1.7.0 cov-core-1.15.0 coverage-3.7.1 docutils-0.12 fasteners-0.14.1 futures-3.0.5 lmdb-0.89 monotonic-1.2 packaging-16.7 pantsbuild.pants-1.2.1 pathspec-0.3.4 pex-1.1.13 psutil-4.3.0 py-1.4.32 pycparser-2.17 pyparsing-2.1.10 pystache-0.5.3 pytest-2.6.4 pytest-cov-1.8.1 pywatchman-1.3.0 requests-2.5.3 scandir-1.2 setproctitle-1.1.10 six-1.10.0 twitter.common.collections-0.3.9 twitter.common.confluence-0.3.9 twitter.common.dirutil-0.3.9 twitter.common.lang-0.3.9 twitter.common.log-0.3.9 twitter.common.options-0.3.9
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/lmdb/__pycache__/lmdb_cffi.c:210:14: fatal error:
      'lmdb.h' file not found
    #include "lmdb.h"
             ^
1 error generated.
Exception caught: (<class 'cffi.ffiplatform.VerificationError'>)
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 44, in main
    PantsRunner(exiter).run()
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 57, in run
    options_bootstrapper=options_bootstrapper)
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 44, in _run
    from pants.bin.local_pants_runner import LocalPantsRunner
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 11, in <module>
    from pants.bin.goal_runner import GoalRunner
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/goal_runner.py", line 14, in <module>
    from pants.bin.engine_initializer import EngineInitializer
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/bin/engine_initializer.py", line 14, in <module>
    from pants.engine.engine import LocalSerialEngine
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/engine/engine.py", line 23, in <module>
    from pants.engine.storage import Cache, Storage
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/pants/engine/storage.py", line 18, in <module>
    import lmdb
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/lmdb/__init__.py", line 40, in <module>
    from lmdb.cffi import *
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/lmdb/cffi.py", line 362, in <module>
    library_dirs=_config_vars['extra_library_dirs'])
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/cffi/api.py", line 437, in verify
    lib = self.verifier.load_library()
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/cffi/verifier.py", line 113, in load_library
    self._compile_module()
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/cffi/verifier.py", line 210, in _compile_module
    outputfilename = ffiplatform.compile(tmpdir, self.get_extension())
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/cffi/ffiplatform.py", line 29, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose)
  File "/Users/frost/.cache/pants/setup/bootstrap-Darwin-x86_64/unspecified/lib/python2.7/site-packages/cffi/ffiplatform.py", line 62, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))

Exception message: CompileError: command 'clang' failed with exit status 1

19:01 scorp10 pants-test$ python -V
Python 2.7.13

[KLFrost]

i would blame lmdb, but

19:04 scorp10 pants-test$ pip install "lmdb==0.89"
Collecting lmdb==0.89
Installing collected packages: lmdb
Successfully installed lmdb-0.89

[kirklg]

Its possible you haven't installed the lmdb header files though.

On my ubuntu machine:

$ pip install lmdb
$ sudo updatedb
$ locate lmdb.h
...
This yields no results, so the lmdb py package doesn't come with these header files.

$ apt-cache search lmdb
golang-github-boltdb-bolt-dev - low-level key/value database for Go
libcatalyst-model-dbic-schema-perl - DBIx::Class::Schema-based database model for Catalyst
liblmdb-dev - Lightning Memory-Mapped Database development files
liblmdb0 - Lightning Memory-Mapped Database shared library
lmdb-doc - Lightning Memory-Mapped Database doxygen documentation
lmdb-utils - Lightning Memory-Mapped Database Utilities
postfix-lmdb - LMDB map support for Postfix
python-lmdb - Python binding for LMDB Lightning Memory-Mapped Database
python3-lmdb - Python 3 binding for LMDB Lightning Memory-Mapped Database
lmdb-dbg - Lightning Memory-Mapped Database debug files
pdns-backend-lmdb - lmdb backend for PowerDNS

$ sudo apt-get install liblmdb-dev
...

$ dpkg -L liblmdb-dev
/.
/usr
/usr/include
/usr/include/lmdb.h
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/liblmdb.a
/usr/lib/x86_64-linux-gnu/pkgconfig
/usr/lib/x86_64-linux-gnu/pkgconfig/lmdb.pc
/usr/share
/usr/share/doc
/usr/share/doc/liblmdb-dev
/usr/share/doc/liblmdb-dev/changelog.Debian.gz
/usr/share/doc/liblmdb-dev/copyright
/usr/lib/x86_64-linux-gnu/liblmdb.so

I think you just have to find the equivalent package for macOS and install those development files, and try again.

[KLFrost]

actually, that's not the solution... but the log above provides a clue with the various Using cached messages. Turns out I must have a corrupted pip cache somewhere (finding where it is on my version of macOS still a mystery), because replacing pip install with pip install --no-cache-dir in the pants script causes the bootstrap to succeed. Thanks!

[KLFrost]

for the benefit of anyone else running into this kind of problem on MacOS: there are two caches which may affect your pants bootstrap. To flush them both,

rm -rf ~/.cache/pants
rm -rf ~/Library/Caches/pip