Homebrew-core: Python problem on 10.12 systems without CLT

[zmwangx]

Multiple users have reported Python failing to import _sqlite3 on 10.12, and reported success after installing CLT. python.rb also prevents users from pouring the bottle when CLT is not installed. (Homebrew/brew#1171, Homebrew/brew#1179, and https://github.com/Homebrew/homebrew-core/issues/1957#issuecomment-248053191 which I just remembered.) Unfortunately existing CLT is overwritten by macOS upgrade and people upgrade from El Capitan may not realize.

I wonder what's the best action we could take to prevent more users from wasting time on Python due to not having CLT.

[MikeMcQuaid]

I wonder what's the best action we could take to prevent more users from wasting time on Python due to not having CLT.

Probably a brew doctor check that warns if you have Python installed but not the CLT.

[tdsmith]

Ideally we fix the sqlite module build! Working on it now.

[tdsmith]

I postulate this is arising from a mismatch between the sqlite include and lib paths that cpython discovers. The relevant code in the cpython distribution hasn't changed since 2013 so I wonder if either a) there's a new incompatibility between Homebrew sqlite3 and Apple's sqlite3 and this has actually always been sorta broken or b) there's a superenv regression.

[tdsmith]

Linking fails with:

*** WARNING: renaming "_sqlite3" since importing it failed: dlopen(build/lib.macosx-10.12-x86_64-2.7/_sqlite3.so, 2): Symbol not found: _sqlite3_enable_load_extension
  Referenced from: build/lib.macosx-10.12-x86_64-2.7/_sqlite3.so
  Expected in: flat namespace
 in build/lib.macosx-10.12-x86_64-2.7/_sqlite3.so

Indeed, the relevant symbol is present in Homebrew's sqlite but not Apple's sqlite:

tim@rocketman:cpython (master)$ nm $(brew --prefix sqlite)/lib/libsqlite3.dylib | grep sqlite3_enable_load_extension
000000000000d364 T _sqlite3_enable_load_extension
tim@rocketman:cpython (master)$ nm /usr/lib/libsqlite3.dylib | grep sqlite3_enable_load_extension
tim@rocketman:cpython (master)$ 

Fix is:

diff --git a/Formula/python.rb b/Formula/python.rb
index daed9bd..970d862 100644
--- a/Formula/python.rb
+++ b/Formula/python.rb
@@ -158,10 +158,16 @@ class Python < Formula
       args << "--enable-universalsdk=/" << "--with-universal-archs=intel"
     end

-    # Allow sqlite3 module to load extensions:
-    # https://docs.python.org/library/sqlite3.html#f1
     if build.with? "sqlite"
-      inreplace("setup.py", 'sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))', "")
+      inreplace "setup.py" do |s|
+        s.gsub! "sqlite_setup_debug = False", "sqlite_setup_debug = True"
+        s.gsub! "for d_ in inc_dirs + sqlite_inc_paths:",
+                "for d_ in ['#{Formula["sqlite"].opt_include}']:"
+
+        # Allow sqlite3 module to load extensions:
+        # https://docs.python.org/library/sqlite3.html#f1
+        s.gsub! 'sqlite_defines.append(("SQLITE_OMIT_LOAD_EXTENSION", "1"))', ""
+      end
     end

     # Allow python modules to use ctypes.find_library to find homebrew's stuff

[tdsmith]

The reason I think this might be a superenv regression is that the link invocation looks like:

clang called with: -bundle -undefined dynamic_lookup -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/cache.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/connection.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/cursor.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/microprotocols.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/module.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/prepare_protocol.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/row.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/statement.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/util.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -L/usr/local/lib -lsqlite3 -o build/lib.macosx-10.12-x86_64-2.7/_sqlite3.so -Wl,-search_paths_first
superenv added:    -pipe -w -Os -march=native --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -isystem/usr/local/include -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/libxml2 -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/apache2 -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/openssl/include -L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib -L/usr/local/opt/openssl/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries -Wl,-headerpad_max_install_names
superenv executed: clang -pipe -w -Os -march=native -bundle -undefined dynamic_lookup build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/cache.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/connection.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/cursor.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/microprotocols.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/module.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/prepare_protocol.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/row.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/statement.o build/temp.macosx-10.12-x86_64-2.7/private/tmp/python-20160929-3593-1t8z1de/Python-2.7.12/Modules/_sqlite/util.o -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -lsqlite3 -o build/lib.macosx-10.12-x86_64-2.7/_sqlite3.so -Wl,-search_paths_first -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk --sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk -isystem/usr/local/include -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/libxml2 -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/apache2 -isystem/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Headers -I/usr/local/opt/readline/include -I/usr/local/opt/sqlite/include -I/usr/local/opt/openssl/include -L/usr/local/opt/readline/lib -L/usr/local/opt/sqlite/lib -L/usr/local/opt/openssl/lib -L/usr/local/lib -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/OpenGL.framework/Versions/Current/Libraries -Wl,-headerpad_max_install_names

I think the -L/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/lib -lsqlite3 early in the linker invocation was causing the linker to find the wrong library and wonder if that -L... shouldn't have been stripped.