Fasttext: Symbol not found errors on OS X with python bindings

[jfelectron]

Tried building from master with both gcc5 and gcc7.

----> 2 from fastText import load_model
3 from config.mongo_settings import COLLECTIONS
4 from utils.text import normalize_fasttext_label
5 from utils.data_generators import mongo_generator

/usr/local/lib/python2.7/site-packages/fastTextpy-0.0.3-py2.7-macosx-10.11-x86_64.egg/fastText/__init__.py in ()
11 from __future__ import unicode_literals
12
---> 13 from .FastText import train_supervised
14 from .FastText import train_unsupervised
15 from .FastText import load_model

/usr/local/lib/python2.7/site-packages/fastTextpy-0.0.3-py2.7-macosx-10.11-x86_64.egg/fastText/FastText.py in ()
11 from __future__ import unicode_literals
12
---> 13 import fasttext_pybind as fasttext
14 import numpy as np
15

ImportError: dlopen(/usr/local/lib/python2.7/site-packages/fastTextpy-0.0.3-py2.7-macosx-10.11-x86_64.egg/fasttext_pybind.so, 2): Symbol not found: __ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev
Referenced from: /usr/local/lib/python2.7/site-packages/fastTextpy-0.0.3-py2.7-macosx-10.11-x86_64.egg/fasttext_pybind.so
Expected in: flat namespace
in /usr/local/lib/python2.7/site-packages/fastTextpy-0.0.3-py2.7-macosx-10.11-x86_64.egg/fasttext_pybind.so

[cpuhrsch]

Hello @jfelectron,

We tried reproducing this issue with Python2 and 3 on macosx 10.13. Can you try to uninstall and reinstall this?

Thanks,
Christian

[kartikrao]

I'm having the same issue, I've tried g++, gcc7 and apple bundled gcc (llvm)

They all result in a dlopen symbol not found error upon importing.

Could it be a conflict between libc++ and libstdc++ ?

My environment

Mac OS High Sierra 10.13.1
Python 3.6 (Anaconda 4.3.30)

Installer output with g++

running install
running bdist_egg
running egg_info
writing python/fasttext.egg-info/PKG-INFO
writing dependency_links to python/fasttext.egg-info/dependency_links.txt
writing requirements to python/fasttext.egg-info/requires.txt
writing top-level names to python/fasttext.egg-info/top_level.txt
reading manifest file 'python/fasttext.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'python/fasttext.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.7-x86_64/egg
running install_lib
running build_py
running build_ext
g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include/python3.6m -c /var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmpqiq_akp4.cpp -o var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmpqiq_akp4.o -stdlib=libc++
g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include/python3.6m -c /var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmp0mp2gla2.cpp -o var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmp0mp2gla2.o -std=c++14
g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include -arch x86_64 -I/Users/kartik/anaconda3/envs/py36/include/python3.6m -c /var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmpw2tza752.cpp -o var/folders/8g/n4syxmpn1vl6f5kcmx3hwvx40000gp/T/tmpw2tza752.o -fvisibility=hidden
creating build/bdist.macosx-10.7-x86_64/egg
copying build/lib.macosx-10.7-x86_64-3.6/fasttext_pybind.cpython-36m-darwin.so -> build/bdist.macosx-10.7-x86_64/egg
creating build/bdist.macosx-10.7-x86_64/egg/fastText
creating build/bdist.macosx-10.7-x86_64/egg/fastText/util
copying build/lib.macosx-10.7-x86_64-3.6/fastText/util/util.py -> build/bdist.macosx-10.7-x86_64/egg/fastText/util
copying build/lib.macosx-10.7-x86_64-3.6/fastText/util/__init__.py -> build/bdist.macosx-10.7-x86_64/egg/fastText/util
creating build/bdist.macosx-10.7-x86_64/egg/fastText/tests
copying build/lib.macosx-10.7-x86_64-3.6/fastText/tests/test_script.py -> build/bdist.macosx-10.7-x86_64/egg/fastText/tests
copying build/lib.macosx-10.7-x86_64-3.6/fastText/tests/__init__.py -> build/bdist.macosx-10.7-x86_64/egg/fastText/tests
copying build/lib.macosx-10.7-x86_64-3.6/fastText/tests/test_configurations.py -> build/bdist.macosx-10.7-x86_64/egg/fastText/tests
copying build/lib.macosx-10.7-x86_64-3.6/fastText/__init__.py -> build/bdist.macosx-10.7-x86_64/egg/fastText
copying build/lib.macosx-10.7-x86_64-3.6/fastText/FastText.py -> build/bdist.macosx-10.7-x86_64/egg/fastText
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/util/util.py to util.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/util/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/tests/test_script.py to test_script.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/tests/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/tests/test_configurations.py to test_configurations.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fastText/FastText.py to FastText.cpython-36.pyc
creating stub loader for fasttext_pybind.cpython-36m-darwin.so
byte-compiling build/bdist.macosx-10.7-x86_64/egg/fasttext_pybind.py to fasttext_pybind.cpython-36.pyc
creating build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/PKG-INFO -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/SOURCES.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/dependency_links.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/not-zip-safe -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/requires.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
copying python/fasttext.egg-info/top_level.txt -> build/bdist.macosx-10.7-x86_64/egg/EGG-INFO
writing build/bdist.macosx-10.7-x86_64/egg/EGG-INFO/native_libs.txt
creating 'dist/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg' and adding 'build/bdist.macosx-10.7-x86_64/egg' to it
removing 'build/bdist.macosx-10.7-x86_64/egg' (and everything under it)
Processing fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg
creating /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg
Extracting fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg to /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages
Adding fasttext 0.0.6 to easy-install.pth file

Installed /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg
Processing dependencies for fasttext==0.0.6
Searching for numpy==1.13.3
Best match: numpy 1.13.3
Adding numpy 1.13.3 to easy-install.pth file

Using /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages
Searching for setuptools==36.5.0.post20170921
Best match: setuptools 36.5.0.post20170921
Adding setuptools 36.5.0.post20170921 to easy-install.pth file
Installing easy_install script to /Users/kartik/anaconda3/envs/py36/bin
Installing easy_install-3.6 script to /Users/kartik/anaconda3/envs/py36/bin

Using /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages
Searching for pybind11==2.2.1
Best match: pybind11 2.2.1
Adding pybind11 2.2.1 to easy-install.pth file

Using /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages
Finished processing dependencies for fasttext==0.0.6

And the Symbol not found error upon importing

Python 3.6.3 |Anaconda, Inc.| (default, Nov  8 2017, 18:10:31)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import fastText
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg/fastText/__init__.py", line 13, in <module>
    from .FastText import train_supervised
  File "/Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg/fastText/FastText.py", line 13, in <module>
    import fasttext_pybind as fasttext
ImportError: dlopen(/Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg/fasttext_pybind.cpython-36m-darwin.so, 2): Symbol not found: __ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEED1Ev
  Referenced from: /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg/fasttext_pybind.cpython-36m-darwin.so
  Expected in: flat namespace
 in /Users/kartik/anaconda3/envs/py36/lib/python3.6/site-packages/fasttext-0.0.6-py3.6-macosx-10.7-x86_64.egg/fasttext_pybind.cpython-36m-darwin.so
>>>

[kartikrao]

After some more testing (Mac OS High Sierra 10.13.1, g++7.2):

• System python (2.7): Installation is successful and can import
• Conda py27: Installation is successful and can import
• Conda py35: Installation is successful and can import
• Conda py36: Installation is successful but cannot import (Symbol not found)

So for me, the issue only manifests when using python 3.6

[cpuhrsch]

Hello @kartikrao,

Thanks for looking into this! I'll try to reproduce this with this specific version of Python. I've not been able to reproduce with this with Linux, so this must be specific to Mac.

Thanks,
Christian

[ezyang]

OK, here is some information that might be useful for us, and also some things for you to try:

Things to tell us:

1. Output of conda list
2. which g++
3. g++ --version

Things to try:

1. Make a fresh conda py36 environment, DON'T install anything in it, and then attempt to install and use fastText
2. Uninstall fastText with pip uninstall fasttext, set CC=clang CXX=clang environment variables before running python setup.py install, and give us the entire log (including all commands run) for when you run it this way.

BTW, if you are using conda install gcc, this DOES NOT WORK with the latest conda distribution, and it is not surprising that there are problems.

[cpuhrsch]

Hello @kartikrao, @jfelectron,

Could you please try again using the latest version of fastText?

Thank you,
Christian

[kartikrao]

Hello @cpuhrsch,
I've been able to install and use v0.8.22 without issues, many thanks for fixing this.

Cheers

[nazariyv]

I am still experiencing this issue. mac OS 10.14, here is my python env:

Python 3.6.7 |Anaconda custom (64-bit)| (default, Oct 23 2018, 14:01:38)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

and here is the issue: https://github.com/vrasneur/pyfasttext/issues/46

[eugene-yang]

I am solving the same problem in other project and came across with this.
Based on the investigation @kartikrao put together, I think it is a Cython issue cython/cython#1880.
Under OSX, they only tested up to Python 3.5.1.

Downgrading to that might help.