Allennlp: Installing allennlp through pip using conda virtual environment fails

Created on 1 Feb 2019 · 32Comments · Source: allenai/allennlp

Describe the bug

I am trying to install allennlp through pip under conda virtual environment, however it fails and leave error message like this:

Building wheels for collected packages: overrides, jsonnet, nltk, parsimonious, numpydoc, msgpack, regex, ujson, dill, jsondiff, PyYAML, wrapt, cytoolz, future, toolz
  Running setup.py bdist_wheel for overrides ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/f7/27/b8/b4f46c59426a11e7f2d4e472b870ec14c21b4beab2e1afa725
  Running setup.py bdist_wheel for jsonnet ... error
  Complete output from command /home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/jsonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-t66ososi --python-tag cp37:
  running bdist_wheel
  running build
  running build_ext
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/desugarer.cpp -o core/desugarer.o
  core/desugarer.cpp: In member function ‘void Desugarer::desugar(AST*&, unsigned int)’:
  core/desugarer.cpp:612:51: warning: this statement may fall through [-Wimplicit-fallthrough=]
                   case BOP_MANIFEST_UNEQUAL: invert = true;
                                              ~~~~~~~^~~~~~
  core/desugarer.cpp:613:17: note: here
                   case BOP_MANIFEST_EQUAL: {
                   ^~~~
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/formatter.cpp -o core/formatter.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/libjsonnet.cpp -o core/libjsonnet.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/lexer.cpp -o core/lexer.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/parser.cpp -o core/parser.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/pass.cpp -o core/pass.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/static_analysis.cpp -o core/static_analysis.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/string_utils.cpp -o core/string_utils.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 core/vm.cpp -o core/vm.o
  g++ -c -g -O3 -Wall -Wextra -Woverloaded-virtual -pedantic -std=c++0x -fPIC -Iinclude -Ithird_party/md5 third_party/md5/md5.cpp -o third_party/md5/md5.o
  building '_jsonnet' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/python
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Iinclude -Ithird_party/md5 -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c python/_jsonnet.c -o build/temp.linux-x86_64-3.7/python/_jsonnet.o
  python/_jsonnet.c: In function ‘cpython_native_callback’:
  python/_jsonnet.c:147:19: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘const long unsigned int’} [-Wsign-compare]
       for (i = 0; i < ctx->argc; ++i) {
                     ^
  creating build/lib.linux-x86_64-3.7
  g++ -pthread -shared -B /home/ichn/anaconda3/envs/torch/compiler_compat -L/home/ichn/anaconda3/envs/torch/lib -Wl,-rpath=/home/ichn/anaconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/python/_jsonnet.o core/desugarer.o core/formatter.o core/libjsonnet.o core/lexer.o core/parser.o core/pass.o core/static_analysis.o core/string_utils.o core/vm.o third_party/md5/md5.o -o build/lib.linux-x86_64-3.7/_jsonnet.cpython-37m-x86_64-linux-gnu.so
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
  build/temp.linux-x86_64-3.7/python/_jsonnet.o: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status
  error: command 'g++' failed with exit status 1

  ----------------------------------------
  Failed building wheel for jsonnet
  Running setup.py clean for jsonnet
  Running setup.py bdist_wheel for nltk ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/f1/98/72/c2ba4734bc46df30b9c3bd3eb037c52ab8ae0110f8fa15200a
  Running setup.py bdist_wheel for parsimonious ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/f1/a4/4b/7cac60fa74b7c16017cd9c67ab65736d3d9318064ae65e0ee0
  Running setup.py bdist_wheel for numpydoc ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/11/76/d4/16c19c2378616c3389916bc6d7b1134b72bfe6f7abd9f80243
  Running setup.py bdist_wheel for msgpack ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/3f/78/5a/92a8797deabe61189baf597a855e9529f6b20a391d9924d968
  Running setup.py bdist_wheel for regex ... error
  Complete output from command /home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/regex/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-6v75m_hn --python-tag cp37:
  /home/ichn/anaconda3/envs/torch/lib/python3.7/site-packages/setuptools/dist.py:470: UserWarning: Normalizing '2018.01.10' to '2018.1.10'
    normalized_version,
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  copying regex_3/regex.py -> build/lib.linux-x86_64-3.7
  copying regex_3/_regex_core.py -> build/lib.linux-x86_64-3.7
  copying regex_3/test_regex.py -> build/lib.linux-x86_64-3.7
  running build_ext
  building '_regex' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/regex_3
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c regex_3/_regex.c -o build/temp.linux-x86_64-3.7/regex_3/_regex.o
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c regex_3/_regex_unicode.c -o build/temp.linux-x86_64-3.7/regex_3/_regex_unicode.o
  gcc -pthread -shared -B /home/ichn/anaconda3/envs/torch/compiler_compat -L/home/ichn/anaconda3/envs/torch/lib -Wl,-rpath=/home/ichn/anaconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/regex_3/_regex.o build/temp.linux-x86_64-3.7/regex_3/_regex_unicode.o -o build/lib.linux-x86_64-3.7/_regex.cpython-37m-x86_64-linux-gnu.so
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/regex_3/_regex.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/regex_3/_regex.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/regex_3/_regex.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/regex_3/_regex.o: unable to initialize decompress status for section .debug_info
  build/temp.linux-x86_64-3.7/regex_3/_regex.o: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for regex
  Running setup.py clean for regex
  Running setup.py bdist_wheel for ujson ... error
  Complete output from command /home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/ujson/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-97buh6vk --python-tag cp37:
  Warning: 'classifiers' should be a list, got type 'filter'
  running bdist_wheel
  running build
  running build_ext
  building 'ujson' extension
  creating build
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/python
  creating build/temp.linux-x86_64-3.7/lib
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c ./python/ujson.c -o build/temp.linux-x86_64-3.7/./python/ujson.o -D_GNU_SOURCE
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c ./python/objToJSON.c -o build/temp.linux-x86_64-3.7/./python/objToJSON.o -D_GNU_SOURCE
  ./python/objToJSON.c: In function ‘PyUnicodeToUTF8’:
  ./python/objToJSON.c:154:18: warning: initialization discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
       char *data = PyUnicode_AsUTF8AndSize(obj, &len);
                    ^~~~~~~~~~~~~~~~~~~~~~~
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c ./python/JSONtoObj.c -o build/temp.linux-x86_64-3.7/./python/JSONtoObj.o -D_GNU_SOURCE
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c ./lib/ultrajsonenc.c -o build/temp.linux-x86_64-3.7/./lib/ultrajsonenc.o -D_GNU_SOURCE
  ./lib/ultrajsonenc.c:156:23: warning: ‘g_hexChars’ is static but used in inline function ‘Buffer_AppendShortHexUnchecked’ which is not static
     *(outputOffset++) = g_hexChars[(value & 0x000f) >> 0];
                         ^~~~~~~~~~
  ./lib/ultrajsonenc.c:155:23: warning: ‘g_hexChars’ is static but used in inline function ‘Buffer_AppendShortHexUnchecked’ which is not static
     *(outputOffset++) = g_hexChars[(value & 0x00f0) >> 4];
                         ^~~~~~~~~~
  ./lib/ultrajsonenc.c:154:23: warning: ‘g_hexChars’ is static but used in inline function ‘Buffer_AppendShortHexUnchecked’ which is not static
     *(outputOffset++) = g_hexChars[(value & 0x0f00) >> 8];
                         ^~~~~~~~~~
  ./lib/ultrajsonenc.c:153:23: warning: ‘g_hexChars’ is static but used in inline function ‘Buffer_AppendShortHexUnchecked’ which is not static
     *(outputOffset++) = g_hexChars[(value & 0xf000) >> 12];
                         ^~~~~~~~~~
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./python -I./lib -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c ./lib/ultrajsondec.c -o build/temp.linux-x86_64-3.7/./lib/ultrajsondec.o -D_GNU_SOURCE
  creating build/lib.linux-x86_64-3.7
  gcc -pthread -shared -B /home/ichn/anaconda3/envs/torch/compiler_compat -L/home/ichn/anaconda3/envs/torch/lib -Wl,-rpath=/home/ichn/anaconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/./python/ujson.o build/temp.linux-x86_64-3.7/./python/objToJSON.o build/temp.linux-x86_64-3.7/./python/JSONtoObj.o build/temp.linux-x86_64-3.7/./lib/ultrajsonenc.o build/temp.linux-x86_64-3.7/./lib/ultrajsondec.o -o build/lib.linux-x86_64-3.7/ujson.cpython-37m-x86_64-linux-gnu.so
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/./python/ujson.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/./python/ujson.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/./python/ujson.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/./python/ujson.o: unable to initialize decompress status for section .debug_info
  build/temp.linux-x86_64-3.7/./python/ujson.o: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for ujson
  Running setup.py clean for ujson
  Running setup.py bdist_wheel for dill ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/f6/d1/a7/c90dbb9c5613295c70d96d60e78c3e2b283143fddbbd57e14d
  Running setup.py bdist_wheel for jsondiff ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/8f/c9/36/f9e8aea16af567ce91abbe6b8b6b650877b9e17ce8aa97fb42
  Running setup.py bdist_wheel for PyYAML ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/11/c5/f8/4e054145468ca00fd2ab4a6c20bf7e09ec57b879572c865ee6
  Running setup.py bdist_wheel for wrapt ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/10/a6/59/eab55ff1e60d10ca0404baf6e7b8baf52908091133608bf289
  Running setup.py bdist_wheel for cytoolz ... error
  Complete output from command /home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/cytoolz/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-ip_zgny_ --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/_signatures.py -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/__init__.py -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/utils_test.py -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/_version.py -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/compatibility.py -> build/lib.linux-x86_64-3.7/cytoolz
  creating build/lib.linux-x86_64-3.7/cytoolz/curried
  copying cytoolz/curried/operator.py -> build/lib.linux-x86_64-3.7/cytoolz/curried
  copying cytoolz/curried/__init__.py -> build/lib.linux-x86_64-3.7/cytoolz/curried
  copying cytoolz/curried/exceptions.py -> build/lib.linux-x86_64-3.7/cytoolz/curried
  copying cytoolz/dicttoolz.pyx -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/itertoolz.pyx -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/utils.pyx -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/recipes.pyx -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/functoolz.pyx -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/dicttoolz.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/__init__.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/recipes.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/utils.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/functoolz.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/itertoolz.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  copying cytoolz/cpython.pxd -> build/lib.linux-x86_64-3.7/cytoolz
  creating build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_none_safe.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_recipes.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_curried.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_tlz.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_itertoolz.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_functoolz.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/dev_skip_test.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_embedded_sigs.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_utils.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_docstrings.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_inspect_args.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_doctests.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_curried_toolzlike.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_serialization.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_compatibility.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_signatures.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_dev_skip_test.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  copying cytoolz/tests/test_dicttoolz.py -> build/lib.linux-x86_64-3.7/cytoolz/tests
  running build_ext
  building 'cytoolz.dicttoolz' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/cytoolz
  gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c cytoolz/dicttoolz.c -o build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o
  gcc -pthread -shared -B /home/ichn/anaconda3/envs/torch/compiler_compat -L/home/ichn/anaconda3/envs/torch/lib -Wl,-rpath=/home/ichn/anaconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o -o build/lib.linux-x86_64-3.7/cytoolz/dicttoolz.cpython-37m-x86_64-linux-gnu.so
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o: unable to initialize decompress status for section .debug_info
  /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o: unable to initialize decompress status for section .debug_info
  build/temp.linux-x86_64-3.7/cytoolz/dicttoolz.o: file not recognized: file format not recognized
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for cytoolz
  Running setup.py clean for cytoolz
  Running setup.py bdist_wheel for future ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/3f/66/fe/9c4fd5c707a9f26993ba157f0752d84d5c7e26aedbefb84f76
  Running setup.py bdist_wheel for toolz ... done
  Stored in directory: /home/ichn/.cache/pip/wheels/73/ad/e1/f8fe78eeb9e2b31ea8396419d92adc107c553ff7eb47ad12d9
Successfully built overrides nltk parsimonious numpydoc msgpack dill jsondiff PyYAML wrapt future toolz
Failed to build jsonnet regex ujson cytoolz
Installing collected packages: overrides, jsonnet, wcwidth, ftfy, singledispatch, nltk, regex, murmurhash, ujson, cymem, dill, idna, chardet, urllib3, requests, msgpack, msgpack-numpy, tqdm, wrapt, preshed, toolz, cytoolz, plac, thinc, spacy, sqlparse, itsdangerous, click, werkzeug, MarkupSafe, Jinja2, flask, flask-cors, editdistance, flaky, cycler, pytz, kiwisolver, python-dateutil, pyparsing, matplotlib, greenlet, gevent, atomicwrites, py, pluggy, attrs, more-itertools, pytest, responses, jsonpickle, aws-xray-sdk, cookies, xmltodict, pbr, mock, jsondiff, jmespath, docutils, botocore, s3transfer, boto3, PyYAML, pyaml, boto, websocket-client, docker-pycreds, docker, asn1crypto, cryptography, future, ecdsa, pycryptodome, python-jose, moto, parsimonious, Pygments, alabaster, sphinxcontrib-websupport, imagesize, snowballstemmer, babel, packaging, sphinx, numpydoc, protobuf, tensorboardX, h5py, conllu, scipy, scikit-learn, unidecode, pytorch-pretrained-bert, colorama, pyasn1, rsa, awscli, allennlp
  Running setup.py install for jsonnet ... error
    Complete output from command /home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/jsonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-3ccckdjt/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_ext
    make: 'core/desugarer.o' is up to date.
    make: 'core/formatter.o' is up to date.
    make: 'core/libjsonnet.o' is up to date.
    make: 'core/lexer.o' is up to date.
    make: 'core/parser.o' is up to date.
    make: 'core/pass.o' is up to date.
    make: 'core/static_analysis.o' is up to date.
    make: 'core/string_utils.o' is up to date.
    make: 'core/vm.o' is up to date.
    make: 'third_party/md5/md5.o' is up to date.
    building '_jsonnet' extension
    creating build
    creating build/temp.linux-x86_64-3.7
    creating build/temp.linux-x86_64-3.7/python
    gcc -pthread -B /home/ichn/anaconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Iinclude -Ithird_party/md5 -I/home/ichn/anaconda3/envs/torch/include/python3.7m -c python/_jsonnet.c -o build/temp.linux-x86_64-3.7/python/_jsonnet.o
    python/_jsonnet.c: In function ‘cpython_native_callback’:
    python/_jsonnet.c:147:19: warning: comparison of integer expressions of different signedness: ‘int’ and ‘size_t’ {aka ‘const long unsigned int’} [-Wsign-compare]
         for (i = 0; i < ctx->argc; ++i) {
                       ^
    creating build/lib.linux-x86_64-3.7
    g++ -pthread -shared -B /home/ichn/anaconda3/envs/torch/compiler_compat -L/home/ichn/anaconda3/envs/torch/lib -Wl,-rpath=/home/ichn/anaconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/python/_jsonnet.o core/desugarer.o core/formatter.o core/libjsonnet.o core/lexer.o core/parser.o core/pass.o core/static_analysis.o core/string_utils.o core/vm.o third_party/md5/md5.o -o build/lib.linux-x86_64-3.7/_jsonnet.cpython-37m-x86_64-linux-gnu.so
    /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
    /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
    /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
    /home/ichn/anaconda3/envs/torch/compiler_compat/ld: build/temp.linux-x86_64-3.7/python/_jsonnet.o: unable to initialize decompress status for section .debug_info
    build/temp.linux-x86_64-3.7/python/_jsonnet.o: file not recognized: file format not recognized
    collect2: error: ld returned 1 exit status
    error: command 'g++' failed with exit status 1

    ----------------------------------------
Command "/home/ichn/anaconda3/envs/torch/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-grk1qblh/jsonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-3ccckdjt/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-grk1qblh/jsonnet/

To Reproduce

I am using conda 5.3.1 under archlinux with pytorch==1.0.0 preinstalled as part of the environment, and gcc of version

(torch) ➜  ~ g++ --version
g++ (GCC) 8.2.1 20181127
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

also tried install gcc through conda and then rerun

pip install allennlp

but fails as well.

Expected behavior

allennlp successfully installed

System (please complete the following information):

Linux
Python version: 3.7.2
AllenNLP version: I installed from master
PyTorch version: 1.0.0

Source

ichn-hu

Most helpful comment

Running into exactly the same issue on Manjaro Linux. This seems to be a compatibility issue with anaconda's linker (see https://github.com/cython/cython/issues/2824) on Arch...

I got it working by renaming anaconda's ld to ld_ (following instructions here: https://github.com/pytorch/pytorch/issues/16683) so the packages are linked with the OS's ld. Ugly solution, though.

kaichengyan on 17 Feb 2019

🚀2 ❤2 👍2 👀1 🎉1

All 32 comments

I ran this on a mac, but I wasn't able to reproduce your issue:

$ conda create -n allennlp-3.7 python-3.7.2
$ conda activate allennlp-3.7
$ pip install allennlp
$ allennlp test-install

Our CI runs pip installs for each commit, although on an ubuntu instance and with Python 3.6.

schmmd on 5 Feb 2019

Not sure if this helps:

$ gcc --version                                                                                                                                                                                                                                                                                       (allennlp-3.7) 
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.1.0 (clang-902.0.39.2)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I won't be able to troubleshoot the GCC problems you're having. Maybe you could try using our Docker image as a workaround?

schmmd on 6 Feb 2019

Running into exactly the same issue on Manjaro Linux. This seems to be a compatibility issue with anaconda's linker (see https://github.com/cython/cython/issues/2824) on Arch...

kaichengyan on 17 Feb 2019

🚀2 ❤2 👍2 👀1 🎉1

I am also on Manjaro. My solution for this is to use conda-forge as the source channel of the installation.

ichn-hu on 19 Feb 2019

👍3

Closing, since this has to do with a particular Linux distribution (Manjaro) and we have a workaround (see above two posts). Please comment further if there's a specific action you would like to see us take!

schmmd on 15 Mar 2019

Reference to this issue: https://github.com/scrapy/scrapy/issues/2115

sudo apt install python3.6-dev

This also works for me

allanj on 25 Mar 2019

👍1

This is also an issue on MacOS Mojave. The conda-forge workaround in #1983 does work but downgrades python 3.7 to 3.6 which is a little bit annoying.

HenryDashwood on 27 Mar 2019

Re-opening since this might be a 3.7 issue.

schmmd on 27 Mar 2019

pip install allennlp with python 3.6 does indeed work fine in Mojave.
But it does work with 3.7 when I do it in Ubuntu.

HenryDashwood on 27 Mar 2019

I am facing the same issue. Works fine in 3.6 but fails if i use python 3.7. Tried in multiple distributions, Arch, and Void Linux, so it definitely looks like a python 3.7 issue

vikigenius on 31 Mar 2019

Interesting--we try out our library on python:3.7.2 (debian) for each commit to master.

schmmd on 1 Apr 2019

I have another update re this issue. It fails when installing on an Amazon Linux AMI in both python 3.6 and python 3.7 conda environments.

HenryDashwood on 1 Apr 2019

I just started a brand new EC2 instance running the Amazon Linux 2 AMI. I was able to successfully install allennlp, here are the commands I ran:

sudo yum update
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
chmod a+x Miniconda3-latest-Linux-x86_64.sh 
./Miniconda3-latest-Linux-x86_64.sh 
bash
conda create -n allennlp python=3.6
conda activate allennlp
sudo yum groupinstall "Development Tools"
pip install allennlp
allennlp train

joelgrus on 1 Apr 2019

on the same machine I was also able to install allennlp in a 3.7 environment:

conda create -n allennlp37 python=3.7
conda activate allennlp37
pip install allennlp
allennlp train

joelgrus on 1 Apr 2019

I'm so sorry Joel, but the error persists!
Here are the specific OS specs, if it helps:

NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"

And here is the final part of the error message:

Command "/home/ec2-user/miniconda3/envs/allennlp37/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-061jz02j/jsonnet/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))"install --record /tmp/pip-record-0mocpfuj/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-install-061jz02j/jsonnet/

The error message also begins with the line Failed building wheel for jsonnet.

HenryDashwood on 1 Apr 2019

ok, try doing gcc --version

on my "Amazon Linux 2" machine, it's 7.3.1. on my "Amazon Linux" machine, it's 7.2.1. so my hypothesis was that jsonnet needs 7.3.1 in order to build correctly. 😬

the question is: how to get the Amazon Linux machine up to 7.3.1. I was able to find these instructions, which worked (but also I have no idea what they're actually doing):

https://gist.github.com/mhart/841afde086604e7b4a86fa140480c84d

after running those, my gcc --version was showing 7.3.1 and I was able to pip install allennlp.

joelgrus on 2 Apr 2019

🚀1

gcc is indeed version 7.2.1 on my AMI:
gcc (GCC) 7.2.1 20170915 (Red Hat 7.2.1-2)

Unfortunately I won't be able to try 7.3.1 today as I don't have root access on this machine. But great debugging nonetheless! I also note that the gcc version in my conda env on macOS is gcc (GCC) 4.8.5

HenryDashwood on 2 Apr 2019

a simpler solution seems to be use the "Amazon Linux 2" AMI, but I get that that's not always an option.

joelgrus on 2 Apr 2019

Absolutely. Many thanks once again!

HenryDashwood on 2 Apr 2019

I have tried to debug this more, and it seems to be an issue with jsonnet and regex packages. Both of them fail to install on conda environements with python 3.7 I followed @kaichengyan 's suggestion and compile it against the system version of gcc and the system's ld. Once Jsonnet and Regex get installed I can just use pip install allennlp

vikigenius on 2 Apr 2019

I ran the gist today and it worked! Much appreciated!

HenryDashwood on 2 Apr 2019

👍1

I just had this issue on a Mac and followed https://stackoverflow.com/questions/52509602/cant-compile-c-program-on-a-mac-after-upgrade-to-mojave to get it to work. Specifically I ran:

https://stackoverflow.com/questions/52509602/cant-compile-c-program-on-a-mac-after-upgrade-to-mojave

schmmd on 11 Apr 2019

How interesting. I updated to Mojave 10.14.4 (18E226) yesterday. I just tried to test this out in a new Conda environment, Python 3.7.3, but it installed perfectly on its own!

HenryDashwood on 11 Apr 2019

I was doing Conda, Python 3.6, on Mojave 10.14.4 (new laptop).

schmmd on 11 Apr 2019

Closing as resolved and dull

DeNeutoy on 27 Apr 2019

I am getting below error in azure Linux environment when try to install allennlp.
Building wheel for jsonnet (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /opt/python/3.6.9/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-aog_zlna/jsonnet/s
etup.py'"'"'; __file__='"'"'/tmp/pip-install-aog_zlna/jsonnet/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read
().replace('"'"'rn'"'"', '"'"'n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ncw4i672 --
python-tag cp36
cwd: /tmp/pip-install-aog_zlna/jsonnet/
Complete output (4 lines):
running bdist_wheel
running build
running build_ext
error: [Errno 2] No such file or directory: 'make': 'make'

ERROR: Failed building wheel for jsonnet

Ravibth on 18 Oct 2019

@Ravibth clearly you need to install make, which is already available on most systems. You might want to take a look at what we install on our base debian system in our Dockerfile. Please open a new issue if you have additional questions.

schmmd on 18 Oct 2019

Hi Michael,

I installed make. But Still I am getting same error.