Essentia: Could not build python extensions

[colonder]

Hi, I'm running Ubuntu 16.04 with Anaconda 3. When I'm trying to configure Essentia, the following error appears:

./waf configure --build-static --with-python --with-cpptests --with-examples --with-vamp
Setting top to                           : /home/xxx/Repositories/essentia 
Setting out to                           : /home/xxx/Repositories/essentia/build 
→ configuring the project in /home/xxx/Repositories/essentia
→ Building in release mode
Checking for 'g++' (C++ compiler)        : /usr/bin/g++ 
Checking for 'gcc' (C compiler)          : /usr/bin/gcc 
Checking for program 'pkg-config'        : /usr/bin/pkg-config 
Checking for 'libavcodec' >= 55.34.1     : yes 
Checking for 'libavformat'               : yes 
Checking for 'libavutil'                 : yes 
Checking for 'libavresample'             : yes 
Checking for 'samplerate'                : yes 
Checking for 'taglib' >= 1.9             : yes 
Checking for 'yaml-0.1'                  : yes 
Checking for 'fftw3f'                    : yes 
Checking for 'libchromaprint'            : yes 
Checking for 'gcc' (C compiler)          : /usr/bin/gcc 
Checking for program 'python'            : /home/shared/anaconda3/bin/python 
Checking for python version >= 2.7.0     : 3.6.4 
→ Configuring for python3
python-config                            : /home/shared/anaconda3/bin/python3.6-config 
Asking python-config for pyext '--cflags --libs --ldflags' flags : yes 
Testing pyext configuration                                      : Could not build python extensions 
The configuration failed
(complete log in /home/xxx/Repositories/essentia/build/config.log)

When I inspect the above file this is what I see

Testing pyext configuration
==>

#include <Python.h>
#ifdef __cplusplus
extern "C" {
#endif
        void Py_Initialize(void);
        void Py_Finalize(void);
#ifdef __cplusplus
}
#endif
int main(int argc, char **argv)
{
   (void)argc; (void)argv;
   Py_Initialize();
   Py_Finalize();
   return 0;
}

<==
[1/2] Compiling ^[[32mbuild/.conf_check_9d38bb485ef1bd6bb20afbc408577ad3/test.cpp^[[0m
['/usr/bin/g++', '-pipe', '-Wall', '-msse', '-msse2', '-mfpmath=sse', '-O2', '-fPIC', '-pthr$
[2/2] Linking ^[[33mbuild/.conf_check_9d38bb485ef1bd6bb20afbc408577ad3/testbuild/testprog.cp$

['/usr/bin/g++', '-pthread', '-shared', '-fPIC', '-Xlinker', '-export-dynamic', 'test.cpp.1.$
err: lto1: fatal error: bytecode stream generated with LTO version 6.0 instead of the expect$
compilation terminated.
lto-wrapper: fatal error: /usr/bin/g++ returned 1 exit status
compilation terminated.
/usr/bin/x86_64-linux-gnu-ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

from /home/xxx/Repositories/essentia/src/python: Test does not build: Traceback (most recen$
  File "/home/xxx/Repositories/essentia/.waf3-1.9.11-af9ceda667eab9a2b3fb893b34583f23/wafli$
    bld.compile()
  File "/home/xxx/Repositories/essentia/.waf3-1.9.11-af9ceda667eab9a2b3fb893b34583f23/wafli$
    raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
 -> task in 'testprog' failed with exit status 1 (run with -v to display more information)

Could not build python extensions
from /home/xxx/Repositories/essentia/src/python: The configuration failed

I checked the gcc and g++ versions and this is what I see

gcc --version
gcc (Ubuntu 6.4.0-17ubuntu1~16.04) 6.4.0 20180424
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

g++ --version
g++ (Ubuntu 6.4.0-17ubuntu1~16.04) 6.4.0 20180424
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[sildeag]

I just ran this test on my repo https://github.com/sildeag/essentia for Ubuntu 18.04
python3 ./waf configure --mode=release --with-gaia --with-gaia-qt5 --with-example=streaming_extractor_music_svm --with-vamp --with-python --build-static

I would not normally use --build-static but it worked. Why do you use it? I have only tested for --with-gaia-qt5 and python 3.6 which is what comes with Ubuntu 17+ not python 3.5. Essentia using Gaia with qt4 needs a little more work on my repo.

There is a current issue on the MTG repo build script src/wscript. See #726

[290/292] Linking build/src/libessentia.a
[291/292] Linking build/src/examples/libvamp_essentia.so
[292/292] Linking build/src/python/_essentia.cpython-36m-x86_64-linux-gnu.so
Waf: Leaving directory/home/xxxxxx/essentia/build'` 'build' finished successfully (6m31.469s)`

No errors.

[colonder]

I would not normall use --build-static but it worked. Why do you use it?

Because it is written in the Compilling Essentia section here http://essentia.upf.edu/documentation/installing.html#compiling-essentia
I've followed this entire page many times before on my Ubuntu 16.04 and it always worked as intended. It is the first time that the command from the link didn't work with python 3. On the other hand I'm neither using --with-gaia nor --with-gaia-qt5 because I'm only interested in the python bindings, not any ready-made predictor. By the way I tried to run command in the link on the Ubuntu 18.04 with Anaconda 3 as well, but it gives me the same error, that is Could not build python extensions, but then the log file says

Testing pyembed configuration
==>

#include <Python.h>
#ifdef __cplusplus
extern "C" {
#endif
        void Py_Initialize(void);
        void Py_Finalize(void);
#ifdef __cplusplus
}
#endif
int main(int argc, char **argv)
{
   (void)argc; (void)argv;
   Py_Initialize();
   Py_Finalize();
   return 0;
}

<==
[1/2] Compiling ^[[32mbuild/.conf_check_cd087e30f350ce90f2ad9ce46f7db1bf/test.cpp^[[0m

['/usr/bin/g++', '-pipe', '-Wall', '-msse', '-msse2', '-mfpmath=sse', '-O2', '-fPIC', '-pthread', '-march=nocona', '-mtune=haswell', '-ftree-vectorize', '-fPIC', '-fstack-protector-strong', '$
[2/2] Linking ^[[33mbuild/.conf_check_cd087e30f350ce90f2ad9ce46f7db1bf/testbuild/testprog^[[0m

['/usr/bin/g++', '-pthread', '-fPIC', '-Xlinker', '-export-dynamic', 'test.cpp.1.o', '-o/home/xxx/Szablony/essentia/build/.conf_check_cd087e30f350ce90f2ad9ce46f7db1bf/testbuild/testprog', '-$
err: lto1: internal compiler error: in lto_tag_to_tree_code, at lto-streamer.h:1005
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-7/README.Bugs> for instructions.
lto-wrapper: fatal error: /usr/bin/g++ returned 1 exit status
compilation terminated.
/usr/bin/x86_64-linux-gnu-ld: error: lto-wrapper failed
collect2: error: ld returned 1 exit status

On Ubuntu 18.04 I have version 7 of both gcc and g++

I have only tested for --with-gaia-qt5 and python 3.6 which is what comes with Ubuntu 17+ not python 3.5

I have python 3.6.4 in my Anaconda, not python 3.5.

[sildeag]

In my view Examples, Python, and Vamp will not currently work on MTG/Essentia due to an issue with src/wscript issue #726. Try your options on my repo, https://github.com/sildeag/essentia, and let me know the results. Your exact options worked on Ubuntu 18.04 without Anaconda but using python 3.6.5.

[dbogdanov]

Hey @colonder. We do not support builds within Anaconda environment, however, we now provide pip packages that you can install within your conda environment. You can try these test builds (run pip install <filename> depending on your python version). Official packages will be available soon on PyPi and we'll then update the installation instructions.

[goodguyben]

Hey @colonder ! Did you get Essentia configured? I am stuck at the same point as you. Would really appreciate some help.

[dbogdanov]

Btw, the official packages are ready. You can pip install essentia on Linux now.

[jarmitage]

I am also trying to install essentia and running into the same issue as @colonder:

Testing pyext configuration
==>

#include <Python.h>
#ifdef __cplusplus
extern "C" {
#endif
    void Py_Initialize(void);
    void Py_Finalize(void);
#ifdef __cplusplus
}
#endif
int main(int argc, char **argv)
{
   (void)argc; (void)argv;
   Py_Initialize();
   Py_Finalize();
   return 0;
}

<==
[1/2] Compiling build/.conf_check_02bb3575a4040bfc70a336934c2010f5/test.cpp

['/Users/xxx/anaconda3/envs/cling/bin/clang++', '-pipe', '-Wall', '-msse', '-msse2', '-mfpmath=sse', '-O2', '-fPIC', '-fPIC', '-g', '-fwrapv', '-O3', '-arch', 'x86_64', '-arch', 'x86_64', '-fPIC', '-I../../../../../../anaconda3/envs/cling/include/python3.6m', '-I../../../../../../anaconda3/envs/cling/include', '-DGTEST_HAS_TR1_TUPLE=0', '-DHAVE_AVCODEC=1', '-DHAVE_AVFORMAT=1', '-DHAVE_AVUTIL=1', '-DHAVE_AVRESAMPLE=1', '-DHAVE_SAMPLERATE=1', '-DHAVE_TAGLIB=1', '-DHAVE_YAML=1', '-DHAVE_FFTW=1', '-D__STDC_CONSTANT_MACROS', '-DPYTHONDIR="/usr/local/lib/python3.6/site-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python3.6/site-packages"', '-DNDEBUG', '../test.cpp', '-c', '-o/Users/xxx/Documents/git/essentia/build/.conf_check_02bb3575a4040bfc70a336934c2010f5/testbuild/test.cpp.1.o', '-fPIC', '-I/usr/local/opt/llvm/include']
[2/2] Linking build/.conf_check_02bb3575a4040bfc70a336934c2010f5/testbuild/testprog.cpython-36m-darwin.so

['/Users/xxx/anaconda3/envs/cling/bin/clang++', '-arch', 'x86_64', '-arch', 'x86_64', '-Wl,-stack_size,1000000', '-bundle', '-undefined', 'dynamic_lookup', 'test.cpp.1.o', '-o/Users/xxx/Documents/git/essentia/build/.conf_check_02bb3575a4040bfc70a336934c2010f5/testbuild/testprog.cpython-36m-darwin.so', '-framework', 'CoreFoundation', '-L/Users/xxx/anaconda3/envs/cling/lib/python3.6/config-3.6m-darwin', '-lpython3.6m', '-ldl', '-lpython3.6m', '-ldl', '-L/usr/local/opt/llvm/lib']
err: ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
ld: -stack_size option can only be used when linking a main executable
clang-5.0: error: linker command failed with exit code 1 (use -v to see invocation)

from /Users/xxx/Documents/git/essentia/src/python: Test does not build: Traceback (most recent call last):
  File "/Users/xxx/Documents/git/essentia/.waf3-2.0.10-cf8ca5fa84387374f6d66295baab59ec/waflib/Configure.py", line 324, in run_build
    bld.compile()
  File "/Users/xxx/Documents/git/essentia/.waf3-2.0.10-cf8ca5fa84387374f6d66295baab59ec/waflib/Tools/errcheck.py", line 140, in check_compile
    ret=self.orig_compile()
  File "/Users/xxx/Documents/git/essentia/.waf3-2.0.10-cf8ca5fa84387374f6d66295baab59ec/waflib/Build.py", line 176, in compile
    raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
 -> task in 'testprog' failed with exit status 1: 
    {task 4438311992: cxxshlib test.cpp.1.o -> testprog.cpython-36m-darwin.so}
['/Users/xxx/anaconda3/envs/cling/bin/clang++', '-arch', 'x86_64', '-arch', 'x86_64', '-Wl,-stack_size,1000000', '-bundle', '-undefined', 'dynamic_lookup', 'test.cpp.1.o', '-o/Users/xxx/Documents/git/essentia/build/.conf_check_02bb3575a4040bfc70a336934c2010f5/testbuild/testprog.cpython-36m-darwin.so', '-framework', 'CoreFoundation', '-L/Users/xxx/anaconda3/envs/cling/lib/python3.6/config-3.6m-darwin', '-lpython3.6m', '-ldl', '-lpython3.6m', '-ldl', '-L/usr/local/opt/llvm/lib']

Could not build python extensions
from /Users/xxx/Documents/git/essentia/src/python: The configuration failed

@dbogdanov I tried pip install essentia and get the same outcome.