Packages: Python-3.9.0 Compilation fails

Created on 28 Nov 2020  路  20Comments  路  Source: openwrt/packages

openwrt master latest
packages master latest

make profile-gen-stamp
make[5]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Building with support for profile generation:
make build_all_generate_profile
make[6]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
make build_all CFLAGS_NODIST=" -fprofile-generate" LDFLAGS_NODIST=" -fprofile-generate" LIBS="-ldl  -lpthread -lm"
make[7]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
 CC='aarch64-openwrt-linux-musl-gcc' LDSHARED='aarch64-openwrt-linux-musl-gcc -shared -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate' OPT='-DNDEBUG -Os -pipe -mcpu=generic -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0=Python-3.9.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro'     _TCLTK_INCLUDES='' _TCLTK_LIBS=''       _PYTHON_PROJECT_BASE=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/build/lib.linux-aarch64-3.9:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata python3.9 ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
INFO: Could not locate ffi libs and/or headers

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_gdbm                 _lzma                 _tkinter
_uuid                 nis                   readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ctypes

Makefile:638: recipe for target 'sharedmods' failed
make[7]: *** [sharedmods] Error 1
make[7]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:531: recipe for target 'build_all_generate_profile' failed
make[6]: *** [build_all_generate_profile] Error 2
make[6]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:505: recipe for target 'profile-gen-stamp' failed
make[5]: *** [profile-gen-stamp] Error 2
make[5]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:516: recipe for target 'profile-run-stamp' failed
make[4]: *** [profile-run-stamp] Error 2
make[4]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:351: recipe for target '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built' failed
make[3]: *** [/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built] Error 2
make[3]: Leaving directory '/home/openwrt/master/openwrt/package/feeds/lang/python/python3'
time: package/feeds/lang/python/python3/compile#0.32#0.06#0.37
package/Makefile:112: recipe for target 'package/feeds/lang/python/python3/compile' failed
make[2]: *** [package/feeds/lang/python/python3/compile] Error 2
make[2]: Leaving directory '/home/openwrt/master/openwrt'
package/Makefile:108: recipe for target '/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile' failed
make[1]: *** [/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/openwrt/master/openwrt'
/home/openwrt/master/openwrt/include/toplevel.mk:240: recipe for target 'world' failed
make: *** [world] Error 2
picture peterwillcn

Most helpful comment

Fix was merged.

picture neheb on 29 Nov 2020
🎉1 👍1

All 20 comments

openwrt master latest
packages master latest

make profile-gen-stamp
make[5]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Building with support for profile generation:
make build_all_generate_profile
make[6]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
make build_all CFLAGS_NODIST=" -fprofile-generate" LDFLAGS_NODIST=" -fprofile-generate" LIBS="-ldl  -lpthread -lm"
make[7]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
 CC='aarch64-openwrt-linux-musl-gcc' LDSHARED='aarch64-openwrt-linux-musl-gcc -shared -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate' OPT='-DNDEBUG -Os -pipe -mcpu=generic -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0=Python-3.9.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro'     _TCLTK_INCLUDES='' _TCLTK_LIBS=''       _PYTHON_PROJECT_BASE=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/build/lib.linux-aarch64-3.9:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata python3.9 ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
INFO: Could not locate ffi libs and/or headers

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_gdbm                 _lzma                 _tkinter
_uuid                 nis                   readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ctypes

Makefile:638: recipe for target 'sharedmods' failed
make[7]: *** [sharedmods] Error 1
make[7]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:531: recipe for target 'build_all_generate_profile' failed
make[6]: *** [build_all_generate_profile] Error 2
make[6]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:505: recipe for target 'profile-gen-stamp' failed
make[5]: *** [profile-gen-stamp] Error 2
make[5]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:516: recipe for target 'profile-run-stamp' failed
make[4]: *** [profile-run-stamp] Error 2
make[4]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:351: recipe for target '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built' failed
make[3]: *** [/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built] Error 2
make[3]: Leaving directory '/home/openwrt/master/openwrt/package/feeds/lang/python/python3'
time: package/feeds/lang/python/python3/compile#0.32#0.06#0.37
package/Makefile:112: recipe for target 'package/feeds/lang/python/python3/compile' failed
make[2]: *** [package/feeds/lang/python/python3/compile] Error 2
make[2]: Leaving directory '/home/openwrt/master/openwrt'
package/Makefile:108: recipe for target '/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile' failed
make[1]: *** [/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/openwrt/master/openwrt'
/home/openwrt/master/openwrt/include/toplevel.mk:240: recipe for target 'world' failed
make: *** [world] Error 2

image

zxlhhyccc picture zxlhhyccc on 28 Nov 2020

me too

openips picture openips on 28 Nov 2020

Put TARGET_LDFLAGS+=-L$(STAGING_DIR)/usr/lib -L$(STAGING_DIR)/lib to /feeds/packages/lang/python/python3/files/python3-package-ctypes.mk after endef.

define Package/python3-ctypes
$(call Package/python3/Default)
  TITLE:=Python $(PYTHON3_VERSION) ctypes module
  DEPENDS:=+python3-light
endef

TARGET_LDFLAGS+=-L$(STAGING_DIR)/usr/lib -L$(STAGING_DIR)/lib

$(eval $(call Py3BasePackage,python3-ctypes, \
    /usr/lib/python$(PYTHON3_VERSION)/ctypes \
    /usr/lib/python$(PYTHON3_VERSION)/lib-dynload/_ctypes.$(PYTHON3_SO_SUFFIX) \
    /usr/lib/python$(PYTHON3_VERSION)/lib-dynload/_ctypes_test.$(PYTHON3_SO_SUFFIX) \
))
Vladdrako picture Vladdrako on 28 Nov 2020

Put TARGET_LDFLAGS+=-L$(STAGING_DIR)/usr/lib -L$(STAGING_DIR)/lib to /feeds/packages/lang/python/python3/files/python3-package-ctypes.mk after endef.

define Package/python3-ctypes
$(call Package/python3/Default)
  TITLE:=Python $(PYTHON3_VERSION) ctypes module
  DEPENDS:=+python3-light
endef

TARGET_LDFLAGS+=-L$(STAGING_DIR)/usr/lib -L$(STAGING_DIR)/lib

$(eval $(call Py3BasePackage,python3-ctypes, \
  /usr/lib/python$(PYTHON3_VERSION)/ctypes \
  /usr/lib/python$(PYTHON3_VERSION)/lib-dynload/_ctypes.$(PYTHON3_SO_SUFFIX) \
  /usr/lib/python$(PYTHON3_VERSION)/lib-dynload/_ctypes_test.$(PYTHON3_SO_SUFFIX) \
))

Add this line of code you provided to compile normally

zxlhhyccc picture zxlhhyccc on 28 Nov 2020

Can confirm this...

Ansuel picture Ansuel on 28 Nov 2020

@ynezz can we fix this? Also is it normal that python3 is compiled even if not selected in menuconfig (no deps based on it, in theory)

Ansuel picture Ansuel on 28 Nov 2020

@zxlhhyccc great !it's works

openips picture openips on 29 Nov 2020

Same. This fix the problem.

Ansuel picture Ansuel on 29 Nov 2020

The problem hasn't been fixed. https://github.com/openwrt/openwrt/commit/524fb5646eec6147aadfdd508219f39bcf8ba8fc
https://github.com/openwrt/openwrt/commit/f98878e4c17d5f11e78994b4fc456e6b60b2660f 

make[4]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Running code to generate profile data (this can take a while):
# First, we need to create a clean build with profile generation
# enabled.
make profile-gen-stamp
make[5]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Building with support for profile generation:
make build_all_generate_profile
make[6]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
make build_all CFLAGS_NODIST=" -fprofile-generate" LDFLAGS_NODIST=" -fprofile-generate" LIBS="-ldl  -lpthread -lm"
make[7]: Entering directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
 CC='aarch64-openwrt-linux-musl-gcc' LDSHARED='aarch64-openwrt-linux-musl-gcc -shared -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/usr/lib -L/home/openwrt/master/openwrt/staging_dir/toolchain-aarch64_generic_gcc-9.3.0_musl/lib -znow -zrelro -L/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 -flto -fuse-linker-plugin -ffat-lto-objects -flto-partition=none -g -fprofile-generate' OPT='-DNDEBUG -Os -pipe -mcpu=generic -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -fmacro-prefix-map=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0=Python-3.9.0 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro'     _TCLTK_INCLUDES='' _TCLTK_LIBS=''       _PYTHON_PROJECT_BASE=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0 _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/build/lib.linux-aarch64-3.9:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata python3.9 ./setup.py  build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
INFO: Could not locate ffi libs and/or headers

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_bz2                  _curses               _curses_panel
_gdbm                 _lzma                 _tkinter
_uuid                 nis                   readline
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


Failed to build these modules:
_ctypes

Makefile:638: recipe for target 'sharedmods' failed
make[7]: *** [sharedmods] Error 1
make[7]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:531: recipe for target 'build_all_generate_profile' failed
make[6]: *** [build_all_generate_profile] Error 2
make[6]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:505: recipe for target 'profile-gen-stamp' failed
make[5]: *** [profile-gen-stamp] Error 2
make[5]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:516: recipe for target 'profile-run-stamp' failed
make[4]: *** [profile-run-stamp] Error 2
make[4]: Leaving directory '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0'
Makefile:351: recipe for target '/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built' failed
make[3]: *** [/home/openwrt/master/openwrt/build_dir/target-aarch64_generic_musl/Python-3.9.0/.built] Error 2
make[3]: Leaving directory '/home/openwrt/master/openwrt/package/feeds/lang/python/python3'
time: package/feeds/lang/python/python3/compile#0.32#0.06#0.38
    ERROR: package/feeds/lang/python/python3 failed to build.
package/Makefile:112: recipe for target 'package/feeds/lang/python/python3/compile' failed
make[2]: *** [package/feeds/lang/python/python3/compile] Error 1
make[2]: Leaving directory '/home/openwrt/master/openwrt'
package/Makefile:108: recipe for target '/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile' failed
make[1]: *** [/home/openwrt/master/openwrt/staging_dir/target-aarch64_generic_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/openwrt/master/openwrt'
/home/openwrt/master/openwrt/include/toplevel.mk:240: recipe for target 'world' failed
make: *** [world] Error 2
peterwillcn picture peterwillcn on 29 Nov 2020

@peterwillcn those fixes are for cmake. python does not use cmake.

neheb picture neheb on 29 Nov 2020

@Ansuel don't forget there is also python3/host

neheb picture neheb on 29 Nov 2020

BTW error messages in this issue are long, but useless, because the compiler output is not visible.

By guessing from the proposed workaround (it's not proper fix):

TARGET_LDFLAGS+=-L$(STAGING_DIR)/usr/lib -L$(STAGING_DIR)/lib

it looks like some issue with the package as those linker flags are provided implicitly by the build system already, otherwise obviously every other package would fail. So something in this broken package is not honoring this linker flags and needs to be fixed.

ynezz picture ynezz on 29 Nov 2020

It's not just this one. google-authenticator-libpam fails because of that commit as well. I expect to find more failing packages once my VM compiles everything.

neheb picture neheb on 29 Nov 2020

@ynezz It is recommended to adjust the OpenWRT compilation architecture log error output to make it easier to track down accurate errors.

I compile using the following command

time make V=s VERBOSE=1 -j$(nproc) JOBS=$(nproc) 2>&1 | tee build.log | grep -i -E "^make.*(error|[1-9]...Entering dir)"
peterwillcn picture peterwillcn on 29 Nov 2020

Error according to the buildbots:

make[6]: Entering directory '/builder/shared-workdir/build/sdk/build_dir/target-arm_xscale_musl_eabi/Python-3.9.0'
find . -depth -name '__pycache__' -exec rm -rf {} ';'
find . -name '*.py[co]' -exec rm -f {} ';'
find . -name '*.[oa]' -exec rm -f {} ';'
find . -name '*.s[ol]' -exec rm -f {} ';'
find . -name '*.so.[0-9]*.[0-9]*' -exec rm -f {} ';'
find . -name '*.lst' -exec rm -f {} ';'
find build -name 'fficonfig.h' -exec rm -f {} ';' || true
/builder/shared-workdir/build/sdk/staging_dir/host/bin/find: 'build': No such file or directory
find build -name '*.py' -exec rm -f {} ';' || true
/builder/shared-workdir/build/sdk/staging_dir/host/bin/find: 'build': No such file or directory
find build -name '*.py[co]' -exec rm -f {} ';' || true
/builder/shared-workdir/build/sdk/staging_dir/host/bin/find: 'build': No such file or directory
rm -f pybuilddir.txt
rm -f Lib/lib2to3/*Grammar*.pickle
rm -f Programs/_testembed Programs/_freeze_importlib
find build -type f -a ! -name '*.gc??' -exec rm -f {} ';'
/builder/shared-workdir/build/sdk/staging_dir/host/bin/find: 'build': No such file or directory
Makefile:1858: recipe for target 'clean-retain-profile' failed
make[6]: [clean-retain-profile] Error 1 (ignored)

https://downloads.openwrt.org/snapshots/faillogs/arm_xscale/packages/python3/compile.txt

neheb picture neheb on 29 Nov 2020

Error according to the buildbots:

That's not that problem as you can see make[6]: [clean-retain-profile] Error 1 (ignored)

The issue is probably cc1: note: someone does not honour COPTS correctly, passed 3 times so likely wrong flags as I've already explained.

ynezz picture ynezz on 29 Nov 2020

Looks like it's just 5 broken packages. I'll make PRs for them.

neheb picture neheb on 29 Nov 2020
👍1

Two packages fail because of libpam. I wonder why. libpam does not use a pkgconfig file.

neheb picture neheb on 29 Nov 2020

Fix was merged.

neheb picture neheb on 29 Nov 2020
🎉1 👍1

@ynezz @neheb @Ansuel Thank all support!

peterwillcn picture peterwillcn on 29 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

ghost picture ghost  路  11Comments
ja-pa picture ja-pa  路  11Comments
zorun picture zorun  路  4Comments
robcollins55 picture robcollins55  路  5Comments
TeHashX picture TeHashX  路  8Comments