Esp-idf: [TW#27526] linker script generation failed for C:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/ld/esp32.common.ld.in

[ammaree]

Environment

• Development Kit: none
• Module or chip used: ESP32-WROOM-32
• IDF version (run git describe --tags to find it):
  // v3.2-beta1-182-g4c881708d
• Build System: Make
• Compiler version (run xtensa-esp32-elf-gcc --version to find it):
  // 1.22.0-80-g6c4433a5
• Operating System: Windows
• Power Supply: external 5V

Problem Description

Trying to build new binary after having updated to latest IDF and MSYS

Expected Behavior

Expect make to be successful

Actual Behavior

make -j9 all
Toolchain path: /opt/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
Toolchain version: crosstool-ng-1.22.0-80-g6c4433a5
Compiler version: 5.2.0
Python requirements from C:/Dropbox/devs/ws/z-sdk/esp-idf/requirements.txt are satisfied.
CC build/iMacs/x_config.o
AR build/iMacs/libiMacs.a
Generating libiMacs.a.sections_info
Generating esp32.common.ld
linker script generation failed for C:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/ld/esp32.common.ld.in
ERROR: C:/Dropbox/devs/ws/z-sdk/esp-idf/Kconfig:59: Could not open '/c/Dropbox/devs/ws/z-sdk/esp-idf/components/bootloader/Kconfig.projbuild' (ENOENT: No such file or directory). Perhaps the $srctree environment variable (which was unset) is set incorrectly. Note that the current value of $srctree is saved when the Kconfig instance is created (for consistency and to cleanly separate instances). Also note that e.g. $FOO in a 'source' statement does not refer to the environment variable FOO, but rather to the Kconfig Symbol FOO (which would commonly have 'option env="FOO"' in its definition).
Traceback (most recent call last):
File "C:/Dropbox/devs/ws/z-sdk/esp-idf/tools/ldgen/ldgen.py", line 101, in
main()
File "C:/Dropbox/devs/ws/z-sdk/esp-idf/tools/ldgen/ldgen.py", line 98, in main
os.remove(output_file.name)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:/Dropbox/devs/ws/z-appl/irmacs/build/esp32/esp32.common.ld'
make: * [/c/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/Makefile.projbuild:51: /c/Dropbox/devs/ws/z-appl/irmacs/build/esp32/esp32.common.ld] Error 1
make: * Waiting for unfinished jobs....

Steps to reproduce

Debug Logs

N/A

Other items if possible

[karawin]

Urgent....

Generating esp32.common.ld
linker script generation failed for C:/msys32/home/jp/esp/esp-idf/components/esp32/ld/esp32.common.ld.in
ERROR: C:/msys32/home/jp/esp/esp-idf/Kconfig:59: Could not open '/home/jp/esp/esp-idf/components/bootloader/Kconfig.projbuild' (ENOENT: No such file or directory). Perhaps the $srctree environment variable (which was unset) is set incorrectly. Note that the current value of $srctree is saved when the Kconfig instance is created (for consistency and to cleanly separate instances). Also note that e.g. $FOO in a 'source' statement does not refer to the environment variable FOO, but rather to the Kconfig Symbol FOO (which would commonly have 'option env="FOO"' in its definition).
Traceback (most recent call last):
File "C:/msys32/home/jp/esp/esp-idf/tools/ldgen/ldgen.py", line 101, in
main()
File "C:/msys32/home/jp/esp/esp-idf/tools/ldgen/ldgen.py", line 98, in main
os.remove(output_file.name)
WindowsError: [Error 32] Le processus ne peut pas acc▒der au fichier car ce fichier est utilis▒ par un autre processus: 'C:/msys32/home/jp/esp/Ka-Radio32/build/esp32/esp32.common.ld'
make: * [/home/jp/esp/esp-idf/components/esp32/Makefile.projbuild:51: /home/jp/esp/Ka-Radio32/build/esp32/esp32.common.ld] Error 1

[detly]

On Linux (Ubuntu 18.04) I am seeing a possibly-related issue after doing a git pull (and submodule update) today:

Generating esp32.common.ld
linker script generation failed for /home/me/Code/esp-libs/esp-idf/components/esp32/ld/esp32.common.ld.in
ERROR: 
LD build/esp-streamer.elf
/home/me/Code/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/5.2.0/../../../../xtensa-esp32-elf/bin/ld: cannot open linker script file /home/me/Code/esp-streamer/build/esp32/esp32.common.ld: No such file or directory
collect2: error: ld returned 1 exit status
/home/me/Code/esp-libs/esp-idf/make/project.mk:452: recipe for target '/home/me/Code/esp-streamer/build/esp-streamer.elf' failed
make: *** [/home/me/Code/esp-streamer/build/esp-streamer.elf] Error 1

[ammaree]

I have now gone through all possibly related posts I have been able find especially #2480 but no luck.
All modules and dependencies are up to date but the symptom remains the same.

[karawin]

Revert the local by
git reflog -10
git reset --hard HEAD@{x} with x to be choosen
and finally
git submodule update

Deleteing the build in the project may help too.

[ammaree]

Nope, no luck, caused lots of other errors.
Used 9 for 'x'
Updated all repos to current latest head, and back to same problem..

[ammaree]

@igrr @projectgus

In order to start with a completely clean setup I deleted the MSYS2 environment, the complete esp-idf and sub-modules then recreated the whole installation using the latest steps EXACTLY.
The prerequisite steps installed nothing new, stated all were met.

On building an existing application, that has worked for the last 18 months it failed with exactly the same error:

Generating esp32.common.ld
linker script generation failed for C:/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/ld/esp32.common.ld.in
ERROR: C:/Dropbox/devs/ws/z-sdk/esp-idf/Kconfig:59: Could not open '/c/Dropbox/devs/ws/z-sdk/esp-idf/components/bootloader/Kconfig.projbuild' (ENOENT: No such file or directory). Perhaps the $srctree environment variable (which was unset) is set incorrectly. Note that the current value of $srctree is saved when the Kconfig instance is created (for consistency and to cleanly separate instances). Also note that e.g. $FOO in a 'source' statement does not refer to the environment variable FOO, but rather to the Kconfig Symbol FOO (which would commonly have 'option env="FOO"' in its definition).
Traceback (most recent call last):
File "C:/Dropbox/devs/ws/z-sdk/esp-idf/tools/ldgen/ldgen.py", line 101, in
main()
File "C:/Dropbox/devs/ws/z-sdk/esp-idf/tools/ldgen/ldgen.py", line 98, in main
os.remove(output_file.name)
WindowsError: [Error 32] The process cannot access the file because it is being used by another process: 'C:/Dropbox/devs/ws/z-appl/irmacs/build/esp32/esp32.common.ld'
make: * [/c/Dropbox/devs/ws/z-sdk/esp-idf/components/esp32/Makefile.projbuild:51: /c/Dropbox/devs/ws/z-appl/irmacs/build/esp32/esp32.common.ld] Error 1

[karawin]

We have to wait unfortunately.

[pilnikov]

https://github.com/karawin/Ka-Radio32/issues/65

[projectgus]

Hi folks,

Sorry for the inconvenience. There was a recent regression in master for Windows users (using MSYS2 & GNU Make) in 8915f482089d2666d81174e9d671af300036472b which causes an error of this format: linker script generation failed for %IDF_PATH%/esp-idf/components/esp32/ld/esp32.common.ld.in ERROR: %IDF_PATH%/esp-idf/Kconfig:59: Could not open '%IDF_PATH%/components/bootloader/Kconfig.projbuild' (ENOENT: No such file or directory). error.

A fix has been merged internally and will be pushed to github very shortly. In the meantime, possible workarounds include:

• Check out Git commit 964f5a91f7fe2d2168d58db5314e11c91289aa17 which is the last commit on master before the regression. (Do a git submodule update after checking out this commit.)
• Switch to using the CMake Build System on Windows.
• Switch to using an IDF stable release

[projectgus]

@pilnikov I notice in the linked issue you mention problems with IDF v3.2-beta1. v3.2 doesn't have the linker script generator feature so doesn't have this specific regression. If you're experiencing different build errors in v3.2-beta1, please open another issue with details and we'll look into it.

[viduata]

Rolling back to the noted commit above allows the compile to go through without error, but the output is now bad. The image has gone from around 300k to 3072 bytes. Flashing and running just gives an infinite scroll of errors and reboots.

Note: I ended up just renaming my old repo, cloning the current master branch, updating the submodule, and everything is working fine now.

[projectgus]

This should be fixed in master as of commit 3970ea60deade62224ad227fd5f986d29b0abdfb

Thanks everyone for reporting and being patient with the fix.

[WayneKeenan]

@projectgus fyi, 964f5a9 works for me, but 3970ea6 doesn't.

[projectgus]

@WayneKeenan Thanks for letting me know.

So the older commit on master is working for you and the newer commit is not? Are you getting the exact same error reported in the original issue? Can you please give some details about your environment, build logs, etc? Feel free to open a new issue if this is more convenient.

[detly]

Link stage still fails on Ubuntu.

[detly]

If this is solely about a bug under Windows/MSYS, do you want me to open a separate ticket?

[projectgus]

@detly yes, if you're getting a linker failure on Ubuntu then a new issue with the details sounds best. Thanks.

[Aiouan]

I am still getting this problem in Windows, I have done a 100% reinstall of the whole toolchain.
It seems ok using MSYS from the command line, but the project(s) intermittently fail to build in Eclipse and even if they do build the output is useless as described above.

[alsaleem00]

@projectgus: I still have the same issue on windows/eclipse. The fixes you mentioned above are already there in the idf install. (same comments as @Aiouan)
I have (v3.3-dev-206-g0d7f2d77c)
Any further help?.

[projectgus]

@alsaleem00 Commit 0d7f2d77c is from almost two weeks ago, can you please try pulling latest master branch?

@Aiouan @alsaleem00 If the issue still occurs on current master branch, please post full output from a failing build.

[alsaleem00]

Now, I have (v3.3-dev-343-g7fa98593b).
I got new error (IOError )
Here is the complete compile output.
Note: I changed to stable 3.1.1 and compiled OK.
idf-error.txt

[JOO200]

I got the same issue. Compiling with msys32.exe works for me.

I printed the expected Kconfig.projbuild path in the build console.
The requested file is C:/msys32/home/Johannes/esp/esp-idf/components/bootloader/Kconfig.projbuild, I can open this path with msys32.exe with nano C:/msys32/home/Johannes/esp/esp-idf/components/bootloader/Kconfig.projbuild
I checked the output from print os.path.exists(filename) too, the answer is for this file false.
The file C:/msys32/home/Johannes/esp/esp-idf/Kconfig can be found.

Thanks for the help.

[JOO200]

Okay, I tried something else:
I edited the python script kconfiglib.py at line 1090.
The file exists if I make a hardcoded request for os.path.exists(..)
The file doesn't exists if I use the variable with name filename.
I checked the bytecode with print ":".join("{:02x}".format(ord(c)) for c in filename) Then I parsed this bytecode with a hex to ascii converter ( https://www.rapidtables.com/convert/number/hex-to-ascii.html ):

The function _open gets two times called. First time the path is C:/msys32/home/Johannes/esp/esp-idf/Kconfig It's all fine, no problems. The bytecode is like the path.
The second time the path is printed as C:/msys32/home/Johannes/esp/esp-idf/components/bootloader/Kconfig.projbuild
After checking the bytecode I was confused: /home/Johannes/esp/esp-idf/components/bootloader/Kconfig.projbuild
Where is the absolute path? I don't know, how the print command printed the absolute path.
os.path.isabs(filename) returns true

I'm running out of ideas...

[JOO200]

I tested it with the fix from @igrr but it doesn't work.
Still the same error.

I'm trying the cmake-version now^^

[ulfalizer]

Hello,

As a note, Kconfiglib itself doesn't do any custom path handling. It just passes paths as given to open(), and joins paths with os.path.join(), which uses the native format for the OS/Python installation (backslashes for plain Windows Python, etc.) If you run into problems, they might be related to whatever Python on MSYS does.

Using / instead of \ as the path separator works fine on Windows too (see os.altsep).

The version of Kconfiglib in Espressif is pretty old by the way. There's lots of improvements in the latest version, like default-UTF-8 decoding, faster parsing, strict syntax checking, support for glob patterns in source, node iteration helpers, menuconfig implementations, a Kconfig preprocessor, and more.

Ask if you have any questions!

Cheers,
Ulf

[projectgus]

Hi all,

Thanks @ulfalizer for the helpful info about kconfiglib. I agree this is not a kconfiglib problem, and that we should update our (patched) kconfiglib soon! :)

@JOO200 @alsaleem00 You're both using Eclipse on Windows, yes? I'm not able to reproduce any problem building in Eclipse on Windows with current master branch, my guess is this was the same issue as #2812 that was fixed in d3938c8.

If you're not still not able to build in Eclipse with current master, please check all the settings are the same as given in the ESP-IDF Eclipse Setup guide. If it's not working, please open a new issue and post as many details as you can about your environment, paths, build logs, etc. and we'll continue from there.