Arduino-cli: mbed target - Linking fails under Windows with large list of source files

[aureleq]

Bug Report

Current behavior

My project contains a library with a large number of source files (roughly 500 .cpp).
Under macOS the project compiles without any issue. However, under Windows 10 linking fails as the 500 object files put as argument make the command line exceeds 32k characters (Windows limit). The linking command line is roughly 64k characters long. The issue appears both with CLI or IDE.

Arduino output:
fork/exec C:\Users\aurel\AppData\Local\Arduino15\packages\arduino\tools\arm-none-eabi-gcc\7-2017q4/bin/arm-none-eabi-g++.exe: The filename or extension is too long.

Expected behavior

As a solution the list of object files should be put in a text file, linking command can read the list from it to avoid exceeding the 32k characters limit.

As a workaround, I created a platform.local.txt to bypass this issue, my linking command has become:

platform.local.txt

recipe.c.combine.pattern="{compiler.path}{compiler.c.elf.cmd}" "-L{build.path}" {compiler.c.elf.flags} {compiler.c.elf.extra_flags} "@{compiler.mbed.ldflags}" "-T{build.variant.path}/{build.ldscript}" "-Wl,-Map,{build.path}/{build.project_name}.map" --specs=nano.specs --specs=nosys.specs {compiler.ldflags} -o "{build.path}/{build.project_name}.elf" @{build.path}\obj_files.txt -Wl,--whole-archive "{build.path}/{archive_file}" {compiler.mbed} -Wl,--no-whole-archive -Wl,--start-group {compiler.mbed.extra_ldflags} {compiler.libraries.ldflags} -Wl,--end-group

This solution should work well for most of projects but this should be managed directly into the Arduino builder.

Environment

• Arduino CLI version: 0.11
• Arduino IDE version: 1.8.12
• OS and platform: Windows 10

Additional context

[ubidefeo]

@aureleq
would you mind testing this build?
https://github.com/arduino/arduino-cli/suites/1223216696/artifacts/18327115

we'd like to hear back from you at your earliest convenience :)

[aureleq]

Hi @ubidefeo,

Just tested it works :) Thanks!

Aurelien

[ubidefeo]

thank you @aureleq
Nice way to close the week, @cmaglie 🚀

[aureleq]

Hello,

FYI we had a similar issue while linking for an ESP32 target. We cannot use the same workaround as the xtensa linker expands the list of files with the @ syntax.

Below is another solution for ESP32 that should work on any target (this workaround is also mentioned here by @facchinm : https://github.com/arduino/arduino-cli/pull/961#issuecomment-704090383):

## Customized platform.local.txt to compile for ESP32 targets under Windows
## This recipe works around the Windows limit of 32k characters in a cmd line, creating an archive with sketch and libraries object files before linking
## Install: put file under your esp32 sub-directory - ie:C:\Users\MYUSER\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4
## You can also find the local arduino folder in the IDE under File -> Preferences -> see preferences.txt location in the bottom part of the window

## hooks

# create a txt file with all object file paths from libraries and sketch folders
recipe.hooks.linking.prelink.1.pattern=cmd /c dir /b /s {build.path}\sketch\*.o > {build.path}\obj_files_tmp.txt
recipe.hooks.linking.prelink.2.pattern=cmd /c "dir /b /s {build.path}\libraries\*.o >> {build.path}\obj_files_tmp.txt 2>nul & exit 0"

# replace \ by \\ in file paths (otherwise escaped by linker) and save in new txt file
recipe.hooks.linking.prelink.3.pattern=cmd /v /c "@echo off && for /f %a in ({build.path}\obj_files_tmp.txt) do (set line=%a && set line=!line:\=\\! && echo !line! >> {build.path}\obj_files.txt)"

# save object files in archive
recipe.hooks.linking.prelink.4.pattern=cmd /v /c "@echo off && for /f %a in ({build.path}\obj_files.txt) do ({compiler.path}{compiler.ar.cmd} {compiler.ar.flags} {compiler.ar.extra_flags} {build.path}\custom_lib.a %a)"

# delete txt file and archive after linking
recipe.hooks.linking.postlink.1.pattern=cmd /c del {build.path}\obj_files.txt
recipe.hooks.linking.postlink.2.pattern=cmd /c del {build.path}\custom_lib.a


## modify compile patterns (use custom_lib.a instead of object_files)
recipe.c.combine.pattern="{compiler.path}{compiler.c.elf.cmd}" {compiler.c.elf.flags} {compiler.c.elf.extra_flags} -Wl,--start-group "{build.path}\custom_lib.a" "{archive_file_path}" {compiler.c.elf.libs} -Wl,--end-group -Wl,-EL -o "{build.path}/{build.project_name}.elf"

[ubidefeo]

@aureleq
I know that @cmaglie is still polishing a workaround, maybe your suggestion for ESP32 can be useful to him in the process.
Thanks 👍