Platformio-core: "Command line is too long" while building large project under Windows

[nathanjel]

• [x] PlatformIO Core.
  If you’ve found a bug, please provide an information below.

Configuration

Windows 10
PlatformIO Core, version 5.0.4

Description of problem

When building my ESP32 project, I get "The command line is too long." at the final firmware linking step.

Steps to Reproduce

pio run -e esp32dev-21pin

Actual Results

Too long line issue - resulting linker command - Pastebin.com

The command line is too long.
*** [.pio\build\esp32dev-21pin\firmware.elf] Error 1

Expected Results

Successful build

If problems with PlatformIO Build System:

The content of platformio.ini:
Too long line issue - platformio.ini - Pastebin.com

Additional info

• The same project builds perfectly on Ubuntu 20.04.1 LTS, same version of PIO
• Resulting linker command was captured by adding -v to the command line
• Replication - as I work on a proprietary solution - is not straightforward, but what should help is that my project imports 76 external libraries thru lib_deps. That results in the final linker command of 10280 bytes. Please see my platformio.ini and the resulting linker command for building the 21-pin version - it should give You a good idea of how to reproduce this issue. I had to mask the files a bit - sorry for this.

[justoke]

I am having the same issue, although out of 3 env defined in the platformio.ini only the node32s builds, whereas the others fail with the "path too long". I tried moving the whole project to a folder in the root of my drive to short it as much as possible without any effect. I am using Windows 10.

[nathanjel]

I just tried with lib_archive = false and it fails on Windows 10

Linking .pio\build\esp32dev-21pin\bootloader.elf
Building .pio\build\esp32dev-21pin\bootloader.bin
esptool.py v3.0
Linking .pio\build\esp32dev-21pin\firmware.elf
xtensa-esp32-elf-g++: error: CreateProcess: No such file or directory
*** [.pio\build\esp32dev-21pin\firmware.elf] Error 1
=============================================================== [FAILED] Took 233.96 seconds ===============================================================
Environment     Status    Duration
--------------  --------  ------------
esp32dev-21pin  FAILED    00:03:53.956
========================================================== 1 failed, 0 succeeded in 00:03:53.956 ========================================================== 
PS C:\Work\E2-Sample-Demonstration> 

[justoke]

I just tried with lib_archive = false and it fails on Windows 10
```
I tried the same with the same failure

[nathanjel]

I just managed to create a workaround based on original PIO builder/tools/piomaxlen.py. The fact it works confirms a need to further develop platformio-core. The existing version supports large linker command line only in case of large set of linker source files, but it does not handle libraries as of yet.

The workaround involves creating an additional script and referencing it from platformio.ini.

1. place win_linker_workaround.py in project root directory, aside platformio.ini

Import("env")

from os.path import isfile, join
import hashlib
import os

def _file_long_data_workaround(env, _data):
    data = env.subst(_data).replace("\\", "/")
    build_dir = env.subst("$BUILD_DIR")
    tmp_file = join(
        build_dir, "longcmd-%s" % hashlib.md5(data.encode()).hexdigest()
    )
    retval = '@"%s"' % tmp_file
    if not isfile(tmp_file):
        with open(tmp_file, "w") as fp:
            fp.write(data)
    return retval

def implant_hook_for_libflags():
    env.Replace(_long_libflags_hook=_file_long_data_workaround)
    coms = {}
    key = "LINKCOM"
    coms[key] = env.get(key, "").replace(
        "$_LIBFLAGS", "${_long_libflags_hook(__env__, _LIBFLAGS)}"
    )
    env.Replace(**coms)

implant_hook_for_libflags()

2. do needed changes to platformio.ini

...
[env]
...
extra_scripts =
  pre:win_linker_workaround.py
...

And for me it solves the problem:

Linking .pio\build\esp32dev-21pin\firmware.elf
Building .pio\build\esp32dev-21pin\firmware.bin
Retrieving maximum program size .pio\build\esp32dev-21pin\firmware.elf
Checking size .pio\build\esp32dev-21pin\firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  14.7% (used 48324 bytes from 327680 bytes)
Flash: [=======   ]  72.5% (used 1140630 bytes from 1572864 bytes)
esptool.py v3.0
============================================================================= [SUCCESS] Took 203.75 seconds =============================================================================

Environment     Status    Duration
--------------  --------  ------------
esp32dev-21pin  SUCCESS   00:03:23.755
============================================================================== 1 succeeded in 00:03:23.755 ==============================================================================

[justoke]

I've tried that but still get the error:

IS there anyway to confirm the script has run?

[nathanjel]

@justoke To my best guess - You're facing a different error - the one You reported occurs during source file compilation. My workaround only resolves the issue at final linking step. I guess in Your case a similar workaround for compile command could do the trick, but explaining it is beyond this forum.

Regarding checking if the script is being run - sure, add something like print("HELLO, IT's ME, THE SCRIPT!!!!!!!!!!!!!") in the _file_long_data_workaround function. You will notice this in the output.

[justoke]

@justoke To my best guess - You're facing a different error - the one You reported occurs during source file compilation. My workaround only resolves the issue at final linking step. I guess in Your case a similar workaround for compile command could do the trick, but explaining it is beyond this forum.

Regarding checking if the script is being run - sure, add something like print("HELLO, IT's ME, THE SCRIPT!!!!!!!!!!!!!") in the _file_long_data_workaround function. You will notice this in the output.

Thanks for the tip - you are right this is compile time, but this could be promising as a workaround. I'll share my findings