Mbed-os: Exporting to Makefile fails to build on Windows

[SeppoTakalo]

Description

• Type: Bug
• Priority: Minor

Bug

Target
Any

Toolchain:
Any Makefile based export.

Expected behavior
mbed export -m <board> -i make_gcc_arm
make

Actual behavior
Error code:
make (e=87): The parameter is incorrect.

This is produced because the final link command is 44,252 characters long.

Windows CreateProcess() only accepts command line maximum length to be 32,768 characters. See here.

Similar issue described here: https://stackoverflow.com/questions/12598933/ndk-build-createprocess-make-e-87-the-parameter-is-incorrect

Steps to reproduce
Import mbed-os-example-mesh-minimal
mbed export -m <board> -i make_gcc_arm

Workarounds

1. Don't use Makefile builds on Windows.

[theotherjimmy]

Yeah, there's not a whole lot I can do about that. Known issue?

[MarceloSalazar]

FYI @adbridge

[theotherjimmy]

@MarceloSalazar What do you want Anna to do about this? Is there something I can do?

[MarceloSalazar]

I've just wanted Anna to be aware of this, as may need to go to the known issue list.

[theotherjimmy]

I can add it. Would you like a known issue?

[theotherjimmy]

Known issue added.

[sg-]

Is there a workaround for this based on how close to root you place the exported project or are all these paths relative to the extract point on disc and due to the amount of features and source code and path expansion in Mbed OS for a particular target?

[theotherjimmy]

@sg- I'd have to check, but I think you may be able to work around this by extracting into an already short path. That being said, if the project exceeds the path limit without any prefixes, then there is no workaround (other than use another exporter, or use Linux)

[SeppoTakalo]

I cannot remember whether it used relative or absolute path names...
I would assume that relative, which would mean that no workarounds exists, other than using proper operating system.

[sg-]

proper operating system.

🤣

[theotherjimmy]

XD "proper" operating system

[theotherjimmy]

https://github.com/ARMmbed/mbed-os-example-bootloader/issues/58 is a duplicate.

[0xc0170]

@ARMmbed/mbed-os-tools Can this be closed or still expecting fixes?

[SeppoTakalo]

@0xc0170 Why would you close issues without fixing them?

Eventually this will pop up again, and we should be able to link to existing issue.

[SeppoTakalo]

There are various workaround to fix this.

For ARM linker there is --via option to allow specifying the filenames in a file: http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0493e/CHDJEJCA.html

For GCC this probably needs working with the linker script file, as the equivalent of --via does not exist: http://ftp.gnu.org/old-gnu/Manuals/ld-2.9.1/html_mono/ld.html#SEC5

For IAR, the linker has -f option that allows specifying all command line parameters in a file.
http://netstorage.iar.com/SuppDB/Public/UPDINFO/006738/arm/doc/EWARM_DevelopmentGuide.ENU.pdf

[theotherjimmy]

Yes @SeppoTakalo That's how things compile in mbed compile. It's a bit of work to include these fixes into the makefiles.

[robmosys]

A colleague of mine just hit this bug.
For gcc, it should be possible to use the @file option to gcc to pass this to linker.
( https://gcc.gnu.org/onlinedocs/gcc-8.1.0/gcc/Overall-Options.html#Overall-Options )
Will test this solution and report back.

Also this page may be of importance:
https://community.arm.com/processors/f/discussions/4748/is-there-a-maximum-length-of-compile-path
Some people may be using old versions of the make command.

[theotherjimmy]

@robmosys note that mbed compile already does many optimizations for all compilers to get around these limits. The make exporters need to be updated to include the same code.

[robmosys]

Ok I've successfully worked round this problem with the @file method.
Replaces the linker rule in the mbed makefile with the following.
```make
$(PROJECT).elf: $(OBJECTS) $(SYS_OBJECTS) $(PROJECT).link_script.ld
$(file > [email protected] , $(filter %.o, $^))
+@echo "link: $(notdir $@)"
$(LD) $(LD_FLAGS) -T $(filter-out %.o, $^) $(LIBRARY_PATHS) --output $@ @[email protected] $(LIBRARIES) $(LD_SYS_LIBS)
````

This only puts the object files in the file but this is all that's necessary to ensure almost all parameters to collect2 and ld are in the @file
I have also confirmed that for ld operations the contents of an @file are passed to the sub-programs collect2 and ld.
note I tried this on windows subsystem for linux, and the $(file ) function would fail, so may be windows-only.

@theotherjimmy you mentioned that fixes are already done in mbed compile could you say where I could find these fixes?

[theotherjimmy]

@robmosys Sure.

• https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/__init__.py#L313-L326 Impl of creating the include files
• https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/arm.py#L181 Option creation for ARMC5
• https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/gcc.py#L179 Option creation for GCC
• https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/iar.py#L171 Option creation for IAR

[theotherjimmy]

• https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/gcc.py#L251-L255 GCC Linker command file
  
  
  
  • https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/arm.py#L283-L286 ARMC5 linker command file
  
  
  • https://github.com/ARMmbed/mbed-os/blob/master/tools/toolchains/iar.py#L225-L229 IAR linker command file
  
  

[robmosys]

In summary both my workaround and the mbed compile command use response files.
I also found out why workaround didn't work on wsl:
My wsl was an old Ununtu with 3.8 version of make. I think the file command was introduced with the 4.0 version of make in 2013.

[theotherjimmy]

@robmosys Yep. (I have yet to see any offiical documentation for "Response files" I have seen them called "command files" (personal favorite term) or "via files").

Thanks for the heads up on make 4.0. maybe we could use these command/via/response files with the file command and require make >= 4

[russetrob]

Regarding the term "Response files" took this from the source, after a quick google for validity: It's used but not so commonly.
See self.RESPONSE_FILES usage in the code snippets you pointed at above.
(replying from home account work account is robmosys)

[robmosys]

The make code snippet for linking above wouldn't work on WSL due to space before comma.
Here's an update that should work on wsl.
Only change is removal of space between in and ,
```make
$(PROJECT).elf: $(OBJECTS) $(SYS_OBJECTS) $(PROJECT).link_script.ld
$(file > [email protected], $(filter %.o, $^))
+@echo "link: $(notdir $@)"
$(LD) $(LD_FLAGS) -T $(filter-out %.o, $^) $(LIBRARY_PATHS) --output $@ @[email protected] $(LIBRARIES) $(LD_SYS_LIBS)
````

[SeppoTakalo]

Why the use of $(filter ...) instead of just $(OBJECTS) $(SYS_OBJECTS)?

Those macros might contain (not sure) other than object files as well. For example binary libraries .a or .ar

[theotherjimmy]

@SeppoTakalo So that you can add more dependencies to the elf file, and expect them to work. Those macros do not contain the binary libraries.

[theotherjimmy]

@robmosys Thanks for the prototype. I'll send a PR soon doing the same thing for all of the compilers shortly.

[theotherjimmy]

see #7583

[JojoS62]

I'm getting the error 87 now also with an export to gnuarmeclipse. Is there any workaround known? As far as I understand the problem is solved by using a response file for the linker, but this is not yet implemented for the gnuarmeclipse exporter?

[EduardShaid]

All of the above methods do not solve anything on the merits !

[robmosys]

@theotherjimmy in your pull request using response files, you had difficulty finding a modern make on Windows. I got my version of make from here: https://gnu-mcu-eclipse.github.io/windows-build-tools/

[EduardShaid]

thanks for the answer, but it also did not help, because there is still a bug at the level of the CreateProcess(...)

[ConductedForce]

Tried @robmosys fix. Works with Make GCC Arm toolchain export. Had this problem when I imported direct from github into Mbed Online and didn't use stock example projects.

[robmosys]

Can someone tell me what the future of this bug is?
I'm trying to integrate a fix in the most mbed compatible way, for our company, and patch #7559 looks good.
But as this patch hasn't been accepted and this bug is closed, I'm wondering how long the patch will be applicable to the tree. Currently I'm inclined to make a local fork, apply the patch and maintain it as necessary.

[cmonr]

@robmosys Sorry we missed your comment.

If this is still an issue, would you mind opening a new issue?

[srberard]

I'm experiencing the same issue as well, this issue was closed. Was the fix merged in? If not, I'm still experiencing this exact issue on 5.10

[cmonr]

@srberard @robmosys This PR was likely closed because at the time, the issue was resolved.

Because Windows filepaths are fundamentally shorter than *nix limitations, Windows has been rather suceptable to this problem. We're working on a more permanent fix, but it tends to come back every once in a while.

Please open new issues with steps to help us reproduce. We're working on a couple of issues and tests to prevent this from regressing.

[srberard]

@cmonr Thanks for the reply. I'm confused as the issue was not previously fixed. On November 2 @robmosys was still asking for this to be integrated. So yes, the proposed fix works it just was not integrated in and needed to be manually applied. Thus I'm not sure what use a new issue is here as this issue correctly outlines the problem (and proposed solution). Can't we just reopen/reuse this issue?

I've manually edited the exported makefile as outlined above. I can confirm that using a response file fixes this on Windows (and presumably this works on other platforms as well).

[cmonr]

@srberard I'm confused. https://github.com/ARMmbed/mbed-os/pull/7559 was accepted and merged before their comment was made: https://github.com/ARMmbed/mbed-os/issues/6335#issuecomment-435458936.

As for the issue, we prefer to have issues opened a-new instead of having older issues ressurected (see https://github.com/ARMmbed/mbed-os/issues/6335#event-1879426534). It's the same reason that forums generally frown upon users that try to ressurect dead forum threads.

I'd be interested to see the diff of your fix, as well as to confirm your tool versions. Part of the finding fixes for this is that we only support a certain set of tools, so one solution that works might not even exist in an older version.

[srberard]

@cmonr Indeed this thread is confusing. From what I can tell the fix is NOT included in the release so I'm not sure what happened. I can open a new issue if that's required.

As for my "fix", I simply made the change indicated by @robmosys here on my local copy of the generated makefile. Worked like a charm.

As for toolchain, I'm using mbed CLI 1.8.3 and the following:

C:\src>make --version
GNU Make 4.2.1
Built for x86_64-w64-mingw32
Copyright (C) 1988-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

C:\src>arm-none-eabi-gcc --version
arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 7-2018-q2-update) 7.3.1 20180622 (release) [ARM/embedded-7-branch revision 261907]
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[ConductedForce]

Faced this again using an Eclipse_GCC_ARM export from online compiler. Came back because I knew a fix was here. Edit: Well, fix doesn't exactly fix, still requires some editing.