Rust: emscripten linking with many dependencies broken on Windows

[Moxinilian]

Hello.

I have been trying to compile a rather big project for wasm32-unknown-emscripten using cargo on Windows.
Unfortunately, the linking process fails when trying to run emcc.bat because the command line invoked is too long for Windows' limitations (about 8k characters).
error: linking with `emcc.bat` failed: exit code: 1
note: The command line is too long.

After manually counting, the command indeed is too big with in my case about 11k characters, as it includes the absolute path of every single .o and .rlib files. My project, while including piston as a dependency, does not have an incredibly large amount of them, which makes me believe this could be an important issue in the long run (this does not apply to Linux however as the maximum character limit usually is much higher).
After a little bit of digging, emscripten seems to provide a workaround for this problem by accepting parameters from files. However it has to be handled by the user.

Thanks in advance for your help.

[dataf3l]

why not use a folder like c:a so that it works?

[Moxinilian]

Well yes, that would work in my specific case.
But do you really want that in a final product?

[ritiek]

Maybe #46113 is related.

[alexcrichton]

@Moxinilian can you provide the full error message? It looks like this may be a bug in emscripten, not in rustc.

[Moxinilian]

@alexcrichton I uploaded the project I tried to compile to this repo
https://github.com/Moxinilian/emscripten_bug
log.txt contains the entire output

edit: i'm adding the verbose output too
edit: done

[alexcrichton]

Ah ok thanks for the extra info @Moxinilian!

So for a bit of background, rustc does indeed have safeguards in place for reinvoking the linker if necessary when the command line is deemed too large. What's happening here, though, is that the error is happening at the wrong time.

The current detection relies on the spawning of the process itself to fail (aka the OS tells us about the failure) but it looks like cmd at least is succeeding. Some spawned process actually prints out The command line is too long but it's not clear to me whether that's cmd or emcc.bat.

[Moxinilian]

@alexcrichton Is there any additional test you'd want me to run to try to figure this out?

[alexcrichton]

If you'd like to debug further that'd be great. We'd first need to understand what's actually failing here (cmd or something in emscripten). Once that's known we could figure out if it's reasonable for us to catch the error and do something about it.

[Moxinilian]

@alexcrichton So I edited emcc.bat to print a dummy message when it gets called.
As the message did not show up, I came to the conclusion that cmd itself refuses to run the command.
I tried to run it with a shorter command and the message showed up.

This is surprising because that means rust would have failed to survive cmd's refusal.
I'm not an OS expert so I'm not sure what raw_os_error actually returns, but here cmd's return code is 1.

[alexcrichton]

Ok so sounds like we successfully spawn a cmd process and then cmd fails to spawn its own subprocess. That's going to be much trickier to handle an catch, but it at least sounds like what's happening here at least.

[Moxinilian]

A dirty way to catch it would be to simply check the error description. At least it is self explanatory so there won't be any false positive.

[Moxinilian]

Another way I just thought of would be to try to summon emcc.bat without cmd, maybe trying to run it using something similar to C's system("emcc.bat args");. Maybe this would allow better handling.

[VictorKoenders]

Assuming emcc.bat and em++ are working in a similar matter, this issue seems to provide a solution. We can simply write all the command parameters to an .rsp file, and then pass that file to the emcc.bat

[Moxinilian]

Honestly, I believe doing this approach by default would be the best solution.
The performance overhead is negligible when it's only about linking, and this would cover all future problems.
It's simply a matter of deleting this match.
I can not try it right now but if anyone is interested to remove it and see if it works.

[projektir]

Writing the arguments to file looks like a good solution here, yeah.

[forbjok]

I'm running into this issue as well when trying to use the cargo-web crate. Any ETA on this being fixed in nightly?

[Moxinilian]

If nobody does it before I do, I'll make file parameters default behavior, check if nothing breaks and make a pull request by Sunday.

[alexcrichton]

I've posted a PR for this at https://github.com/rust-lang/rust/pull/47507