Emscripten: Robust emsdk builds

[kripken]

We need to figure out a robust solution to emsdk build generation. Over the last months we had a few weeks where this was broken, and users could only get incoming and build it manually.

The main thing we need to ensure is that more than one person can do this, unlike now where only @juj can.

Some thoughts:

• Aside from our own emsdk efforts, others exist like OS X homebrew and a linux docker image. I also constantly have a local linux build myself. Maybe we can collaborate there, make it easy to upload builds the emsdk can use?
• Another option discussed has been to use a port of llvm+clang to asm.js or wasm, which would be a portable build usable anywhere. However, it would be slower.

[saschanaz]

wasm build will always be valuable as it will be able to run on arbitrary OS, including ones currently unsupported (ChromeOS? and older/newer versions of Ubuntu). And it will better match npm (#5774). (npm install llvm!)

[juj]

Emsdk build generation is being migrated to Amazon EC instances, so one solution might be to hand off credentials to those to more people. Would that be enough of a solution?

[kripken]

@saschanaz : yeah, I think the wasm build could be useful for a bunch of things. What worries me there is (1) it'll be slower, but more convenient, so people might end up using it a lot, and (2) it's another build to maintain and test (easier than each of the existing native builds, but still, another build target).

@juj : I think that would be enough, unless it requires a windows machine to do something for the windows build - like debug it when building fails (I don't have one of those currently, for example). What would those people be able to do, and what would the interface for them be?

[saschanaz]

it'll be slower, but more convenient, so people might end up using it a lot

npm supports a post-installation hook where we can warn that this wasm-based package can be slower.

[kripken]

Another important thing for robustness is to monitor that our emsdk builds are being generated (there are reports right now of new breakage in #5713). A dashboard perhaps? Anyhow, this is lower priority than the other stuff here, probably.

[saschanaz]

CircleCI has Git Tag Job Execution where we can automatically trigger build jobs, exactly what the current bots do. My expectation is that the CI can build each tools within its 50 min limit but currently I don't have enough knowledge about cmake to actually try it...

(Edit: talking about wasm build)

[kripken]

@saschanaz Oh, nice! Yeah, that's worth trying.

[lastmjs]

Can I help with this? I'm interested specifically in helping with a WASM build of Emscripten. I know very little of the internals of Emscripten, but I've been messing around with Emscripten's WASM compilation as a user. I could really use a WASM build for a Node.js based project where I need dynamic compilation of C/C++ and other files, unfortunately it takes forever to install the Emscripten WASM toolchain, and I can't control the machines of my users. Having a WASM build should fix all of that I'm hoping. If anyone could point me in the right direction and explain the complexity of this, I'll see if I have the time.

[kripken]

@lastmjs glad to hear you're interested to help :)

Overall, to get a wasm/js build of emscripten so that we can run it through node would need a few things:

• Emscripten ports of the LLVM tools: llc, clang, llvm-link, opt. That means getting them to build using emscripten. The LLVM build system is a little tricky since it builds stuff it runs in the build process (tablegen), but this has been done (recently, clang-in-browser).
• A solution for the python stuff. One option is to just run the python interpreter in wasm. We may need to add some new code for that as it likely uses system APIs we don't support yet, and may need the emterpreter for synchronous code, but it should be doable. Another option is to just assume python is present, which is usually easy to install if it isn't.

Btw, are you building emscripten from source? It shouldn't take a long time to install (which you mentioned) if there is a binary build and it's just downloaded. If that's not working properly for you then could be a bug there, but it should be working on all major platforms.

[lastmjs]

I meant building from source, not installing :) Okay, so Emscripten ports of the LLVM tools:

• [x] llc
• [x] clang
• [x] llvm-link
• [ ] opt

Are there any other examples of this besides https://tbfleming.github.io/cib/ ? Also, is that an exhaustive list of the LLVM tools that would need to be ported to Emscripten? Would I be compiling them to WASM or just JavaScript? Where should I be hosting these changes (repo)? Is there a better place for me to ask questions, I imagine I'll have quite a few more

[lastmjs]

My plan right now is to play around with https://github.com/kripken/emscripten-fastcomp and try to build https://github.com/kripken/emscripten-fastcomp-clang with Emscripten. I can fork those repositories and make any changes necessary to them and pull request back. Is this a good direction?

[lastmjs]

I'm here in building clang with Emscripten:

emcmake cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86;JSBackend" -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DCLANG_INCLUDE_TESTS=OFF

What should I have for the option DLLVM_TARGETS_TO_BUILD?

[lastmjs]

The command above failed on the first try, but succeeded on subsequent tries. Now I've run into this issue:

[lastmjs@lastmjs build]$ emmake make
[  1%] Built target LLVMDemangle
[  1%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Path.cpp.o
In file included from /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Path.cpp:1048:
/home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Unix/Path.inc:390:13: error: no member named 'f_flags' in 'statvfs'
  return !!(STATVFS_F_FLAG(Vfs) & MNT_LOCAL);
            ^~~~~~~~~~~~~~~~~~~
/home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Unix/Path.inc:103:35: note: expanded from macro 'STATVFS_F_FLAG'
#define STATVFS_F_FLAG(vfs) (vfs).f_flags
                            ~~~~~ ^
/home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Unix/Path.inc:390:35: error: use of undeclared identifier 'MNT_LOCAL'
  return !!(STATVFS_F_FLAG(Vfs) & MNT_LOCAL);
                                  ^
2 errors generated.
ERROR:root:compiler frontend failed to generate LLVM bitcode, halting
make[2]: *** [lib/Support/CMakeFiles/LLVMSupport.dir/build.make:2464: lib/Support/CMakeFiles/LLVMSupport.dir/Path.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:431: lib/Support/CMakeFiles/LLVMSupport.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

I assume there is an issue with including statvfs, any pointers here?

[lastmjs]

Looks like the issue above comes down to emscripten-fastcomp/lib/Support/Unix/Path.inc:

#if defined(__NetBSD__)
#define STATVFS_F_FLAG(vfs) (vfs).f_flag
#else
#define STATVFS_F_FLAG(vfs) (vfs).f_flags
#endif

I guess my system (Arch Linux) is failing the __NetBSD__ check, and so it assumes the statvfs struct should have an f_flags member, but it does not. So a quick fix is to just get rid of the s. This should probably be addressed with less of a hack:

#if defined(__NetBSD__)
#define STATVFS_F_FLAG(vfs) (vfs).f_flag
#else
#define STATVFS_F_FLAG(vfs) (vfs).f_flag
#endif

[lastmjs]

That still leaves the following issue:

[lastmjs@lastmjs build]$ emmake make
[  1%] Built target LLVMDemangle
Scanning dependencies of target LLVMSupport
[  1%] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/Path.cpp.o
In file included from /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Path.cpp:1049:
/home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Unix/Path.inc:392:35: error: use of undeclared identifier 'MNT_LOCAL'
  return !!(STATVFS_F_FLAG(Vfs) & MNT_LOCAL);
                                  ^
1 error generated.
ERROR:root:compiler frontend failed to generate LLVM bitcode, halting
make[2]: *** [lib/Support/CMakeFiles/LLVMSupport.dir/build.make:2464: lib/Support/CMakeFiles/LLVMSupport.dir/Path.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:431: lib/Support/CMakeFiles/LLVMSupport.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

Which I hacked by inlining the MNT_LOCAL value from sys/mount.h. I couldn't figure out how to include the header correctly...even when I explicitly included the header outside of any if statements, MNT_LOCAL was not defined. Here's the temporary fix in /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/lib/Support/Unix/Path.inc:

#else
  return !!(STATVFS_F_FLAG(Vfs) & 0x00001000);
#endif

[lastmjs]

So, I think I've been doing this wrong. I'm following the advice here and here, and my new command is:

cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="X86;JSBackend" -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DCLANG_INCLUDE_TESTS=OFF -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/home/lastmjs/development/emscripten/cmake/Modules/Platform/Emscripten.cmake

and then:

make

I'm now at this halting issue:

[lastmjs@lastmjs build]$ make
[  1%] Built target LLVMDemangle
[  6%] Built target LLVMSupport
[  6%] Built target LLVMTableGen
[  8%] Built target obj.llvm-tblgen
[  8%] Built target llvm-tblgen
[  8%] Built target CONFIGURE_LLVM_NATIVE
[  8%] Building libLLVMTableGen for native TableGen...
make[3]: *** No rule to make target 'LLVMSupport'.  Stop.
make[2]: *** [CMakeFiles/NATIVE_LIB_LLVMTABLEGEN.dir/build.make:61: LIB_LLVMTABLEGEN] Error 2
make[1]: *** [CMakeFiles/Makefile2:196: CMakeFiles/NATIVE_LIB_LLVMTABLEGEN.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

I was actually getting to this same issue with the previous emcmake and emmake commands. Any pointers would be useful, it looks like there is no LVMSupport file in the directory: /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/build/NATIVE, and build.make is trying to execute this command:

cd /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/build/NATIVE && $(MAKE) -C /home/lastmjs/development/fastcomp-emscripten-build/emscripten-fastcomp/build/NATIVE LLVMSupport

[lastmjs]

I've cleared my build folder and removed the X86 target. If we don't need the X86 target, then my problem above is gone. Let me know if this command is adequate and I'll keep working on the new errors:

cmake .. -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="JSBackend" -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF -DCLANG_INCLUDE_TESTS=OFF -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/home/lastmjs/development/emscripten/cmake/Modules/Platform/Emscripten.cmake

[lastmjs]

I had to throw in SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11") into the emscripten-fastcomp/CMakeLists.txt file, and get rid of the native compilation, since we don't need it. Comment out emscripten-fastcomp/CMakeLists.txt line 584, 585, and 644 so that there is no more native target. My build is at 73% and going strong :)

[lastmjs]

I keep running into errors like the following:

[ 90%] Linking CXX executable ../../../../bin/clang-import-test.js
Unsupported:   %263 = icmp slt i80 %262, 0
LLVM ERROR: Signed comparisons not yet supported for integer types larger than 64 bits
Traceback (most recent call last):
  File "/home/lastmjs/development/emscripten/emcc.py", line 2812, in <module>
    sys.exit(run())
  File "/home/lastmjs/development/emscripten/emcc.py", line 1639, in run
    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
  File "/home/lastmjs/development/emscripten/tools/shared.py", line 2009, in emscripten
    call_emscripten(cmdline)
  File "/home/lastmjs/development/emscripten/emscripten.py", line 2361, in _main
    temp_files.run_and_clean(lambda: main(
  File "/home/lastmjs/development/emscripten/tools/tempfiles.py", line 93, in run_and_clean
    return func()
  File "/home/lastmjs/development/emscripten/emscripten.py", line 2366, in <lambda>
    DEBUG=DEBUG,
  File "/home/lastmjs/development/emscripten/emscripten.py", line 2272, in main
    temp_files=temp_files, DEBUG=DEBUG)
  File "/home/lastmjs/development/emscripten/emscripten.py", line 94, in emscript
    backend_output = compile_js(infile, settings, temp_files, DEBUG)
  File "/home/lastmjs/development/emscripten/emscripten.py", line 123, in compile_js
    shared.jsrun.timeout_run(subprocess.Popen(backend_args, stdout=subprocess.PIPE, universal_newlines=True), note_args=backend_args)
  File "/home/lastmjs/development/emscripten/tools/jsrun.py", line 20, in timeout_run
    raise Exception('Subprocess "' + ' '.join(note_args) + '" failed with exit code ' + str(proc.returncode) + '!')
Exception: Subprocess "/home/lastmjs/development/fastcomp-manual-build/emscripten-fastcomp/build/bin/llc /tmp/tmp56ToWx/clang-import-test.bc -march=js -filetype=asm -o /tmp/tmphHQWid.4.js -emscripten-stack-size=5242880 -O2 -emscripten-global-base=8 -emscripten-no-exit-runtime" failed with exit code 1!
make[2]: *** [tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/build.make:142: bin/clang-import-test.js] Error 1
make[1]: *** [CMakeFiles/Makefile2:11635: tools/clang/tools/clang-import-test/CMakeFiles/clang-import-test.dir/all] Error 2
make: *** [Makefile:152: all] Error 2

I've tracked down the issues slightly, anything I do would be hacks, insight would be appreciated. If I comment out the contents of emscripten-fastcomp/tools/clang/tools/diagtool/ShowEnabledWarnings.cpp, emscripten-fastcomp/tools/clang/tools/clang-import-test/clang-import-test.cpp, emscripten-fastcomp/tools/clang/tools/clang-format/ClangFormat.cpp, emscripten-fastcomp/tools/clang/tools/arcmt-test/arcmt-test.cpp, and emscripten-fastcomp/tools/clang/tools/c-arcmt-test/c-arcmt-test.cpp, I can proceed.

[lastmjs]

Alright, I keep running into the above issue over and over again. I'm 90% of the way there, but I would love some help on fixing this issue before I proceed. If even just knew the int datatype that was greater than 64 bits, that would help. Looks similar to this: https://github.com/kripken/emscripten/issues/4540

I might as well add that as soon as that issue is fixed, I'll be facing a symbol multiply defined error, as detailed here: https://github.com/kripken/emscripten/issues/6239

[kripken]

About the x86 backend: we don't really need it. The crucial thing is the JS backend. (however, having the x86 backend would be nice, since then we could build native executables like the js optimizer normally - but, we can do some more work and build them to wasm instead)

About Signed comparisons not yet supported for integer types larger than 64 bits - that's hitting a JSBackend limitation (the legalization is limited). I'm surprised we hit this on LLVM - we build so many other codebases and never saw it (we do see it on arbitrary bitcode, not coming from clang). The options here are:

• Look into fixing that limitation in the JSBackend.
• See if it's easy to work around in the LLVM source.
• If neither of those, then we may want to change our focus to porting the LLVM wasm backend, which has full legalization. The downsides to doing that are that it isn't yet our default compiler backend since it isn't ready for that yet, and also it means no asm.js support. But, eventually those should be fixable.

[lastmjs]

Since we control the source, kind of, for LLVM it might be best to track that down and change it. Any ideas you have about where those large ints might be hidden before I dig in?

[lastmjs]

Unless fixing that limitation in the JSBackend isn't too difficult. What's the process there?

[juj]

Can I help with this? I'm interested specifically in helping with a WASM build of Emscripten. I know very little of the internals of Emscripten, but I've been messing around with Emscripten's WASM compilation as a user.

Nice to see you working on this front, more developers working on in-browser compilation of C++ to Wasm is good. This bug entry discusses the maintenance of the native Emscripten SDK and not running Emsdk inside a browser, so let's keep this one clear for that, and collaborate on Emscripten build of Emscripten in a separate tracked entry. I think there was one already either in GitHub or emscripten-discuss mailing list.

[kripken]

@lastmjs

To find which function it's in, you can run with EMCC_DEBUG=1, then the bitcode file is saved in the temp dir. Then do llvm-dis on it, and you can search for that line, then see the function (search backwards for define).

Fixing the JSBackend would involve finding where that error is emitted (one of the passes under lib/Target/JSBackend/NaCl/, can search for the error text), and then implementing the missing thing there. In this case, it's probably going to be implementing lowering of signed comparisons of things like 150 bit integers into 32-bit ones. Not super-hard, but needs to be done carefully with all the corner cases.

[lastmjs]

@juj I can't seem to find the GitHub issue, do you know where it is, or perhaps I should open a new issue?

[kripken]

@lastmjs I can't seem to find a specific issue for that either - might as well open a new one I think.

[andrewrk]

I'm trying to get Zig built with emscripten, and to do that the first step is the LLVM dependency. I get stuck at the error noted in https://github.com/emscripten-core/emscripten/issues/6015#issuecomment-378127510. My cmake line is:

emconfigure cmake .. -DCMAKE_INSTALL_PREFIX=$HOME/local/llvm8-wasm -DCMAKE_PREFIX_PATH=$HOME/local/llvm8-wasm -DCMAKE_BUILD_TYPE=Release -DLLVM_TARGETS_TO_BUILD="WebAssembly" -DLLVM_ENABLE_LIBXML2=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TESTS=OFF

In file included from /home/andy/downloads/llvm-project/llvm/lib/Support/Path.cpp:1110:
/home/andy/downloads/llvm-project/llvm/lib/Support/Unix/Path.inc:410:13: error: no member named
      'f_flags' in 'statvfs'
  return !!(STATVFS_F_FLAG(Vfs) & MNT_LOCAL);
            ^~~~~~~~~~~~~~~~~~~
/home/andy/downloads/llvm-project/llvm/lib/Support/Unix/Path.inc:91:35: note: expanded from macro
      'STATVFS_F_FLAG'
#define STATVFS_F_FLAG(vfs) (vfs).f_flags
                            ~~~~~ ^
/home/andy/downloads/llvm-project/llvm/lib/Support/Unix/Path.inc:410:35: error: use of undeclared
      identifier 'MNT_LOCAL'
  return !!(STATVFS_F_FLAG(Vfs) & MNT_LOCAL);
                                  ^
2 errors generated.
shared:ERROR: compiler frontend failed to generate LLVM bitcode, halting

This is on NixOS with llvm 8.0.1, emcc 1.38.28.

[kripken]

@andrewrk cool, always interesting to hear about a new language here!

I'd actually suggest not building fastcomp - it's only guaranteed to work on clang, and not other LLVM-using compilers. Also, a more recent upstream LLVM is more likely to build out of the box on recent compilers, so it might not hit the errors you are seeing. We intend to switch to upstream LLVM pretty soon, we'll probably have a blogpost about that next week. Meanwhile, you can just get normal upstream LLVM and build it according to their instructions (making sure to build clang and lld, using something like -DLLVM_ENABLE_PROJECTS=lld;clang'). Then just point emscripten to that.

[andrewrk]

Thanks for the tips @kripken! Just to be clear I wasn't using fastcomp - I don't even know what that is. I see now that it's mentioned above in this issue. I'll try with llvm trunk.

[andrewrk]

@kripken I tried with LLVM trunk (git mono repo sha 046d49a8dcbbb99596ace2592aa0386ddf2f6bce) and got the same compile error. Is #6432 a better place to discuss this?

[kripken]

@andrewrk that sounds like a bug in upstream llvm then, or it's not compatible with your compiler - which version is it? (recent clang should work, as should recent gcc, but old ones, or bleeding-edge versions of gcc, may be risky in my experience)

[andrewrk]

See https://github.com/emscripten-core/emscripten/issues/6432#issuecomment-503771125

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[kripken]

The original issue here has been fixed by the now over 1 year old chromium build infrastructure that powers the emsdk.

There are some other issues discussed in the comments, though - please open new issues for those if still relevant.