Crystal: Crystal Compiler can't target asm.js

[keplersj]

_crystal build hello.cr --single-module --target "asmjs-unknown-emscripten"_

No available targets are compatible with this triple, see -version for the available targets.
*raise<String>:NoReturn +70 [26303048]
*Crystal::TargetMachine::create<String, String, Bool>:LLVM::TargetMachine +219 [26303048]
*Crystal::Compiler#compile<Crystal::Compiler, Array(Crystal::Compiler::Source), String>:Crystal::Compiler::Result +214 [26303048]
*Crystal::Command::run<Array(String)>:(Crystal::Compiler::Result | File | Bool | CFileIO | Nil) +243 [26303048]
__crystal_main +10024 [26303048]
main +48 [26303048]
__libc_start_main +245 [26303048]
_start +41 [26303048]

_crystal --version_

Crystal 0.6.1 [48461ba] (Wed Mar  4 22:54:56 UTC 2015)

_cat hello.cr_

subject = "world"
puts "Hello #{subject}!"

I was attempting an experiment to build a simple Crystal script, compile it, and transpile the Object file into JavaScript using Emscripten. However when I was using emcc to transpile the .o created in the .crystal folder I had gotten this error::

WARNING  root: .crystal/home/kepler/Developer/CrystalJS/hello.cr/main.o is not valid LLVM bitcode
ERROR    root: no input files
note that input files without a known suffix are ignored, make sure your input files end with one of: ('.c', '.C', '.cpp', '.cxx', '.cc', '.c++', '.CPP', '.CXX', '.CC', '.C++', '.m', '.mm', '.bc', '.o', '.obj', '.dylib', '.so', '.a', '.ll', '.h', '.hxx', '.hpp', '.hh', '.H', '.HXX', '.HPP', '.HH')

After seeing this error I attempted to use the .bc file created in the same directory as well, and got this error:

WARNING: Linking two modules of different data layouts: '/home/kepler/.emscripten_cache/libc.bc' is 'e-p:32:32-i64:64-v128:32:128-n32-S128' whereas '/home/kepler/Developer/CrystalJS/.crystal/home/kepler/Developer/CrystalJS/hello.cr/main.bc' is 'e-i64:64-f80:128-n8:16:32:64'
WARNING: Linking two modules of different target triples: /home/kepler/.emscripten_cache/libc.bc' is 'asmjs-unknown-emscripten' whereas '/home/kepler/Developer/CrystalJS/.crystal/home/kepler/Developer/CrystalJS/hello.cr/main.bc' is 'x86_64-unknown-linux-gnu'
Value:   %4 = call %CFileIO @"*CFileIO::new<Pointer(Void)>:CFileIO"(i8* %3)
LLVM ERROR: Unrecognized struct value
Traceback (most recent call last):
  File "/usr/local/emsdk_portable/emscripten/master/emcc", line 1301, in <module>
    shared.Building.llvm_opt(final, link_opts)
  File "/usr/local/emsdk_portable/emscripten/master/tools/shared.py", line 1461, in llvm_opt
    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output
AssertionError: Failed to run llvm optimizations: 

I'll keep poking around to see if there is a fix I can do on my side, outside of the compiler, but I thought I'd pass this on to you guys to see what you might be able to do about it. There is of course the issue that asm.js is not an upstreamed backend in the LLVM yet, so I would wholly understand if you guys cannot do anything until then.

[jhass]

I tried fooling around a bit by compiling crystal against emscripten-fastcomp, but without much luck (it compiled, but it doesn't seem to have the target available). But even if we get past this point, in Crystal head pretty much every program depends on libevent, libpcl and libpcre, so these would need to be ported first anyway.

[ysbaddaden]

You may use the --prelude=empty to skip loading src/prelude.rb which will prevent loading the core library. You're down to the raw, with only the Crystal syntax available (not even puts is available), but it can help to start compiling some basic code, and to only enable some parts of the core library.

[keplersj]

I've made some progress fairly recently.

_cat number.cr_

1

_./bin/crystal build --mcpu asmjs --emit llvm-ir --verbose --single-module --prelude=empty number.cr_

Using compiled compiler at .build/crystal
'asmjs' is not a recognized processor for this target (ignoring processor)
'asmjs' is not a recognized processor for this target (ignoring processor)
'asmjs' is not a recognized processor for this target (ignoring processor)
'asmjs' is not a recognized processor for this target (ignoring processor)
'asmjs' is not a recognized processor for this target (ignoring processor)
'asmjs' is not a recognized processor for this target (ignoring processor)
cc -o "/Users/kepler/Developer/crystal-lang/crystal/number" "${@}"  -rdynamic  _main.o

_cat number.ll_

; ModuleID = 'main_module'
target datalayout = "e-m:o-i64:64-f80:128-n8:16:32:64-S128"
target triple = "x86_64-apple-macosx"

@ARGC_UNSAFE = global i32 0
@ARGV_UNSAFE = global i8** null

define internal i32 @__crystal_main(i32 %argc, i8** %argv) {
entry:
  store i32 %argc, i32* @ARGC_UNSAFE
  store i8** %argv, i8*** @ARGV_UNSAFE
  ret i32 1
}

declare i32 @printf(i8*, ...)

; Function Attrs: uwtable
define i32 @main(i32 %argc, i8** %argv) #0 {
entry:
  %0 = call i32 @__crystal_main(i32 %argc, i8** %argv)
  ret i32 0
}

attributes #0 = { uwtable "no-frame-pointer-elim"="true" "no-frame-pointer-elim-non-leaf"="true" }

_emcc -o number.js -rdynamic number.ll_

warning: Linking two modules of different data layouts: '/Users/kepler/.emscripten_cache/asmjs/libc.bc' is 'e-p:32:32-i64:64-v128:32:128-n32-S128' whereas '/var/folders/kt/npxm5bgn7_97lg4r5k91p1bc0000gn/T/tmphmCu4C/number_0.o' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'

warning: Linking two modules of different target triples: /Users/kepler/.emscripten_cache/asmjs/libc.bc' is 'asmjs-unknown-emscripten' whereas '/var/folders/kt/npxm5bgn7_97lg4r5k91p1bc0000gn/T/tmphmCu4C/number_0.o' is 'x86_64-apple-macosx'

warning: Linking two modules of different data layouts: '/Users/kepler/.emscripten_cache/asmjs/dlmalloc.bc' is 'e-p:32:32-i64:64-v128:32:128-n32-S128' whereas '/var/folders/kt/npxm5bgn7_97lg4r5k91p1bc0000gn/T/tmphmCu4C/number_0.o' is 'e-m:o-i64:64-f80:128-n8:16:32:64-S128'

warning: Linking two modules of different target triples: /Users/kepler/.emscripten_cache/asmjs/dlmalloc.bc' is 'asmjs-unknown-emscripten' whereas '/var/folders/kt/npxm5bgn7_97lg4r5k91p1bc0000gn/T/tmphmCu4C/number_0.o' is 'x86_64-apple-macosx'

warning: incorrect target triple 'x86_64-apple-macosx' (did you use emcc/em++ on all source files and not clang directly?)

_cat number.js_

Huge file, showing only transpiled relevant Crystal code:

function ___crystal_main($argc,$argv) {
 $argc = $argc|0;
 $argv = $argv|0;
 var label = 0, sp = 0;
 sp = STACKTOP;
 HEAP32[31] = $argc;
 HEAP32[32] = $argv;
 return 1;
}
function _main($argc,$argv) {
 $argc = $argc|0;
 $argv = $argv|0;
 var label = 0, sp = 0;
 sp = STACKTOP;
 (___crystal_main($argc,$argv)|0);
 return 0;
}

[keplersj]

From what I can tell Emscripten comes with a version of musl for its libc, so Crystal should now be able to support it thanks to @ysbaddaden's posix project.

With an ABI it seems to me that the compiler might now have to be coded to switch between Emscripten LLVM and the regular System LLVM. Both have their own linker.

[keplersj]

I just realized that Emscripten comes with a tool to aid building projects with Makefiles. I got an interesting result after running Crystal through it. I'll look more into how the library dependencies should be handled soon.

_emmake make_

/usr/local/Cellar/emscripten/1.36.3/libexec/em++ -c -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc `/usr/local/bin/llvm-config-3.6 --cxxflags`
/usr/local/Cellar/emscripten/1.36.3/libexec/emcc  -fPIC   -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: warning for library: src/ext/libcrystal.a the table of contents is empty (no object file members in the library define global symbols)
CRYSTAL_CONFIG_PATH=`pwd`/src ./bin/crystal build  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib
WARNING:root:_main.o is not valid LLVM bitcode
WARNING:root:emcc: cannot find library "event"
WARNING:root:emcc: cannot find library "pcre"
WARNING:root:emcc: cannot find library "gc"
WARNING:root:emcc: cannot find library "pthread"
WARNING:root:emcc: cannot find library "iconv"
WARNING:root:emcc: cannot find library "dl"
--: /Users/kepler/.cache/crystal/crystal-run-macro-run-_Users_kepler_Developer_crystal-lang_crystal_src_ecr_process.cr.tmp: Permission denied
Error in ./src/compiler/crystal.cr:3: instantiating 'Crystal::Command:Class#run()'

Crystal::Command.run
                 ^~~

instantiating 'run(Array(String))'

in ./src/compiler/crystal/command.cr:38: instantiating 'Crystal::Command#run()'

    new(options).run
                 ^~~

in ./src/compiler/crystal/command.cr:54: instantiating 'init()'

        init
        ^~~~

in ./src/compiler/crystal/command.cr:150: instantiating 'Crystal::Init:Module#run(Array(String))'

    Init.run(options)
         ^~~

in ./src/compiler/crystal/tools/init.cr:41: instantiating 'Crystal::Init::InitProject#run()'

      InitProject.new(config).run
                              ^~~

in ./src/compiler/crystal/tools/init.cr:153: instantiating 'Array(Crystal::Init::View+:Class)#each()'

        views.each do |view|
              ^~~~

in ./src/array.cr:813: instantiating 'each_index()'

    each_index do |i|
    ^~~~~~~~~~

in ./src/array.cr:813: instantiating 'each_index()'

    each_index do |i|
    ^~~~~~~~~~

in ./src/compiler/crystal/tools/init.cr:153: instantiating 'Array(Crystal::Init::View+:Class)#each()'

        views.each do |view|
              ^~~~

in ./src/compiler/crystal/tools/init.cr:154: instantiating 'Crystal::Init::View+#render()'

          view.new(config).render
                           ^~~~~~

in ./src/compiler/crystal/tools/init.cr:131: instantiating 'to_s()'

        File.write(full_path, to_s)
                              ^~~~

in ./src/object.cr:75: instantiating 'String:Class#build()'

    String.build do |io|
           ^~~~~

in ./src/string.cr:234: instantiating 'String::Builder:Class#build(Int32)'

    String::Builder.build(capacity) do |builder|
                    ^~~~~

in ./src/string.cr:234: instantiating 'String::Builder:Class#build(Int32)'

    String::Builder.build(capacity) do |builder|
                    ^~~~~

in ./src/object.cr:75: instantiating 'String:Class#build()'

    String.build do |io|
           ^~~~~

in ./src/object.cr:76: instantiating 'to_s(String::Builder)'

      to_s io
      ^~~~

in macro 'embed' /Users/kepler/Developer/crystal-lang/crystal/src/ecr/macros.cr:81, line 1:

  1.     {{ run("ecr/process", "/Users/kepler/Developer/crystal-lang/crystal/src/compiler/crystal/tools/init/template/example_spec.cr.ecr", "__io__") }}
  2.   

    {{ run("ecr/process", "/Users/kepler/Developer/crystal-lang/crystal/src/compiler/crystal/tools/init/template/example_spec.cr.ecr", "__io__") }}
    ^

expanding macro
in macro 'embed' /Users/kepler/Developer/crystal-lang/crystal/src/ecr/macros.cr:81, line 1:

  1.     {{ run("ecr/process", "/Users/kepler/Developer/crystal-lang/crystal/src/compiler/crystal/tools/init/template/example_spec.cr.ecr", "__io__") }}
  2.   

    {{ run("ecr/process", "/Users/kepler/Developer/crystal-lang/crystal/src/compiler/crystal/tools/init/template/example_spec.cr.ecr", "__io__") }}
       ^~~

Error executing run: ecr/process "/Users/kepler/Developer/crystal-lang/crystal/src/compiler/crystal/tools/init/template/example_spec.cr.ecr" "__io__"

Got:



make: *** [.build/crystal] Error 1

[rdp]

I had the thought recently that it might be nice to target emscripten/asm.js with crystal :) (or the JVM? but that's a bit tricker). Interesting ideas...

[mverzilli]

Now with Web Assembly in the picture, does it still make sense to target asm.js? Unless someone still strongly believes otherwise, I'd vote to close this issue.

[keplersj]

@mverzilli It's the same backend

[mverzilli]

That was fast :). Thanks for the clarification!

[keplersj]

Of course! #3634 should still work with WebASM due to LLVM awesomeness, the only standing issue is the dependencies like Boehm

[catmando]

The ruby world now has a stable near perfect ruby -> js transpiler. Of course stdlib stuff like threads, filemgt, etc etc and a very fee language features are simply not possible. However the subset makes it possible to code complete websites in a single language.

A emscriptem targeted crystal subset would be a logical and highly desirable next step for a lot of people using opal and ruby.

@keplersj any further news on your efforts?

[keplersj]

Compiling the language with an asm.js or wasm backend isn't entirely too difficult. I imagine that most of the code from my asm.js branch should still be applicable. The difficult part comes in with the Crystal standard library and garbage collection. If you create the subset you're talking about @catmando it should be more than possible to compile and execute it.

[catmando]

@keplersj tx for quick reply. GC is written in C correct? So can it just also be run through emscriptem?

[refi64]

I don't believe so. The Boehm GC relies on a lot of platform-specific, and IIRC some of it is also assembler. You'd basically have to redo the whole thing.

[keplersj]

@catmando If you create your own prelude with the subset you imagine, this technique I use above with emscripten should work to output runnable asm.js or wasm. That technique above certainly isn't foolproof, but it doesn't require a custom build of the compiler. Could be worth a try, might be helpful for building that web-focus subset you imagine.

[rdp]

Maybe don't need GC because javascript land takes care of it for you? Not too familiar with asm.js... :)

[asterite]

The GC has very little to do here. The problem is fibers and context switching via inline assembler.

[RX14]

Neither wasm or asm.js well-support coroutines or multiple stacks, which makes porting fibers impractical. wasm has support for this timetabled, I think we'll be waiting for that and GC. And probably not asm.js - at least not with fibers.

[rdp]

I guess rust can target wasm, what does it do with threads, just run single threaded, anybody know? Also wonder how golang does GC in wasm if it targets it...

[mmKALLL]

I started speccing a VM for WASM written in Crystal, but compiling Crystal into WASM seems like a much more practical project. I'll be keeping an eye out on this topic, and would like to contribute later on. Are there any actionable plans brewing at the moment?

[RX14]

@mmKALLL no, we're waiting for wasm to mature.

[rdp]

Can we close this in favor of #829 since "asm.js is old and wasm is its enw replacement" maybe? :)

[keplersj]

I'm gonna say no. In an ideal world the Crystal compiler would be able to output both asm.js and wasm, letting the user decide which output they need. If a user still needs to support browsers pre-wasm, or even just older JavaScript runtimes that support ES5 (with or without asm.js optimizations), they should still be able to output their codebases.

[catmando]

FYI: I just put a discussion topic around this general topic: https://forum.crystal-lang.org/t/crystal-js-transpiler/903