Crystal: cannot fork, out of memory when compiling

[styler1972]

I was able to get this to compile and run, and then it stopped compiling and gave me the following error.

Previous output:

Hello world! The time is 2017-06-29 22:43:48 +0000HTTP::Headers{"HOST" => "192.217.123.137:8081"}#<HTTP::Request:0x556357bc8900>

Error:

crystal# crystal build main.cr
Unhandled exception in spawn:
fork: Cannot allocate memory (Errno)
0x55b5be: ??? at ??
0x11184c5: ??? at ??
0x1514ea8: ??? at ??
0x55e178: ??? at ??
0x0: ??? at ??
Unhandled exception in spawn:
fork: Cannot allocate memory (Errno)
0x55b5be: ??? at ??
0x11184c5: ??? at ??
0x1516fc8: ??? at ??
0x55e178: ??? at ??
0x0: ??? at ??

Source:

require "http/server"

server = HTTP::Server.new("0.0.0.0", 8080) do |context|
  context.response.content_type = "text/plain"
  context.response.print "Hello world! The time is #{Time.now}"

  context.response.print context.request.headers

  context.response.print context.request
end

puts "Listening on http://127.0.0.1:8080"
server.listen

Thanks! Great work!

Crystal 0.22.0 [3c71228] (2017-04-20) LLVM 3.5.0

[refi64]

Out of curiosity:

• Does this work with the latest version?
• What's the output of free -h before and after compiling?

[akzhan]

Aside this issue:

• LLVM 3.5 is buggy anyway (https://github.com/crystal-lang/crystal/issues/4104)
• Crystal 0.23 just released.

[styler1972]

I'll check this out now! Thanks guys :)

[styler1972]

@kirbyfan64

# free -h
              total        used        free      shared  buff/cache   available
Mem:           484M         97M        222M        3.2M        164M        369M
Swap:            0B          0B          0B

and

Crystal 0.23.0 [3c3d3e2] (2017-06-28) LLVM 3.8.1

will let you know if i see this again

[styler1972]

Whoops! just happened, but worse this time.

# crystal build main.cr Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x1555225: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ?? Unhandled exception in spawn: fork: Cannot allocate memory (Errno) 0x5a9495: ??? at ?? 0x1163325: ??? at ?? 0x155bc55: ??? at ?? 0x5acd28: ??? at ?? 0x0: ??? at ??

[styler1972]

Seems to be happening more with crystal 0.23.0 and llvm 3.8.1

[styler1972]

I actually can't get it to successfully compile at all, even after a reboot

[styler1972]

@akzhan @kirbyfan64 you both have invitations waiting for you if you want to check out the new source code

[bew]

What is the output of free -h after a fresh reboot?

[sdogruyol]

@styler1972 how about using swap memory?

[straight-shoota]

A total of 484 MB memory with no swap could be a bit to low for Crystal compiler. You're probably gonna need more space.

[refi64]

Yeah, you definitely don't have enough memory for compiling Crystal code. Type inference is expensive!

Try making sure nothing else is open before compiling, too. It doesn't look like you have anything else open, but Chrome/Electron-based applications can use quite a bit.

[matiasgarciaisaia]

@mverzilli I'm not sure this is a bug. Compiling Crystal programs require considerable amounts of memory - that's a trade-off we make for awesomeness.

We always _want_ to lower the memory amount needed, but as long as we can still compile the compiler in a 32bit environment (~4GB), I wouldn't consider this a major bug we should fix ASAP.

TL;DR: I'd remove this from 0.23.1

[mverzilli]

Cool. I thought the problem was worsened by 0.23, but I guess you're right, it doesn't merit a hotfix.

[matiasgarciaisaia]

I _think_ using the newer LLVM 3.8 implies you need more memory to compile, so that's a good reason for the bug to happen more often with 0.23.0, but I don't thinks it's enough for a hotfix anyway.

I'd think about that if there's any pretty common scenario in which this change makes things really worse (say, previously compiling apps now hitting Docker Hub's automated build memory limits), but not as is.

PRs and suggestions on improvements on memory footprint are always welcome, anyways!

BTW, @styler1972 building with --no-debug has proven to lower the memory footprint. You could try that as a workaround if you ran out of alternatives.

[rdp]

If it's a VM maybe you can add some swap? Anyway (as a note to followers) building without "--release" builds using much less RAM FWIW. Also you might be able to build "for it" on a different box and copy it over there. Though maybe there's something that could be done internally to help as well. GL!

[asterite]

Crystal needs a lot of memory to compile, and we know it. There's nothing immediate that we can do here, so I'm closing this.