Crystal: GC Warning: Repeated allocation of very large block

[MrSorcus]

Bug or feature?

require "http/server"

server = HTTP::Server.new(8080) do |context|
  context.response.content_type = "text/plain"
  context.response.print "Hello world! The time is #{Time.now}"
end

puts "Listening on http://127.0.0.1:8080"
server.listen

Compile & run.

Then run this command:
while true; do nc 127.0.0.1 8080 < /dev/zero ; done

Output:

laptop% ./test                                  
Listening on http://127.0.0.1:8080
GC Warning: Repeated allocation of very large block (appr. size 67112960):
        May lead to memory leak and poor performance
GC Warning: Failed to expand heap by 2149683200 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
GC Warning: Out of Memory! Heap size: 2070 MiB. Returning NULL!
GC Warning: Failed to expand heap by 2149076992 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
GC Warning: Out of Memory! Heap size: 2070 MiB. Returning NULL!
GC Warning: Failed to expand heap by 2149076992 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
GC Warning: Out of Memory! Heap size: 2070 MiB. Returning NULL!
GC Warning: Failed to expand heap by 2149076992 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
GC Warning: Out of Memory! Heap size: 2070 MiB. Returning NULL!
GC Warning: Failed to expand heap by 2149076992 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
GC Warning: Out of Memory! Heap size: 2070 MiB. Returning NULL!
GC Warning: Failed to expand heap by 2149076992 bytes
GC Warning: Failed to expand heap by 2147487744 bytes
...

But nginx working and returning HTTP/1.1 400 Bad Request

Version:

Crystal 0.20.5 (2017-01-25)

[luislavena]

Hello,

This is happening due a combination of string allocations and a tight loop, that is not allowing the GC to kick in.

You can find more details in #3997.

---

Now, compared with nginx, the main difference is that nginx seems to be discarding the nc request for being invalid while Crystal might be accepting it, exhausting available ports and memory.

[sdogruyol]

This is not labeled. We're aware that the current Boehm GC is not optimized for this case however overally it does a good job :+1:

[RX14]

I think this should be fixed in current crystal, since we've added limits for HTTP io ops.

[RX14]

Yup, this was fixed in https://github.com/crystal-lang/crystal/pull/4428.

[asterite]

@RX14 Thank you! :)