Crystal: TCPSocket error: Cannot assign requested address

[nsweeting]

Hi there,

I am currently developing a microservice for webhook delivery. Its just a basic Redis-backed queue system, but with a single job to execute - making an HTTP post request. I'm hitting some kind of bottleneck though. I'm thinking it might be an OS issue (Ubuntu 16 btw). Anyways - here is the backtrace:

Error connecting to 'localhost:3000': Cannot assign requested address (Errno)
[5739278] *raise<Exception+>:NoReturn +78
[5885161] *TCPSocket#initialize<String, Int32, (Float64 | Nil), (Float64 | Nil)>:Nil +745
[5884092] *HTTP::Client#socket:(OpenSSL::SSL::Socket+ | TCPSocket+) +140
[5883120] *HTTP::Client#exec_internal<HTTP::Request>:HTTP::Client::Response +32
[5870615] *HTTP::Client::exec<String, String, Nil, Nil, Nil>:HTTP::Client::Response +1383
[5867113] ~procProc(Nil) +249
[4693552] *Fiber#run:(IO::FileDescriptor | Nil) +64

I have been able to reproduce the error using the code below. The error occurs at almost exactly the same spot everytime - around the 28000'th post request.

require "redis"
require "http"
require "kemal"

logging false
post "/" { "test" }
spawn { Kemal.run }

total = 100_000
redis = Redis.new
total.times { redis.lpush("testqueue", "test") }

posters = [] of Poster
5.times do
  poster = Poster.new
  posters << poster
  poster.start
end

sleep

class Poster
  def start
    spawn do
      redis = Redis.new
      loop do
        pop = redis.brpop(["testqueue"], 2).as(Array(Redis::RedisValue))
        next unless pop.size == 2
        response = HTTP::Client.post("http://localhost:3000/")
        puts response.success?
      end
    end
  end
end

Any help would be appreciated.

[sdogruyol]

Why do you need to spawn for Kemal.run? And why there is a sleep?

[nsweeting]

Sleep is there because poster.start is spawning a 'worker' for post requests. Without it, the script would terminate before any work is completed. It appeared as though just Kemal.run would stall the script.

[luislavena]

Hello @nsweeting, if I'm not mistaken, you're simply exhausting available file/socket descriptors of your system, which is why is finding it unable to establish a new connection.

On each HTTP::Client.post call, a new connection is established, which might not be GC'ed until later.

With minor modification and using connection/pool and a pool of HTTP::Client, I was able to run the script (and of course, drive my redis instance crazy due BRPOP happening that fast).

If the webhook nature of your microservice requires to connect to different hosts depending on the value obtained from Redis, I would recommend combine do a hash of pools to keep up multiple _keep-alive_ connections to the different servers to avoid the TCP/TLS handshake, exhaust IO descriptors and stressing the GC.

Hope that helps.

[nsweeting]

Hi @luislavena

Yeah - that helped a bunch. Thank you very much. I was already using connection/pool for Redis. Worked like a charm!