Crystal: Fiber#yield not completing spawn with HTTP::Client.get

[Daniel-Worrall]

When Fiber.yield'ing to spawns that include http calls, the program ends before they are completed. Replacing it with sleep works, but is not useful for a runtime script that doesn't want to stay open.

The only other option I have at the moment is to use a buffered channel and send data on each spawn until they're all finished. Haven't been able to get this to work.

Reproducible Code

require "http/client"
spawn do 
  response = HTTP::Client.get "https://google.com"
  print 1
end
Fiber.yield

Exits without printing.

[Exilor]

How about this?

require "http/client"
ch = Channel(Nil).new
spawn do
  response = HTTP::Client.get "https://google.com"
  print 1
  ch.send(nil)
end
ch.receive

[jhass]

Yes, this is sort of intentional. The HTTP call is doing IO, so the spawned fiber yields as that IO blocks. Given you didn't suspend the main fiber indefinitely, just once to give the other fiber a chance to run, it continues now and the only thing it has left to do is shutdown the program, so that happens.

The example and issue description is a bit reduced to fully address how to properly solve your problem, but your striked out note and @Exilor's comment goes into the right direction, you want to block the main fiber until some kind of completion signal or use it as the fiber to do the processing, aggregation and/or presentation of your results, whatever the last step in your pipeline is. Usually that part can also easily decide when to stop the program, by then just breaking the processing loop.

[straight-shoota]

Also: Fiber.yield gives the scheduler a chance to resume other fibers, but it doesn't make any guarantee that a specific fiber is resumed.

In this case, it will probably switch to the spawn fiber. But that ends up waiting for IO and then switching to the main fiber which exits.

[asterite]

Also this behavior is mentioned in https://crystal-lang.org/reference/guides/concurrency.html

I hope the given answers are helpful. Otherwise please reopen!

[Daniel-Worrall]

Thank you for the comments.

It wasn't clear that Fiber.yield didn't guarantee fiber execution.

My use case is spawning multiple get requests so I'll resolve this with a channel and receiving for each spawn.

require "http/client"
ch = Channel(Nil).new
links = ...
links.each do |link|
  spawn do
    HTTP::Client.get link
    ch.send(nil)
  end
end
links.size.times { ch.receive }

[bcardiff]

@Daniel-Worrall checking parallel macro at https://crystal-lang.org/api/0.31.1/toplevel.html#parallel(*jobs)-macro might be useful

[Daniel-Worrall]

@bcardiff I don't believe this will be useful for this use-case as I don't care about the order of execution, but it's a useful tool I'll keep in mind in the future. Thanks!

[asterite]

as I don't care about the order of execution

But parallel doesn't care about order of execution. However you get the results in the same order you pass the tasks to parallel.

[Daniel-Worrall]

I don't care about getting the results in the same order either