Crystal: Unhandled exception: HTTP::Client::Response#body_io cannot be nil (NilAssertionError)

[arloan]

code:

require "log"
require "http/client"

url = "https://www.apple.com/v/mac-mini/j/images/overview/hero__x8ruukomx2au_large.jpg"
r = HTTP::Client.get url
if !r.success?
  Log.error { "Cannot retrieve image data: [%d] %s" % [r.status_code, r.body] }
  bytes = Bytes.empty
else
  n = r.headers["Content-Length"].to_i
  Log.info { "server responded with content of %d bytes" % n }
  bytes = Bytes.new(n)
  Log.info { "reading content to memory ..." }
  r.body_io.read_fully(bytes)
  Log.info { "#{bytes.size} bytes read." }
end
Log.info { bytes.inspect }

result:

2020-11-11T09:53:24.177622Z   INFO - server responded with content of 77573 bytes
2020-11-11T09:53:24.177765Z   INFO - reading content to memory ...
Unhandled exception: HTTP::Client::Response#body_io cannot be nil (NilAssertionError)
  from ../../../usr/share/crystal/src/http/client/response.cr:9:3 in 'body_io'
  from main.cr:15:3 in '__crystal_main'
  from ../../../usr/share/crystal/src/crystal/main.cr:105:5 in 'main_user_code'
  from ../../../usr/share/crystal/src/crystal/main.cr:91:7 in 'main'
  from ../../../usr/share/crystal/src/crystal/main.cr:114:3 in 'main'
  from __libc_start_main
  from _start
  from ???

> uname -a
Linux 487e0793cd90 5.4.0-1019-gcp #19-Ubuntu SMP Tue Jun 23 15:46:40 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

> crystal -v
Crystal 0.35.1 [5999ae29b] (2020-06-19)

LLVM: 8.0.0
Default target: x86_64-unknown-linux-gnu

[bararchy]

@arloan use the secret consume_body_io on the response, if there is a body_io it will consume it, and allow you to just handle the body instead.

[arloan]

@arloan use the secret consume_body_io on the response, if there is a body_io it will consume it, and allow you to just handle the body instead.

In actual codes, I need to retrieve binary data (say, an image) from the response, consume_body_io uses IO#gets_to_end, maybeit's not suitable for binary data IMO.

[asterite]

I think we should introduce two separate HTTP::Client::Response types: one streaming, one consumed. That way there won't be any confusion anymore. I don't know why we made a single class behave in two different ways...

[bararchy]

@asterite or even we can go with

non stream:

response = HTTP::Client.get(...)

stream:

HTTP::Client.get do |response|

end

[asterite]

@bararchy That's how it currently works. The problem is that you have to remember to use body in the non-streaming version, and body_io in the streaming version. If you use body_io in the non-streaming version you get a runtime error which could easily be a compile-time error instead.

What I'm suggesting is that in both cases the body is accessed with body: in the non-streaming version it's a string; in the streaming version it's an IO.

I'll try to see if I can implement this.

I'd like this change to go before 1.0 because I think right now this is a bit messy.

[asterite]

I sent an RFC: https://github.com/crystal-lang/crystal/issues/9901

[arloan]

I updated my first comment to refelct my actual needs for downloading an image, which is not suitable to use consume_body_io. Would any one kindly provide another workable way to download a binary file from the Web? I'm stucked here, thanks so much.

[Blacksmoke16]

@arloan Do you just want to save it to a file or ?

If it's the former could you not just do like:

require "http/client"

url = "https://www.apple.com/v/mac-mini/j/images/overview/hero__x8ruukomx2au_large.jpg"
HTTP::Client.get url do |response|
  File.open("out.jpg", "w") do |file|
    IO.copy response.body_io, file
  end
end

EDIT: The issue here is that .body_io only "exists" when you use the block version of a request. Otherwise you need to use .body.

[arloan]

I think we should introduce two separate HTTP::Client::Response types: one streaming, one consumed. That way there won't be any confusion anymore. I don't know why we made a single class behave in two different ways...

u'r a genius, it works! I just reviewed the API document and I can't believe that I missed the point that the body_io only exists in the block version, what a shame for me.

[asterite]

@arloan Nah, given that Crystal is a compiled language the compiler should be helping you here. It's really trivial to make body_io not be there when you are not using the block form. This is what's being discussed in #9901