Crystal: Array#pack and String#unpack

[chendo]

Are these going to be implemented as part of the stdlib? If so, would we like it to work the same way as the Ruby one? I'm willing into look into porting this over.

[asterite]

I'm not sure. Ruby's String#unpack is too dynamic. It returns an array of a mix of types, like String, Fixnum, etc. Then Array#pack assumes you have an array to pack, which means you had to allocate memory to create that array. In Crystal we try to avoid allocations by reading from IOs and writing to them without creating intermediate arrays.

I like what's said in the beginning of this article: "C programming language allows developers to directly access the memory where variables are stored. Ruby does not allow that. There are times while working in Ruby when you need to access the underlying bits and bytes. Ruby provides two methods pack and unpack for that."

But in Crystal you do have access to the underlying bits and bytes. You have pointer manipulation and types that map exactly to a group of bytes. So, as the article says, I believe these methods exist to make it possible (not easier) to do byte manipulation.

But... because these are generic and return dynamic content, and maybe allocate arrays, people often do C extensions instead to access the bytes. So it means (and in my case I actually had to rewrite a part of a library that used pack/unpack to a C extensions to speed up things) that these methods are slow. Or at least they could be faster.

What would be nice it maybe to have a macro that would generate code to pack/unpack data. But I don't know how would that work.

[jhass]

Could you imagine simply adding to_x methods to Slice instead? Like Slice#to_s(16) would return a hexstring, Slice#to_i(:little_endian), Slice#to_c_array... and so on. I didn't look into if that approach would be able to support all formats, and it sort of falls apart when you think about the way back, that would require a collections of helpers I guess.

[Exilor]

There could be a dedicated module like Math dealing with this kind of thing.

Or something like BigInt does. Not included by default but when it's required it adds a bunch of new methods to the Number family. That way it could be more "integrated" but optional.

I have a file that I include when I'm dealing with numbers at a low level: https://dl.dropboxusercontent.com/u/1335129/bitwise.cr
Not something I'd dare to make a pull request with though (Int32.new. I bet that would raise a few eyebrows) but you might find something useful in it.

[chendo]

@asterite That's fair enough. My end goal is to convert binary data to and from C structures. I've gotten somewhere by messing with Pointers and pointerof but it feels foreign to me at the moment. I think it does make sense for this to be another library to not bloat out the stdlib.

[asterite]

I actually think that something like what @jhass says will be in the std, it's just that right now we don't do much of this.

@Exilor: there's nothing wrong with defining new new for any class, as long as you use restrictions. For example if you require json we add Int32#new(pull : Json::PullParser), and so on for primitive types, Array and Hash, to support json deserialization in a very generic but powerful way.

[jwoertink]

Now that's it's been a few years, is there any chance of this being a thing, or maybe a way to handle this?

[straight-shoota]

This question has just recently come up again on chat. So it seems to be a constant topic for Ruby developers. Although I've personally never used it in Ruby and I can really never tell what that cryptic format string means. And honestly, I don't care.

In Crystal, we have a way better tool for solving the problems that pack/unpack solves: IO. You can do the same things as with pack/unpack and it even offers better readable code, type safety, and more performance and flexibility.

n = [65, 66, 67]

# Ruby
n.pack("ccc") #=> "ABC"

# Crystal
String.build do |io|
  n.each do |number|
    io.write_byte number.to_u8
  end
end #=> "ABC"

Sure, the Crystal implementation is a bit more verbose, but that only makes it that you can actually understand what the code is doing instead of having to decipher the meaning of a cryptic format string.

[jhass]

Have a look at IO::ByteFormat, IO#read_byte, IO#read_bytes, IO#write_byte, IO#write_bytes.

[jwoertink]

Yeah, I don't mind a little verbosity especially when it makes it clearer on what's going on. The one specifically I'm looking to do is [5].pack('e*').bytes which returns an array of ints. I found this but I still haven't managed to figure out how to get what I'm looking for.

[straight-shoota]

# Equivalent to `numbers.pack('e*')`:
numbers.each do |n|
  io.write_bytes n.to_f32, IO::ByteFormat::LittleEndian
end

[jwoertink]

Boom! Thanks 😄 I was stuck on that one for a while there.

[bcardiff]

My take is that there could be a shard that provides a macro via run program that could translate de pack syntax and even generate typed results.

[straight-shoota]

@bcardiff Please, no that would still have cryptic format strings end up in Crystal code bases 😆 Then I'd rather prefer a webservice at https://transform-crazy-pack-format-to-beautiful-crystal.com as a tool for porting Ruby code.

[jhass]

More importantly I still think that it would be a mistake to treat and promote String as generic byte sequence container, that's what we have Slice(UInt8) aka Bytes and IO (including IO::Memory) for.

[straight-shoota]

Since this has been coming up very frequently in the last weeks, I've added an entry to the FAQ: https://github.com/crystal-lang/crystal/wiki/FAQ#user-content-is-there-an-equivalent-to-rubys-arraypackstringunpack

[Blacksmoke16]

@straight-shoota wouldn't it fit better under the crystal for rubyists section?