Go: math/big: add Int.FillBytes

[FiloSottile]

When implementing cryptography, what we need is almost always a fixed size buffer representing a value in big endian. What math/big.Int.Bytes provides is a variable size buffer, so all over the place there are snippets of code that do make, Bytes and copy with a len dependent index. I just implemented one the other day for https://golang.org/cl/208484, and just saw one in x/crypto/acme.

I'd be willing to bet that every Bytes invocation in a crypto package is doing something similar. I also learned that random bugs that occur approximately every 256 executions are probably due to missing this step, and I have found such a bug in the wild at least twice.

I propose we solve this at the math/big API level, and add (*math/big.Int).BytesWithSize.

// BytesWithSize returns the absolute value of x as a big-endian
// byte slice of length size. If x doesn't fit in such a buffer,
// BytesWithSize will panic.
func (x *Int) BytesWithSize(size int) []byte {
    b := x.Bytes()
    switch {
    case len(b) > size:
        panic("math/big: value won't fit requested size")
    case len(b) == size:
        return b
    default:
        buf := make([]byte, size)
        copy(buf[size-len(b):], b)
        return buf
    }
}

I don't have a strong opinion on the len(b) > size behavior. Where we use it in crypto packages we always know an upper bound, and if we cross it we might as well panic because something went catastrophically wrong. (And the current code would either panic or silently truncate, which is worse.)

/cc @griesemer @katiehockman

[griesemer]

Would it make sense to provide the byte slice to avoid a copy?

func (x *Int) AsBytes(buf []byte)

[robpike]

Yes, I like @griesemer's design. Panic is OK if documented. A nil argument could allocate, perhaps?
That is, if you provide a buffer it must be big enough; otherwise one will be allocated.

[FiloSottile]

Oh, I like that!

The nil argument behavior would make sense if we didn't have Bytes, but that's there to stay.

I am not a fan of the fact that one can't really tell which is which based on the names (Bytes vs. AsBytes) but I don't have better suggestions. Maybe BytesWithBuffer? It's a mouthful but hopefully clear? WriteBytes?

[magical]

ToFixedBytes? Since the difference from Bytes is that it writes to a fixed-size array (with left padding).

WriteBytes?

The usual name for this would be Read, not Write 😉

[griesemer]

My original thought was also Read, but that name suggests that there's a result (n int, err error) as well which is perhaps overkill, especially if we never expect an error and if we don't care about the n since the buffer will be 0-padded. Also, given a Read method, one would expect a client to be able to call Read repeatedly to get all the bytes of an Int, which is not the case.

Maybe FillBytes if you don't like AsBytes? The former implies more of an "action" (of filling in the bytes) rather than a function call (that returns a value) as does the latter.

[FiloSottile]

I like FillBytes, thank you!

// FillBytes sets buf to the absolute value of x in big-endian, zero
// padded as necessary. If x doesn't fit in buf, FillBytes will panic.
func (x *Int) FillBytes(buf []byte)

[robpike]

// FillBytes sets buf to the absolute value of x, storing it as a zero-extended
// big-endian byte stream. If the encoded value doesn't fit in buf, FillBytes will panic.
func (x *Int) FillBytes(buf []byte)

[robpike]

I'm a little unsettled by the absolute value thing, but that's what Bytes does.

[FiloSottile]

I'm a little unsettled by the absolute value thing, but that's what Bytes does.

Same, but on the other hand every way of encoding negative numbers I know of went terribly wrong, so I see where that choice is coming from.

[rsc]

Based on the discussion, this sounds like a likely accept.

Leaving open for a week for final comments.

[rsc]

No final comments, so accepted.

[gopherbot]

Change https://golang.org/cl/230397 mentions this issue: math/big: add (*Int).FillBytes

[FiloSottile]

Implemented this in CL 230397.

Note that I made FillBytes return the buffer, because it was nicer to use in all call sites I updated. Let me know if there are objections to that.

[FiloSottile]

Fixed by c9d5f60eaa4450ccf1ce878d55b4c6a12843f2f3, which for some reason didn't close this.

[lukechampine]

Sorry for not commenting earlier, but can I ask why an append-style API wasn't used here? If all I saw was the signature:

func (*Int) FillBytes(buf []byte) []byte

I would assume that it behaved like (hash.Hash).Sum (and friends), i.e. that FillBytes(nil) was equivalent to Bytes(). Given that it returns a slice, I certainly wouldn't expect it to panic.

I don't think we can change the behavior now, unfortunately, but maybe an AppendBytes is warranted...?

[FiloSottile]

@lukechampine What this API was specifically addressing, more than the performance advantage of saving an allocation, was the need to fill a fixed-size buffer with a possibly shorter big.Int, because big.Ints don't have an innate size. There is no way to do that with an append-like API.

[lukechampine]

Ah, because of the big-endianness, right. Thanks for the clarification.