Go: encoding/gob: tooBig is too small

Created on 12 Sep 2018 · 50Comments · Source: golang/go

What version of Go are you using (`go version`)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

arch linux with 16gb ram

What did you do?

I parse large CSV files for social media analytics, creating in-memory data structures from the data. When persisting these data structures to disk using gob, I receive gob: encoder: message too big error from this code:

encoder := gob.NewEncoder(file)
err := encoder.Encode(MapData)

One set of input data is a 4GB csv file which results in a data structure that is 200MB when saved with gob. The second set of input data is a 15GB csv file which should result in a 1200MB file when saved with gob, but I receive the error above.

The error is thrown at a check if the data structure is larger than the tooBig constant at https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/encoder.go#L70

The tooBig constant is originally defined with a comment TODO: Make this adjustable here https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L17 https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L18

What did you expect to see?

I expect gob to save large data structures of ~2GB that dont get close to my RAM limits of 16GB without any problems.

What did you see instead?

I got the gob: encoder: message too big error.

I fixed the error by compiling my own go and changing the tooBig check in encoder.go to 1<<31. Afterwards the data structure from the large 15GB CSV file was successfully saved by gob in a 1500MB gob file.

Therefore I suggest this change could be done for all users, as the existing TODO comment in the golang source code suggests https://github.com/golang/go/blob/50bd1c4d4eb4fac8ddeb5f063c099daccfb71b26/src/encoding/gob/decoder.go#L17

FrozenDueToAge NeedsFix

Source

👍3

All 50 comments

/cc @robpike

agnivade on 12 Sep 2018

This was first introduced in 9c3fc838ba982571e704c1674e9f97678f8a6e93, because earlier it was easy to get encoding/gob to panic with large amounts of data due to overflows.

My initial reaction would be "we can just bump this limit on 64-bit", or "we can just use 64-bit sized integers in the encoder and decoder", although I don't know the internals of the gob package that well.

The original issue was https://github.com/golang/go/issues/8084. Interestingly enough, it happened on amd64.

I guess it's fine to double the limit while we don't have a better long-term solution, but I'll leave that to Rob to decide.

mvdan on 12 Sep 2018

👍1

@mvdan thanks for your input. I tried to looking through the gob source but I can't really make out what kind of integers are used.

But what I can tell you is that I changed the data structure from map[int]map[int]map[int]map[uint64]bool to map[int8]map[int8]map[int8]map[uint64]bool in order to save disk space and potentially work around the error message. I found out the size of the gob dump file did not change when using int8 instead of the standard int (which afaik is at least int32).

bf on 12 Sep 2018

@bf FYI, gob uses a variable-length format to encode integers, so it makes sense that changing the integer type in your data structure did not affect the output size.

magical on 13 Sep 2018

❤1

Maybe tooBig should be bigger on 64-bit. On 32-bit I think 1GB is fine. Thoughts, @robpike?

rsc on 19 Sep 2018

Sure.