Julia: sizeof reports wrong size for array of non-power-of-2 primitive types

[imciner2]

This grew out of a discussion on Slack about the size taken by arrays of BitIntegers, but basically it seems that summarysize will return the bytes Julia thinks the object takes in memory by just adding up the size of the array elements without regard for their actual alignment on chip.

Setup (this is actually independent of the BitIntegers package, and is true for any type created using a primitive type):

using BitIntegers
BitIntegers.@define_integers 40

S40 = zeros(UInt40, 10000); S64 = zeros(UInt64, 10000) 

Results:

julia> Base.summarysize( S40 )                                                                                                                                                       
50040

julia> Base.summarysize( S64 )                                                                                                                                                       
80040

However, on chip the size taken by each element in the array is:

julia> Base.elsize( S40 )                                                                                                                                                            
8

julia> Base.elsize( S64 )                                                                                                                                                            
8

So summarysize isn't actually reporting the size on chip, it is reporting the size according to the number of bytes that Julia used to create it since it just calls Core.sizeof on the type: https://github.com/JuliaLang/julia/blob/1b6101af566fd0e5981082d01e4ee113bf9fa0f3/base/summarysize.jl#L115.

The main question is, what should "memory" refer to in the documentation for summarysize? Should there be a different exposed function to actually get the on-chip memory usage?

[vtjnash]

Yeah sounds like we're not aligning the data when making the array:

julia> primitive type UInt40 40 end

julia> S40 = Vector{UInt40}(undef, 10000); S64 = Vector{UInt64}(undef, 10000);

julia> sizeof(S40)
50000

julia> Base.datatype_alignment(UInt40)
8

[KristofferC]

Yeah sounds like we're not aligning the data when making the array:

Are you sure?

julia> using BitIntegers

julia> BitIntegers.@define_integers 40
@uint40_str (macro with 1 method)

julia> S40 = UInt40[1,2,3];

julia> unsafe_wrap(Vector{UInt8}, Ptr{UInt8}(pointer(S40)), 20)
20-element Array{UInt8,1}:
 0x01
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x02
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x00
 0x03
 0x00
 0x00
 0x00
...

This looks aligned to me. Maybe it is just sizeof that reports the wrong size?

[vtjnash]

I suspect we might not have asked malloc for the right size also in that case

[kimikage]

I think it is better to clarify what the sizeof is used for and what the sizeof should return.
The current docs only mention simple cases and are not practical because in simple cases we don't need to overload the sizeof.

[JeffBezanson]

It looks like a bug in codegen for sizeof. If you call Core.sizeof directly it gives 80000, and everything is correctly allocated and aligned.